Dave Dash

Alphabetical sorting in Sphinx

2010-04-21T00:00:00-07:00

Sphinx 0.9.9 is great at searching full text, but treating actual strings as attributes takes some work.

Initially I employed the strategy of indexing my full text fields and storing them as attributes. E.g.:

sql_query = SELECT name, name AS name_ord FROM documents
sql_attr_str2ordinal = name_ord

This stores each attribute in lexical order. Meaning if your name's are Apple, Aardvark, Button, Choco-room they would be given the ordinal 2, 1, 3, 4 respectively.

However, this is case-insensitive. So trying this approach:

sql_query = SELECT name, UPPER(name) AS name_ord FROM documents
sql_attr_str2ordinal = name_ord

Will allow for case-insensitive alphabetical sorting in Sphinx.

Test Driven Confidence

2010-04-20T00:00:00-07:00

If you're already testing your web applications, you can skip this post.

One of the bugs I am working for AMO on involves porting a small, but moderately complicated checkbox from our PHP site and rewriting it for Django.

I decided to look at the existing implementation and found it to not work correctly at all. This was frustrating, especially since I verified that my own code worked, and that QA verified that it worked as well.

This is frustrating on many levels. Chances are some minor assumption I made changed, and thus broke this functionality. Discovering regressions is never fun, and fixing them is can be long and tedious if you can't automatically verify that everything is working correctly.

Lucky for me, coming up with tests is easy, you just do what you would do to verify the code satisfies the requirements and then code it. Sometimes the tests can take longer than writing the actual code, but ultimately you can ship with confidence. You can be confident that your feature won't break in the future without immediate notice, and you can be confident that your new code won't break anything else.

Finding the most common Firefox issues

2010-03-18T00:00:00-07:00

Cheng Wang of the Mozilla Support team, a few months back, decided to present on some design ideas for Firefox Support. One of the issues he noted was that there are a lot of repeated issues and that it would be useful to group them. Grouping them lets you see how often something occurs, and secondly let's you see how urgent it might be.

Luckily grouping and clustering text is something computers can do. So I wrote this utility that does just that.

I ran this script over a sampling of data from the last week:

Firefox won't start after update. (65 related issues)
- 5.6: Firefox updated, Gmail not delivering mails
- 5.6: How to change My Profile when Firefox won't load?
- 7.5: Once I close firefox, cannot start firefox again except system restart
- 5.6: When intalling updates Firefox uninstalls itself
- 16.8: firefox won't start after update 3.6
- 11.2: Upgraded to Firefox 3.6 and now it won't start
- 14.9: Firefox won't start with most extensions
How do I add a bookmark to more than one folder? (64 related issues)
- 8.9: How do I get my bookmarks on the bookmarks toolbar to show up as an icon only with no text?
- 7.5: Bookmarks lost after upgrade and cannot save new bookmarks
- 7.5: why do i have to add the .com now to addy's?
- 8.7: When I open sidebar to edit bookmarks, I only see the folder for Bookmarks Toolbar. I do not see a folder just called Bookmarks nor do I see my list of bookmarks, that separately appear under bookmarks menu at top of screen
- 7.5: All my impoted bookmarks go to the same webpage
How do I remove the \ask toolbar\"?" (50 related issues)
- 14.9: How do I remove an unwanted toolbar?
- 5.6: how to remove temporary video files from computer
- 7.5: I have no Toolbars or searchbar and i cant bring them back
- 7.5: nowhere says how to REMOVE a toolbar - only how to add or modify one
not able to open youtube videos (45 related issues)
- 5.6: Cannot open bookmark/history sidebar
- 5.6: After working well for years Firefox will now not open
- 6.7: opening bookmarks do not open in new tab
- 5.6: I can't watch videos on youtube with firefox, but on internet explorer i can
I cannot download Firefox 3.6. I've tried erasing the download file. I cannot get beyond logging out of Firefox. (44 related issues)
- 8.4: when downloading files firefox download manager will freeze and i will have to start over the file download
- 5.6: Firefox will not let me download anything! Can someone help?
- 6.3: cannot download epixHD.com: not compatible with firefox 3.6
- 5.0: Several tabs are coming up when i try to downloads things
- 5.0: Firefox wont open since I downloaded the 3.6 update.

The number on the right of the related issue is a score of how strongly it relates to the main issue.

The full sample is 352 clusters from an original 3000+ issues. That's a lot less stuff to go through. We can tune this to have either less clusters, and more related issues in a cluster, or we can make more clusters of issues and that might result in more accuracy.

Despite the inaccuracy of clustering we can make some general observations:

Firefox not starting is a big issue.
Bookmarks are either confusing or broken.
People don't like toolbars
Opening things is hard
Downloading things or Firefox is hard

Hopefully we can fine tune these reports and have them run regularly... maybe automatically posting to Tumblr?

A few weeks in Chrome

2010-03-17T00:00:00-07:00

A number of weeks ago I got annoyed with Firefox and decided to use Chrome for a while. This reminded me of the olden days where I used Netscape for a while, and then IE6 came out, and then Phoenix came out all the while I'd keep switching to the newest shiniest thing (note: I'm not sure about the timeline of all the browsers either).

My browser of choice since Firefox was released has been Firefox. For some time - nothing shiny in browser-land was coming out. Little UI things in Safari kept me away (and the lack of extensions), but Chrome finally showed promise. WebKit, out of process plugins, process separated tabs and now extensions. This was great.

I immediately felt like I was going to really love Chrome, and be that guy at the office (I work at Mozilla) who insists on using Chrome (just like I was that guy at Yahoo! who used Google for everything). I also wanted to answer the question as to why so many people really like Firefox in spite of Chrome's amazing speed -- even many Googlers will admit to preferring Firefox.

Overall I'm happy with Chrome, but I'm switching back to Firefox for now. Here's some things I observed:

"/" in Firefox let's you search. Which to me seems more natural than Ctrl-F. I am pleased that Chrome supports a lot of Firefox's shortcuts, like Cmd-1..9 for switching tabs, or Cmd-Shift-T for reopening a closed tab.
No titlebar... I kind of miss it.
XML is way easier to work with in Firefox. It's collapsable and always looks pretty.
Certain sites don't work well in Chrome, like Rypple or the AmericanExpress web site. Rypple surprisingly enough is built using the Google Web Toolkit. I really wish their was a "FirefoxTab" that would open certain sites in Firefox instead.
There's a number of Jetpacks and Extensions that only exist for Firefox or they are severely lacking.
- The Jetpacks for Mozilla's Bugzilla instance are awesome.
- The Delicious and AdBlock extensions on Chrome aren't nearly as good as the ones for Firefox.
Firebug is much better than the Chrome developer tools. For example, you can adjust css values instantly.
Extensions die... and don't come back without restart and appear to never have been installed unless you remember them crashing.
AwesomeBar (the location bar in Firefox) queries your history much better than the OmniBar (the location bar in Chrome)
- At first I thought this was because of a sparse history, but after several weeks I still have a hard time finding sites I've been to.
- Chrome will show you a handful of results, and then let you know there are more results, but that takes you to a new screen which is a jarring UI.
Chrome can be slow. The extensions can take a while, and even switching between tabs can be slow. At this point startup time can be a moot point.

Overall this was a healthy exercise, since I really like to be up on new browsers, and Chrome really seems like it's can be a good browser for many people. I'll probably try it again after the next major Chrome update.

Making our tests run thrice as fast

2010-03-16T00:00:00-07:00

I've written a faster version of TransactionTestCase and packaged it with test_utils. It's mysql specific since it relies on SET FOREIGN_KEY_CHECKS=0 to flush the database.

The long story...

Why speed matters

We're closing in on 300 tests for Zamboni. As of yesterday, to run our entire test suite it would have taken approximately 5 minutes. If you run tests before code-reviews, during a code-review, and before you push to master - you've spent about 15 minutes doing tests for a single feature or bug-fix. We have about 5 developers, so this cycle happens many times in a work day. In that time many sandwiches can be made and consumed.

Even shortcuts, like running a subset of tests will only go so far, and ultimately we do want to validate that all our tests pass for any code-change.

Testing Sphinx search with `TransactionTestCase`

Django recently sped up testing by running tests in a transaction. However, this means that data never gets committed to the database and therefore external tools, like the Sphinx indexer, will never see any of that data. So we resort to TransactionTestCase which will commit the data.

Unfortunately TransactionTestCase is painfully slow. The accepted practice is to only use TestCase if you want your tests to be fast. So, I decided to complain to one of our new hires and he and I decided to tinker in mysql to figure out what was slow. We discovered the following:

delete from [table] is slow
truncate [table] is slow
... unless you SET FOREIGN_KEY_CHECKS=0

So we decided we should do our own tear down. After some tinkering with cProfiler I discovered that TransactionTestCase does a (slow) database flush on setup for a test case. This wouldn't do.

Making our own `TransactionTestCase`

I decided to make our own TransactionTestCase and it would just run SET FOREIGN_KEY_CHECKS=0 and TRUNCATE on each table at tear down time. It would also not do a flush on set up.

We write our tests with the idea that they clean up after themselves. Rather than having them cleanup after the last test. This is a requirement for us since django-nose doesn't reorder tests (nor should it) and a standard django.test.TestCase assumes a clean database.

Looking at a single test test_sphinx_indexer, using django.test.TransactionTestCase took ~30 seconds. Using our new TransactionTestCase it takes ~4 seconds!

Fast tests are good

We can now run our 275 tests in ~100 seconds versus the ~300 seconds it used to take. Furthermore, skipping our sphinx tests (which are the only tests that use TransactionTestCase) only saves us ~10seconds. That's not a lot of overhead for better coverage.

This took me the better part of a day, but solving this now, means we're going to more often than not run our sphinx tests all the time rather than skip them. Our QA team will assure you that search is probably the most regression prone part of our site, so running these tests are vital to quality.

If you need to use TransactionTestCase in mysql, give ours a try.

λ^2: safely doing class based views in Django

2010-03-09T00:00:00-08:00

When I started rewriting the API for addons.mozilla.org, my views were mostly the same: get some data and render it as either JSON or XML. I also wanted all my API methods to take an api_version parameter, so I decided class based views would be best. This way my classes could just inherit from a base class.

To do this I had to implement a __call__ method. This works fine, except I wanted to store things into the class -- after all the whole point of my use of classes was to keep the code a bit more compact, and cleaner. So, why pass the api_version around everywhere? Unfortunately thread-safety comes to play, and you need a separate instance of your class for each request.

λ

Django's urlpatterns expects a callable object. So you can't give it an instance of AddonDetailView(). But you could give it a callable that creates an instance of AddonDetailView() and passes it *args and **kwargs. Luckily python has lambda functions. You can note how we solved that in our urlpatterns.

λ λ

But wrapping all your urls with lambda is tedious and remembering to pass *args and **kwargs is error prone.

So let's make a lambda function that returns... a lambda function that turns an instance of our class into a callable.

We can now return to coding and not think about thread safety.

λλλ

django-fixture-magic: Testing issues with real data.

2010-03-05T00:00:00-08:00

I just released Fixture Magic.

When dealing with legacy data, you'll run into all kinds of edge cases. Perhaps, an object might not display correctly unless it has the right parameters, or if it has null parameters it might not display at all. So when testing Django, it's nice to actually use non-dummy data.

Luckily Django has a way of pulling real data out of your database using django.core.serializers:

from addons.models import Addon
a = Addon.objects.get(id=3615)
from django.core.serializers import serialize
jsonize = lambda a: serialize("json", a, indent=4)
jsonize([a])

This solution runs well in a Django shell and can be lots of fun for the whole family... until things get complicated.

Serializing alone isn't enough.

Serializing a fixture with foreign keys means you'll have an un-loadable fixture unless you serialize the dependent fixtures. Even for one or two foreign keys, this can be a pain. For addons.mozilla.org, we have a spidery-web of dependencies: Files need a Version which needs an Addon which need Translations.

Thus begat the dump_object management command. Give it an app, model name and a pk and it will give you not only a serialized JSON of that object, but all the objects that it requires.

Example:

./manage.py dump_object files.file 64874 64876 > my_new_fixture.json

This looks for the File model in the files app and pulls out of the database Files instances with pks of 64874 and 64876. It then recursively searches for any required objects.

Too much serial

If you create a lot of fixtures, you'll eventually have overlapping serialized objects. In addons.mozilla.org we have Addons, Versions (which depend on Addons) and AddonCategorys (which depend on Addons and Categorys). If we wanted to get serialize a specific Addon, it's dependent Versions and AddonCategorys it makes sense to start with dump_objecting the related Version and then dump_objecting the AddonCategory. Both dump_object commands will fetch the Addon in question, resulting in duplicated data.

To combat this we can use merge_fixtures to dedupe our fixtures:

./manage.py dump_object versions.version 64874 > 1.json
./manage.py dump_object categories.addoncategory > 2.json
./manage.py merge_json 1.json 2.json > happy_fixture.json

This should make creating test data slightly less painful. So give it a try.

Retrieving elements in a specific order in Django and mySQL

2010-02-11T00:00:00-08:00

If you have a list of ordered ids and you want to turn them into an ordered result set you can use FIELD() in mysql:

SELECT * FROM addons
ORDER BY FIELD(id, 3, 5, 9, 1);

This is a handy trick if you use an external search engine which gives you an ordered list of ids and you want to pull out entire row sets.

We do this in addons.mozilla.org using the Django ORM like so:

The code in action.

Palm Pré: A retraction, I really like it now

2010-01-11T00:00:00-08:00

So before I went on trip to Minnesota last month, I decided maybe I would give the Palm Pré another shot. After all, my parents have no internet access, so having the Pré... if I could overcome my issues, might be a welcome distraction.

Before I packed it, I updated to WebOS 1.3.x (a few days later I updated to 1.3.5) and I was blown away. The horsepower was increased by utilizing the GPU. The following problems were fixed:

* The device was no longer hot all the time
* Shutdown and startup were long, but not nearly as long as before.
* Render times were quicker
* All the elements usually rendered quickly in an app
* Network was fairly steady
* Phone calls also seemed fairly drop-free.

All these improvements helped me get over

* The tiny keyboard... not so bad in practice.
* No soft keyboard - I missed it, but I could deal without it.

Overall the device was great, it was fast enough to use, and most of the errors were annoying, but things I could deal with. Cut and paste could be improved, and I wish the USB connector was the same as the one for HTC devices (I can't keep micro or mini USB types straight).

So I love the device, and Murphy's Law dictates if work gives you a phone you don't like you get to keep it... until you start liking it again. So I sent the phone back into rotation for other people at Mozilla to try. Have at it.

Google Chrome Extensions Puzzle

2010-01-06T00:00:00-08:00

I went to Add-on-Con some weeks back to represent my employer, the Mozilla Corporation.

One of the goodies you get as a registrant was a jigsaw puzzle from the Google Chrome Extensions team.

Perfect, my wife and I love solving jigsaw puzzles. We finally finished a few days ago. Anybody who has started at all will realize the puzzle is of a QR-code. The QR-code is an extension that will eventually lead you to a prize. It was a bit of a mini-puzzle not nearly as difficult as finding the QR code.

Although finding a QR code scanner was a bit difficult, I had to borrow a HTC Magic from Justin Scott and installed a decent barcode scanner.

Django: Model Inheritance or Related Tables wrt AMO

2009-12-15T00:00:00-08:00

When I attended DjangoCon this year, I lamented that our flagship web property was difficult to test, and not fun to develop. I figured DjangoCon was a way to placate me, and Django might mean something for some of the smaller projects at Mozilla. However, Wil Clouser, our lead web developer, announced development changes for addons.mozilla.org (AMO) that says we'll be moving to Django.

Wil was open to Django and knew that's what we in the dev team wanted. Jeff spawned our foray into a new AMO with Zamboni. I've been working on some grunt-work tasks inside and outside of Django.

One of those tasks is building a transparent layer in Django to keep users logged in from our PHP-based site. That kind of problem almost immediately forces you to ask one of the most fundamental questions you ask when using any framework:

How much do I change my app, in order to accommodate the framework?

More specifically:

Should I use the django.contrib.auth User module, and to what extent?

The more we looked into what features of Django we might want to use, django.contrib.auth was heavily tied into other things we wanted, so it made sense for us to use it. The next question is whether we try the inheritance approach or do we treat our legacy users table as a sort of User Profile and utilize the User module using the related table approach?

Using model-inheritance seems real nice, because we can pretend that our legacy user is the same thing as a djaango.contrib.auth User - but this isn't true:

Looking at our users table more closely:

mysql> explain users;
+-------------------------+---------------------+------+-----+---------------------+----------------+
| Field                   | Type                | Null | Key | Default             | Extra          |
+-------------------------+---------------------+------+-----+---------------------+----------------+
| id                      | int(11) unsigned    | NO   | PRI | NULL                | auto_increment |
| email                   | varchar(255)        | YES  | UNI | NULL                |                |
| password                | varchar(255)        | NO   |     |                     |                |
| firstname               | varchar(255)        | NO   |     |                     |                |
| lastname                | varchar(255)        | NO   |     |                     |                |
| nickname                | varchar(255)        | YES  | MUL | NULL                |                |
| bio                     | int(11) unsigned    | YES  | MUL | NULL                |                |
| emailhidden             | tinyint(1) unsigned | NO   |     | 0                   |                |
| sandboxshown            | tinyint(1) unsigned | NO   |     | 0                   |                |
| homepage                | varchar(255)        | YES  |     | NULL                |                |
| display_collections     | tinyint(1) unsigned | NO   |     | 0                   |                |
| display_collections_fav | tinyint(1) unsigned | NO   |     | 0                   |                |
| confirmationcode        | varchar(255)        | NO   |     |                     |                |
| resetcode               | varchar(255)        | NO   |     |                     |                |
| resetcode_expires       | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notifycompat            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| notifyevents            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| deleted                 | tinyint(1)          | YES  |     | 0                   |                |
| created                 | datetime            | NO   | MUL | 0000-00-00 00:00:00 |                |
| modified                | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notes                   | text                | YES  |     | NULL                |                |
| location                | varchar(255)        | NO   |     |                     |                |
| occupation              | varchar(255)        | NO   |     |                     |                |
| picture_type            | varchar(25)         | NO   |     |                     |                |
| averagerating           | varchar(255)        | YES  |     | NULL                |                |
+-------------------------+---------------------+------+-----+---------------------+----------------+

You can very easily argue that this is a profile table, which happens to have credential information thrown in.

I can see overtime, I'll just struggle to keep our legacy User to act like a Django User, whereas a UserProfile is fairly standard.

Had I been writing this app from scratch, I would have chosen the UserProfile route. This is extra data which takes up a lot of space, and changes far more often than user credentials. Changing 4M+ rows sucks, by making users our UserProfile table, any changes to that table, don't tie up the table used for sign-ins.

I'm curious what other people who port their apps to Django have done.

Palm Pre: Always hot

2009-11-19T00:00:00-08:00

So I borrowed a Palm Pré that we had at Mozilla to see what it was like. I was at first very excited, I remember before the Pre was released there was a lot of talk about how awesome-fantastic it was going to be. The stories of awesomeness sort of died, and I had thought nothing of it.

Immediately upon using the Pre I figured out why. In short, it's a crappy phone. It makes a very good attempt to do a lot, but it does them with such piss-poor performance, that nothing good is noticed.

I am disappointed. It's not even in the same class as an iPhone - maybe a future generation of Palm devices will be, but not this one. I was hoping WebOS would be a good alternative to the iPhone. It looks like Google will be doing that, though their phones haven't impressed me much either. I am hoping that maybe this phone is just a dud.

Here's what I didn't like: * The Palm was always hot. * The first run experience is painfully slow. * The first run was an indicator of things to come, startup and shutdown are ridiculously slow. * Every application is slow to render. * Not all elements of an app render. * The keys are too small. Some people aren't migrating from a Treo and aren't used to mini keys. * No soft keyboard. * The palm website doesn't let you use plus-style addressing * Media Mode was not self explanatory - and forced the phone to not work. * Network would constantly drop out. Couldn't use a lot of the data features. * Phone calls didn't work so great. * Did I mention it was ass slow, even the dialing program was slow. * The battery dies quickly * I could only cut/paste when composing, but I couldn't cut a string of text from an email. * Felt too much like an old palm

Despite the sadness there were a few good things:

When it did fetch email, and other notices, it displayed them nicely
The unification of Facebook and Gmail was pretty cool - it also made me want to trim some of those friends from highschool off my facebook - I ain't ever gonna call em.
The Icons were pretty.
The card interface was interesting.
The travel charger could be modified to work in non US chargers fairly easily.

All in all, I'm glad that I had a chance to try out this device. It showed me, that user interfaces above all need to be very fast and responsive. Furthermore, everything you try to do should be done exceptionally well. I'm hopeful that software updates can alleviate some of the problem, but I think the root of the problem is slow hardware.

AMO Search: Powered by Sphinx

2009-09-30T00:00:00-07:00

Last night, I gave a talk at the Addons Meetup at Threadless HQ in Chicago on the new search engine powering addons.mozilla.org. I'll recap the technical portion of the talk and give a bit more details.

First, I'd like to thank Harper and Threadless. It was a great location in the greatest city in the universe. Before and after the meetup, Harper was just an all-around great guy to hang with and the threadless headquarters was a nice hangout place for meeting people interested in addons.

Shortly after my talk, our Engineering Ops team deployed the new AMO 5.1 complete with a new Sphinx powered search engine.

So let's talk about search. Note: parts of this are a rehash of my talk, so feel free to skip around.

A bit about addons

Addons is a huge growing space. Arguably it's Mozilla's best kept secret. Sure readers of this blog probably know what Addons are, but ask people who aren't as web-savvy. Most people don't know what a browser is - and it's hard to explain it to people without getting technical.

We can just skip that step. Because Addons are small things that people can easily "get".

"It's an easy way to customize the internet when your surfing."

While perhaps not technically correct, its one way of explaining it to people. Maybe a better way is just showing people what they can do with addons.

On my flight out to Chicago, I talked to a person on the plane who didn't know what a browser was, but after showing her AMO she was really intrigued.

If everyday non-technical people can realize the potential of addons, it's only a matter of time before they start knocking down the doors to AMO.

So we better be prepared to handle them, and get them what they want.

The technical details of addons.mozilla.org

Everytime you open Firefox, it pings AMO to see if there's any updates to any of the addons that happen to be installed. Over a third of the people using Firefox have at least one addon, and Firefox is roughly 22% of the browser market. That means roughly 7% of people opening their browsers are pinging our servers for updates.

Needless to say it's a lot of traffic, and to support it we need a fair amount of hardware. AMO is clearly the largest site in the Mozilla universe in both respects.

Some stats:

1 mySQL master
4 mySQL slaves
2 memached servers
2 Sphinx indexer/search daemons
24 Web Frontend
Multiple Zeus ZXTM clusters all

Most of this is standard, we'll talk about Sphinx later, but Zeus is amazing. I didn't know what Zeus was until earlier this year when I interviewed with Mozilla's VP of Engineering Operations. All our requests get cached so much of our hits actually hit our Zeus cluster and not our web servers.

To see just how amazing they are read our mrz's ops blog.

Why search matters

If you have any kind of custom content and unique meta data a custom search solution is a must. Browsing through a site isn't going to cut it. Browsing is dead. Search is how you find things on a web site. On AMO you may see an addon that's featured somewhere, or you might want to see what's out there, but the right search query will find you the right addon in two clicks.

Improve Search

So my first job on AMO was to improve addons search. It was a vague request and born out of frustration with what we had. It wasn't a problem that certain things were indexed, or unicode didn't work, or results weren't sorted. We may have had all those problems, but as a product search needed to be replaced.

To me it meant that we needed some framework that would allow developers to quickly debug and fix any future search calamities at a moments notice.

So here were the goals I made for myself:

Do something that sucks less than what we’ve got
Do something that makes it easier to suck less in the future
Do something that’s easy to use for our operations team, web developers and most importantly, end-users
Reduce strain on our databases, developers and operations teams

Complex Data

Our data set is small (we have 5,000 addons), but there's a lot of secondary meta data about the addons that we track:

Addons work in 1 or more locales (e.g. en-US, fr, de, etc)
Addons are optionally platform specific (Linux, OS X, etc)
Addons work with one or more products (Firefox, Thunderbird, Seamonkey, Sunbird or Fennec)
Addons come in multiple flavors (extensions, themes, dictionaries and more)

We want to index all this data. Unfortunately to get at much of this data it involves either numerous queries, or numerous joins which put a strain on mysql. How much strain?

At peak we get about 10 search queries per second. If we do something smarter this won't have to cause a lot of strain.

Using Sphinx

Sphinx is an open source search indexer and daemon. It's used by Craigslist, the Pirate Bay and Mozilla Support. It was very easy to use and despite a complicated set of data and business logic, Sphinx was up to the task.

The challenges

We needed to search for addons in several languages. So indexing just addons wouldn't work, we need to make sure we have every translation of every addon indexed. For those counting, we have 5,000 addons, but 18,000 translations of addons.

All the joining and filtering that needed to be done for our old search still needs to be done, but we can do this all in one shot by using a mysql view. This view is a flat list of each translated addon as well as all meta data associated with it. This then gets fed into the sphinx indexer.

Along the way we ran into some issues which used to be dealt with outside of mysql, such as comparing versions. It was gross and quite a hack, so we turned the variety of acceptable version strings into integers.

We also learned that stemming wasn't a good idea as we assumed it would be. Stemming was great for searching through lots of text, but a great deal of addon searches were really just searches for product names, so we opted for substring searches. We'll see how that fares. There is probably room for improvement.

Much of this, however involved knowing our data, and knowing how it will be used by our users. Once we got that down, we could hammer it all out using Sphinx.

Wins

So Sphinx gains us a bit architecturally. We have a complicated query, but it only gets run once every 5 minutes versus the 180,000 times it was run "on demand."

Indexing happens rather quickly, just over a minute.

The API was a breeze to work with, and was easy to drop into our own codebase.

Because of our relatively small data set, and quick indexing, we're able to scale this simply by cloning and load balancing. Meaning, we just need to scale for traffic, but addon growth (which is slower than traffic growth) we can safely not worry about for a while.

Our ops team can monitor the sphinx clusters and just deploy additional nodes as needed.

Building a platform

What we've done is built a foundation for search. Not all the problems are gone, but a lot of the problems that our QA team finds are able to be resolved quickly. We have a nice pile of unit tests as well that help us keep our results in check when we start tweaking dials.

We even have the groundwork for some nifty advanced search syntax, that hopefully we can inject into future releases of AMO.

Enjoy. And if you find anything, let me know.

Getting started with pipe viewer

2009-09-16T00:00:00-07:00

Despite working on slimming the addons.mozilla.org database through dieting and exercise - I still have to occasionally do long running database tasks. So I finally tried out pipe viewer. As someone who's impatient this has been awesome. Here's some quick examples:

[root@ml-db10 sun]# pv -cN source < addons_remora.2009.09.15.sql.gz | gunzip|pv -cN gunzip > addons_remora.2009.09.15.sql
   gunzip: 10.1GB 0:06:48 [25.5MB/s] [   <=>                                  ]
   source: 3.47GB 0:06:48 [8.72MB/s] [======================>] 100%

Here we are calling pipe viewer with an argument that says to title this progress meter as source, and feeding it the gzip'd file. Pipe viewer will output two things the progress, and the actual file. We pipe that file into gunzip to unzip it, and back into another instance of pipe viewer (again with a title, of gunzip) and the standard output gets redirected to our destination file.

Now a simpler example is checking the progress of loading a large sql file into mysql:

[root@ml-db10 sun]# pv -cN sql < addons_remora.2009.09.15.sql | mysql -uroot addons_remora -p$PWD
      sql: 2.55GB 0:18:19 [5.68MB/s] [=====>                  ] 25% ETA 0:54:30

We could have probably combined all this, however:

[root@ml-db10 sun]# pv -cN source < addons_remora.2009.09.15.sql.gz | gunzip|pv -cN gunzip | mysql -u root addons_remora -p$PWD

Armed with this knowledge you can determine whether to grab a soda, a sandwich or a 2-hour lunch.

DjangoCon wrapup

2009-09-15T00:00:00-07:00

I went to DjangoCon this past week for work. Django is one of my favorite frameworks. I dropped PHP and the symfony framework to learn python and Django and I haven't looked back. I think for Mozilla's webdev team it would be the framework of choice. We have 100s of sites in many frameworks, but not a lot of resuability. Django apps are built to built to be reusable. If you build correctly you don't have to refactor, it's already done.

Here's a collection of notes I collected through the conference.

Day one

Keynote - Avi Bryant

Frameworks lock us into RDBMS = bad

This keynote mentioned the limits of modern frameworks and modern web development. Essentially frameworks are great for getting started, but as a site grows, the framework gets replaced little by little. Sometimes it can get in the way - such as with limitation of database choices.

UR doing it wrong - James Bennet

James outlined a few key problems that many Django developers run into:

learning python as you go
- doesn't work unless you know some programming upfront
- do the python tutorial
- read python in a nutshell or dive into python
Things you should know:
- subclasses
- super()
- slides went too fast... hopefully they'll be posted

All in all RTFM for python and Django :)

Learn about other py packages... like twisted. If Twisted Matrix was implemented in Ruby it would be advertised as the second coming of Christ.

Bennet's Django App review smoketests:

installable via pip, easy_install or setup.py
- read distutils-guide
- stay away from setuptools
have a README
INSTALL file list deps
Write DOCUMENTATION
- use sphinx.pocoo.org
- store it in your package and upload package docs
LICENSE (most Django apps use BSD)
Write unit tests
django-lint - to look over code (like pep8.py)

pro-django is a decent book, but not written by Bennet.

Testing - Eric Holscher

Django 1.1 encourages you to test by auto-creating tests.py.
Support for:
- Unittests
- Doctest
- Tests done in a db transacation
Test Driven Documentation (TDD + DDD)
Doctest
- easy
- can't use PDB
- Hides certain failures
Unittests via Django TestCase
- XUnit
- setup/Teardown
- adds db fixtures
- assertions
- mail testing/inbox testing
- url testing
TestCase
- Browserless Request/Response testing
- Similar to sfBrowser in symfony
Google Summer of Code (for Django 1.2)
- Coverage reports!
I need to learn PDB

Deploying Django -

Run mod_wsgi in daemon mode.

Day 2

Keynote - Ian Bicking

GNU Manifest:

I consider that the golden rule requires that if I like a program I must share it with other people who like it. Software sellers want to divide the users and conquer them, making each user agree not to share with others. I refuse to break solidarity with other users in this way. I cannot in good conscience sign a nondisclosure agreement or a software license agreement. ...

So that I can continue to use computers without dishonor, I have decided to put together a sufficient body of free software so that I will be able to get along without any software that is not free.

GNU manifesto was the idea of sharing software amongst friends
GNU has purpose - BSD, etc is just a rule - free to share
Free is not just the absense of copyright
Free is not a reaction to existing rules, but a golden rule
Not just a fight against MS
Need to find morality (the why) within the practical (the law, or what you can do)
Open sourcing closed source code isn't building open source
This might apply to Mozilla... as webkit has taken off more than Gecko.
Open source is person to person not company to company - despite sponsorship.

Using Django in Non-standard ways - Eric Florenzano

Django loosely coupled
Replace templating with Jinja 2
Copy Django methods into djangoext to easily customize Django behavior
Not using django.contrib.auth
- reasons: writing a fb app - no auth needed
- no shoehorning needed - saves time - less overhead
skip the orm?
- legacy dbs
- non standard or db (or non-relational database)
- no database
wsgi middleware has some cool shit
- repose.bitblt: autoscales images
- repose.squeeze: will concat js/css on the fly based on statistical analysis
non standard Django based apps
- YARDBird - IRCBot framework
- djng micro framework
- Jngo- singlefile cms
using admin in a nonstandard way is hard/impossible coupled with ORM and auth

Real-time web and other Buzzwords - Chris Wanstrath

more than just getting your rss feeds faster
push vs. pull
1 persisting connection vs polling
comet/flash-xml/or html5 web socket
orbitted - open source python comet server
zeddicus - does the business logic
orbitted has its own js libs - its a simple port/socket thing for your server code to deal with - not request/response.
all connections are persisting browser/orbitted orbitted/zeddicus
You can even use orbitted to connect straight to IRC and write a client in JS
Jetty also is good for comet

Also: * see webhooks * see pubsubhubub

Pluggable, Reusable Django Apps: A Use Case and Proposed Solution - Shawn Rider and Nowell Strite

PBS moved from perl to django - build a lot of reusable apps
convincing your superiors
- need a good story -
- existing base of python helped
- With Django easy to do things right without doing things slow
- be really good...
built a lot of apps to be very reusable, and pluggable based on requirements PBS had

Day 3

Keynote - Ted Leung - Sun

Django jobs are a growing market
Preferred by startups
Bespin/wave - cool
APIs are big... still
Physically impossible to create purely server-side interactions that are usable enough - rely on rest/comet/ajax/etc to bridge gap

Scaling Django Mike Malone

MM from Pownce (now sixapart)
Slides started out as "Building Scalable Web Applications"
Django didn't get in the way too much when it came to scaling
Django had tons of caching support
Cached objects by hand (memcached) and object ID lists
Use memache for sessions too
use signals to signal cache invalidation
race conditions...
Queue shit... gearman, rabbit mq, etc.
Memecached incr/decr operators are awesome
See gh/mmalone/django-caching
See gh:.../django-multidb
to combat slavelag use a memcache key to alternate between master or slave

Gearman - working later - Chris Heisel

Gearman - a work later alt to rabbit mq
Makes the most sense for something like cesium, with a bazillion worker ~~bees~~ foxes feeding off a single queue

Also at the con, I talked to someone about rebuilding large apps... and they took a PHP app and used URL rewriting to and a lot of PHP/Python glue code to build a seamless transitory app. The rule is, all new functionality was done up in python while the old app was in maintenance mode.

More talks here!

Snow Leopard for Macports and Mysql users

2009-09-02T00:00:00-07:00

I use mysql and macports on OSX and both were broken when I upgraded to Snow Leopard.

Mysql was a quick fix:

ln -s /usr/local/mysql-5.1.35-osx10.5-x86 /usr/local/mysql

(you're installed version might be different). It turns out a symlink was removed during the Snow Leopard upgrade.

As for MacPorts, I had to install Xcode from the Snow Leopard CD, install the Snow Leopard version of MacPorts and then follow this migration guide.

Fun with HTML5, Blockchalk, Bookmarklets and Google Maps

2009-09-01T00:00:00-07:00

I've been filing away neat things that I've learned. Like:

Firefox supports GeoLocation (which varies in accuracy, but is really accurate for me)
Stephen Hood released the Blockchalk API
Google Maps lets you use GeoRSS feeds as a term

This solves the problem I had with BlockChalk, which is I wanted a way that I could see what's going on near where I am - and I don't have an iPhone.

So I wrote two bookmarklets:

Blockchalk Me which will list Blockchalk listings near you

and

Blockchalk this Google Map which only works if Google Maps is open. It will load Blockchalks that are near the center of the open Google Map.

Unfortunately Blockchalk doesn't have a lot of data yet, and will return no results if there's nothing within a mile radius. Hopefully a radius parameter will be included for the API call.

So there's no guarantees on the first bookmarklet, but the second bookmarklet should yield nice results for this location.

Enjoy.

git svn rebase... forever?

2009-08-12T00:00:00-07:00

While working on addons.mozilla.org I ran into an issue of git svn rebase continually asking me to merge a file, over and over.

I had a branch open for a bug. In that branch I wrote a library. While that bug was under review, I had to use that library in a new branch for another bug - and had to develop on it a bit.

$ git co -b bug1 master
$ vi libs/mylib.php # make the lib
$ git add .
$ git commit -m "my new lib"
$ git checkout -b bug2 master
$ git checkout bug1 libs/mylib.php # copies this file from one branch to the next
$ git commit -m "lib copied over"
$ vi libs/mylib.php # hack on the lib 
$ git commit -m "awesomized lib"
$ git svn dcommit # push it up
$ git checkout bug1
$ git svn rebase #... oh shit

So the rebase was happening. This is git trying to merge your changes in bug1 and bug2 and play them together in realtime nicely, asking you each step of the way to merge things manually. I thought something weird was happening since "libs/mylib.php" kept needing manual merging. Then I noticed that git is applying a series of patches, and that eventually this will resolve and your site will be rebased.

Don't lose hope, git svn rebase will finish.

V is for Version Hell

2009-08-07T00:00:00-07:00

Versioning is quite difficult to deal with. Versions are nearly-numbers, but you can't quite sort them using standard numerical algorithms.

While the following is true:

1.1 < 1.2

The following is also true:

1.2 < 1.18 < 1.20

The "." is not a decimal point but a separator.

Mozilla uses a modestly complicated versioning system that involves stars, plusses, and sometimes "x".

I found a very convoluted way to translate these versions into large integers. The versions for applications in the AMO database have four parts at most, they are potentially alpha or beta and potentially a pre-release. In some cases we have multiple versions represented with .*, .x or + at the end.

The Toolkit docs let us translate "+" to mean "pre-release of the next version". E.g. 1.0+ is 1.1pre0. Since my primary purpose of all this is for sorting, .* and .+ may as well just be a very large "version part." Since all the version parts I deal with are a maximum of 2-digits, I turned .* and .+ into .99.

For example:

3.5+ => '03'+'05'+'99' => 030599

We also need to deal with versions that may be alpha, beta or not. If everything else is equal:

3.5a < 3.5a5 < 3.5b < 3.5b2 < 3.5 < 3.5+

We assign a single integer to represent a version's "non-alphaness":

a => 0
b => 1
non alpha/beta => 2

We assume that 3.5a = 3.5a1. Therefore:

'3.5a => 3.5.0a1 => '03'+'05'+'00'+'0'+'01' => 030500001

Similarly if it's a pre-release we assign a 0 or 1 to represent "non-pre-releaseness":

'3.5a pre2 => 3.5.0a1pre2
=> '03'+'05'+'00'+'0'+'01'+'0'+'02
=> 030500001002

So what does this get us? Integers which we can use for comparison, sorting, etc. It's a one time calculation for each version and we can do some nice SQL statements in AMO like:

mysql> SELECT version,version_int FROM appversions WHERE application_id = 1 ORDER BY version_int LIMIT 15;
+---------+--------------+
| version | version_int  |
+---------+--------------+
| 0.3     |  30000200100 |
| 0.6     |  60000200100 |
| 0.7     |  70000200100 |
| 0.7+    |  80000200000 |
| 0.8     |  80000200100 |
| 0.8+    |  90000200000 |
| 0.9     |  90000200100 |
| 0.9.0+  |  90100200000 |
| 0.9.1+  |  90200200000 |
| 0.9.2+  |  90300200000 |
| 0.9.3   |  90300200100 |
| 0.9.3+  |  90400200000 |
| 0.9.x   |  99900200100 |
| 0.9+    | 100000200000 |
| 0.10    | 100000200100 |
+---------+--------------+
15 rows in set (0.00 sec)

I can now index these integers using Sphinx and do some very easy searches for addons based on version number.

Delicious keeps you in the know

2009-08-04T00:00:00-07:00

My last task at Delicious was to build along with the amazing Vik Singh was to build a new feed of bookmarks that was heavily influenced by Twitter. It was one of the most interesting and enjoyable pieces of code that I worked on at Delicious.

Over two months since my final check-in, the code is now in production. It is mostly as intended, but is lacking an RSS or JSON feed (which I had already built). This is somewhat disappointing since I was hoping that Delicious would remain as open as it had been in the past.

The algorithm is fairly simple we take a look at what trending topics exist at any moment in time (via Google Trends and Twitter) and we combine it with a list of popular terms. We take the whole lot of these items and query search twitter and store an in-memory data table of tweets. We also take a snapshot of new URLs to the Delicious corpus (basically anything on Delicious recent with 1 save). We cluster the Delicious URLs and then find tweets that are similar to each of these clusters.

The code for this is similar to Vik's TweetNews - but I think the Delicious data is a nicer fit.

We optimized this quite a bit and built a very fast inverted-index and tweaked the code to run in about a minute. Like TweetNews the heart of this was built using Python. Python while being a dynamic language is quite amazing for manipulating and iteratiting over sets of data.

While building this tool, it became my way to feel pulse of what's going on. I could ditch a lot of my RSS feeds and rely solely on Delicious to be on the up and up. Unfortunately I can't subscribe to a feed for this. Either delicious has made a mistake and didn't launch their feeds at the same time as their web (entirely possible, since Delicious hasn't been updated for most of 2009) or they are deliberately taking a step backwards.

This step backwards is weird from the usability issue. Delicious has always been a tool that allowed for multiple types of consumers and a tool that appealed to developers thanks to its myriad of RSS and JSON feeds. I'm glad I didn't have to be on the losing side of that decision. Delicious relies heavily on Google Trends and Twitter Search. While there is no requirement for them to share the data they are mashing up, it would be the right thing to do.

Let me know what you think of the new feeds. I wish I could share a github link or something snazzy so you could play around with it, but this post should be a good starting point for other real-time data mashups.

Update: Read Vik's account of this on the Delicious Blog.

Have unique descriptive page titles

2009-07-31T00:00:00-07:00

One of my internet pet-peeves is people using the same page title for every page on their web site. Take a look at this search for Sphinx. As you can see virtually all the links for Sphinx are titled "Sphinx - Free open-source SQL full-text search engine" which blows for usability when it comes to searching, or even managing the various pages you might have open in your web browser.

To get an idea of the page I want I need to look at the abstract which may or may not give me a clue. Even the forum posts which usually have subjects, have their <title> set to the site-wide default.

The first step in solving this, is identifying you have a problem in the first place. So I wrote a tool in python to determine how unique the page titles you have are:

% python measure.py sphinxsearch.com
1050 titles found for sphinxsearch.com
483 unique titles found for sphinxsearch.com
46% of the pages on sphinxsearch.com have unique titles

This is fairly telling. It means over half the pages on sphinxsearch.com have a generic title.

This site faired a bit better:

% python measure.py spindrop.us
988 titles found for spindrop.us
822 unique titles found for spindrop.us
83% of the pages on spindrop.us have unique titles

So please, think about people trying to use the information on your site. Design your templates in such a way that you can come up with unique titles.

Feel free to expand on this tool, it could easily output the offending pages or titles.

Comprehensive list of international dialing codes

2009-07-01T00:00:00-07:00

I get bored with mundane tasks. So I create little adventures for myself. I had to create a list of countries and country codes to use on the Firefox mobile home page. The first few lists were incomplete, so I made my own by parsing a list provided by the International Telecommunication Union. I stripped it down to simple forms of the country names and removed codes that are very rare (satellite phones).

Well this could be a boring task for most people, so I placed it on [github][http://github.com/davedash/International-Dialing-Codes/]. Feel free to use this in your own projects, I even include a perl one-liner in the README to convert this into a drop down HTML list.

Question: Building a Better Search Engine

2009-06-18T00:00:00-07:00

So I finally have one of those jobs where I can tell people almost every little detail about what I'm doing and I'm encouraged to talk to people on the intar-webs and solicit opinions.

Uh - this is more or less how I've operated at previous jobs, just now I can be overt about it.

So my new task is to work on improving the addons.mozilla.org search engine. I've built various "search engines" over time in PHP, powered by Lucene and most recently in python using an inverted index.

One tool that I've been looking at briefly is Sphinx. While my record count is low (5-10K), Sphinx basically bakes in a lot of the things I would want in a search engine. Indexing, merging, etc.

Since I'm fairly new to the add-ons team I'm still understanding the basics of what we need:

Fast automated indexing of addons for Firefox, Thunderbird and any other Mozilla product
Quick result sets
Easy deployability
Extendible
Customized ranking
Filtering (e.g. by Firefox version, etc).
Basics: Stemming and stop-words

Whether it's Sphinx, Lucene or some home grown solution, I have all that to support. But this should be fairly straight forward. What are people's thoughts?

From Delicious to Mozilla

2009-05-26T00:00:00-07:00

Today I said my good-byes to Delicious.com and Yahoo! and tonight I went to the Addons Meetup @ Mozilla to get a sneak peak at what I'll be working on in less than two weeks.

I was thrilled. I had no idea how many people to expect, but the Mozilla living room was packed - and most people were there the whole time. Real developers with really cool addons giving feedback to addons.mozilla.org directly. No matter how many blog comments, forums answered, customer care emails I responded to at Delicious - nothing beats the real insight and instant feedback you get from meeting a group of users face to face.

My brain, because of Delicious is always in data mining and analysis mode so through each presentation and each question asked, my brain was churning through things that I could build to bring some level of utility to the community.

I'm also happy to be joining an organization where everything is open sourced and available for comment. So I'm hoping to post a lot more on some of the cool tricks I do at Mozilla.

Reading urlopen and probably any file-ish things in python fast

2009-05-21T00:00:00-07:00

So I've been churning away in my last few days in Delicious-land trying to optimize some python code.

I was doing essentially this:

and it was taking almost a minute... for a ~3000 line json feed. Being the n00b that I am, I started getting my python learn on... and thought... an open url connection seems like a bad idea... let's do this:

Zoom! Took less than a second... actually putting real code in their (e.g. simplejson.loads()) bumped it to 2 seconds. Hooray.

Python Generators

2009-04-08T00:00:00-07:00

Someone had mentioned "generators" in python to me, so I decided to figure out what it was... and I figured it out. I think a simple example would help explain it:

Basically you can iterate over this "generator" which is a special function that "yield"s more than one value if you keep pinging it.

Adding this to the toolkit in my brain. I'm thinking one potential use is taking a resultset from a datastore (e.g. mysql) and converting it into an object.

Fixing the DiggBar with Delicious

2009-04-02T00:00:00-07:00

I've been tooling around with the DiggBar and I've been quite pleased... but I was a bit annoyed that it was hard to bookmark things in Delicious via the digg bar. I'd have to either close bar and then bookmark in delicious, or right click on the url in the DiggBar and then change the title.

I'm fundamentally a lazy programmer, so I wrote some greasemonkey to save me some time. Hooray... diggBar+Delicious.

Designing a tagging system: what is tagging

2009-03-30T00:00:00-07:00

There's a lot of collective knowledge about tagging and how it works, but everybody seems to have their own way of doing it and understanding it.

As the delicious engineer who was responsible for making sure the data integrity was maintained during transitions, I ran a series of audit tests on our bookmarks. This kept me cozy with our tags on a product level but with some insight into the engineering behind it as well.

I think the biggest struggle with tagging is that its surrounded with "Web2.0" hype. Technologically tagging is not new. The way its used is very clever, but its fundamentally simple.

Tagging is search

Tagging is no different than keywords assigned to a document. I'm using document to mean any such object that might be tagged or indexed by a search indexer. It's clever because rather than using a machine to index a document, or even a publisher to assign keywords, end-users directly assign keywords. This creates the "folksonomy."

Let's say I have a tagging system, and I tag a photograph with the term "milkshake". I can conceivably go to a page called "Things tagged as milkshake" and one of the items will be my photograph. I've just performed a search.

If this sounds simple, that's because it is. Tags are a user-instantiated index. It is different because you can do some cleverly cool things with it. With delicious, I determine my personal index. That means, I can very quickly find comical videos or neat stuff about django without too much trouble.

The fact that it's "just search" means when you build a tagging system, you can do the same tricks you can do with search. A mysql database with a tags table can be your search index - or you could use Lucene or Sphinx.

The point is tagging and retrieving tags is search, so if you can build a search engine, you can build a tagging system.

Optimizing via YSlow

2009-03-22T00:00:00-07:00

I'm not a performance person per se. I am a heuristics type of guy. Meaning you tell me a bunch of things to look out for, I get an understanding for those things, and if I agree, I'll fix them. I work off lists :)

YSlow is therefore a boon to me. If you aren't familiar with it, it is a plugin for Firefox (with a dependency on Firebug) that will grade the performance of a given site based on a set of rules determined by YDN. This is my list, and YDN went into great detail to explain each of these items is important.

The following is my journey through optimization. I went from a D- to a C in a single day. While that might not seem like much, I could have nabbed an A if I used a CDN, like CloudFront, or even a fake CDN and dropped Google AdSense and Analytics. Those tools are helpful for me, and worth keeping around, so I opted to take the grade penalty.

Begin: D - 61

I began with a D. It's actually not that bad. In most dimensions I was getting an A. Namely because I use a web framework (Django) that is optimized for being cached and served well.

YUI Combo-loading: Sneaking up to D - 62

My first goal was to reduce the HTTP requests.

I use the YUI CSS and javascript framework as the foundation for my CSS and javascript. One major benefit is that they offer to host this for you on a true CDN. The javascript framework is very thorough and can do quite a lot. Because it's so robust, it is split into several files. Originally I had been doing this:

That was 4 separate calls from the web browser to this server. While they are all fast requests, it'd be better to make a single request that catches all of this.

Luckily, YUI has "combo-handling". I was able to turn those four requests into the following:

A very quick fix, that brought my overall grade up by a single point.

Concatenation and Minification: Instant C- - 72

I still had a large number of HTTP requests. Luckily YUI has the YUI Compressor. It compresses javascript and CSS and even gives you tips on writing better code (or more compressible code).

This used to be scary and not something I'd want to deal with, but if I can run it from the commandline, then I can add it to my deployment script. Now on each deployment my CSS and javascript are concatenated and compressed and uploaded to my server. When my app is in production mode, it seeks out minified code, versus more verbose human-readable code.

The javascript compression was great. The output was at least compressed in half. This skyrocketed the score to 72. Less HTTP requests and Javascript minification were the individual rules in YSlow that were boosted.

Google: They want you to fail

This is of course tongue-in-cheek. Google has some wonderful tools like AdSense and Analytics that do hinder the score and ever so slightly the speed of your site.

As far as javascript is concerned, I am linking to: * YUI * My own JS * Google Analytics * Google Adsense

Google Adsense is by far the worst offender. You think you're calling just one solitary call to Google to fetch an ad, but it's actually 4 calls. I'm sure this could be engineered better. The good news is it's usually not too slow.

There's an option to self-serve ga.js the Analytics code. While this might result in a higher score, there are a few issues.

The process is more error prone, there's a chance that the file you download from Google might get corrupted and your unit tests might miss it.
You are serving from your own servers - which could either be expensive or slower.
Many sites include a call for ga.js and therefore there's a high likelihood that it's already browser-cached.
The code could get out of date.

For this reason, it's better to lose about 5 points for not serving it. This score is large because it effects a lot of YSlow's rulesets. If you'd still like to self-serve here's some discussion on that.

This will likely give you an "F" on the Expires Header ruleset as well.

Content Delivery Network (CDN)

A lot of people freak that CDN is on YSlow and feel like not all sites need/require a CDN. They are right that the don't need a CDN, but in my opinion they don't necessarily need to get an "A" either. All this is an approximation of doing the best you can do to make your frontend code as fast and efficient as possible.

With that said, CDN's are becoming more and more in reach. Amazon Cloud Files which I mentioned is a pay as you go system, which is quite affordable (I estimated < $1/month for my needs).

I configured YSlow to list some of the standard Google and Yahoo API hosts as CDNs. This didn't affect my score, and I didn't want to integrate S3/Cloudfiles at this stage, so I took the "F".

Gzip files (still at 72)

Adding Gzip to the Nginx configuration was easy, however my grade still stayed at 72, but it did up my score for that specific rule. Gzip compression is supported in most modern browsers and is usually a config change away.

Again, note that ga.js is not served compressed.

Upgrading from `urchin.js` (up to 73)

In the process of tweaking my Gzip configuration, I noticed that I was still using the legacy urchin.js instead of the more modern ga.js. Simply changing this boosted me to 73.

Move Javascript to the bottom (still at 73)

Moving the javascript to the bottom is another trick. This at least loads most of the HTML before the Javascript loads (and in many cases, waits).

Some refactoring of Django templates makes this a breeze.

Conclusions

I was able to raise the score from a low D to a mid C, at some point I could make the score a high C, but it was inevitably not worth it.

From my perspective, Google could do a few things to help webmasters who use their widgets:

Add a decent expires header to ga.js - this file is safe to cache for some degree of time. Being at the latest and greatest ga.js is nice, but not necessary.
Serve files Gzipped
Use a single host for serving public APIs. Yahoo can improve on this as well. They can potentially do combo-hosting a la Yahoo Developer Network

Ultimately YSlow is a good guideline for speeding up your page render times. It's not perfect, nor exact. There's no realistic way to determine a users experience with page load times. Slow network connectivity and bad browsers can make all your attempts in vain. However, YSlow is an easy to follow set of heuristics which every site owner should attempt to implement as well as possible.

YSlow's future incarnations will yield some more flexibility. Even in its default state it should yield better scores for people who are doing the right thing.

AppEngine is not a free CDN

2009-03-08T00:00:00-08:00

I've been doing some personal research with YSlow and looking into CDNs and reading comments on various blogs about CDN solutions.

I think people make a lot of assumptions as to what is a CDN. Even people who correctly define what a CDN is, will later make this mistake.

To quickly summarize in layman's terms: A CDN is a system which sends data to a user using nearby servers, rather than a central server. So your web site might have dynamic content hosted in one location, but a user requesting static assets (images, stylesheets, videos, etc) will have it served from a server that is physically close to them. So I might have my logo ultimately served from San Jose, whereas someone in New Jersey might have it served from a server in New York.

This is not the service that Google is offering with AppEngine. It is not the service you get for hosting with Yahoo's unlimited small business hosting, and it's not even the service you get with Amazon's S3.

These are all very specific services that don't have any guarantee that the data will be served from a location that is close to the end user. It's just specialized hosting.

With that said, all these services are worthwhile if you want to stop serving static content and rely on someone else to do it. Chances are they can do a better job than you at hosting these static assets, and allow you to focus on serving dynamic content. The one thing to remember is be sure you are using the right tool for the right job. AppEngine is great for writing apps on the Google Infrastructure, but it doesn't necessarily offer the same SLA for serving content as S3, or Yahoo! Small Business might.

Also note, that if you are using S3, and really need a CDN, Amazon offers CloudFront.

Automating Django Redirects

2009-03-07T00:00:00-08:00

Django has a very simple redirects system. Simple as in it's easy to understand. If someone encounters a 404, Django Redirects catches this and does a final lookup to see if there's an entry to a new URL in the redirects table.

The real win, is when your slug fields change (and thus the get_absolute_url() of your objects), you can simply automate the creation of a redirect:

Note you must have enabled the redirects app before trying this. You also need a get_absolute_url() function defined for your MyObjectClass. This is the true value of Django Middleware. Let those truly repetitive site-wide tasks get done in one spot.

A stitch in Fabric saves time

2009-03-02T00:00:00-08:00

Each day as I grow as a developer I pick up better habits. One is automating anything that I'll have to do more than once.

Therefore, deploying important projects has evolved:

Edit files live on the site.
FTP files from my local machine onto the live site and do tweaking as needed.
Develop using a cross-platform capable framework and deploy live via rsync.
One step deployment.

I looked at a few methods for performing one-step deployment. Shell scripts seemed to basic, and things like Puppet seemed to large a scale. So I looked into Capistrano and Fabric, and settled on Fabric, because it was barebones and python.

I'm at a stage with my project where I need to test it on an external server, so rather than uploading it and winging it I actually thought about a few things.

The task

I was thinking the ideal situation is a single command line that I can type that will upload my files to the server. Furthermore it should upload first to a new directory and then symlink and restart the server to do the cutover. I can easily write another script to rollback.

The way I setup my Nginx and Apache servers was they would look for the symlinks: staging, test, production to serve up the respective environments (I'm using a single host for this example).

Directory Structure

I use a very particular directory structure:

$WWW_APPS/mysite.com/ - is my root app deployment directory.
- releases/ - each code push is stored in a unique directory
  - 1/
  - 2/
  - ...
  - 100/ - the directory name is the svn revision, but is a series of precise svn exports, not a checkout.
- staging - this symbolic link links to a specific revision directory that has been designated for staging purposes. This might be the last directory in releases/, e.g. $WWW_APPS/mysite.com/releases/100/
  - config/ - any config files go here (e.g. nginx or apache)
  - mysite.com/ - the actual Django project
  - scripts/ - any utility scripts, mostly database model changes
  - site-packages/ - any related libraries that I need, this isn't the best approach I need to investigate virtualenv
  - static/ - all my static assets go here
- staging.rollback - the former staging symlink is demoted to staging.rollback
$WWW/mysite.com/
- staging - This is where the static assets get served from according to nginx. It's a symbolic link to $WWW_APPS/staging/static. This link does not change during deployment.
- uploads_staging/ - This directory contains uploads (user data). It requires manual adjustment when there are data changes (deletes, adds, updates). For the most part it can stay unchanged from one deployment to the next.

Subversion

I'm currently using subversion to maintain my code. If you are starting a project anew, I suggest using git as it is designed with branching in mind. The script below should be adaptable for git instead of svn.

That being said, when I deploy code, I use the revision number as the directory name for that deployment (e.g. $WWW_APPS/mysite.com/releases/100/ for r100) .

This forces me to commit my changes and not deploy code that is not checked in.

Other viable alternatives would be time-stamped directories. Anything that is unique between deployments will be sufficient.

The `fabfile.py`

Here's the fabfile.py I constructed. It's by no means final

def staging():
    "Pushes current code to staging, hups Apache"
    # get the build number    
    local('svn up mysite.com')
    
    config.svn_version   = svn_get_version()
    
    if not config.svn_version:
        abort()
    
    config.static_path   = '/var/www/static.mysite.com'
    config.svn_path      = 'http://svn.mysite.com/trunk'
    config.svn_export    = 'svn export -q -r %(svn_version)s'
    
    run('mkdir %(path)s', fail='abort')
    
    # svn export mysite.com to path 
    run('%(svn_export)s %(svn_path)s/mysite.com %(path)s/mysite.com', fail='abort')
    
    # svn export site-packages to site-packages
    run('%(svn_export)s %(svn_path)s/site-packages %(path)s/site-packages', fail='abort')
    
    # svn export mysite.com to path 
    run('%(svn_export)s %(svn_path)s/scripts %(path)s/scripts', fail='warn')
    
    # svn export configs
    run('%(svn_export)s %(svn_path)s/config %(path)s/config', fail='abort')
    
    # export /var/www/static.mysite.com/releases/%(svn_version) 
    run('%(svn_export)s %(svn_path)s/static %(path)s/static', fail='abort')
    
    # symlink to images from /var/www/static.mysite.com/staging/images/menuitems/* new release dir
    run("rm -r %(path)s/static/images/menuitems", fail=abort)
    run("ln -s %(static_path)s/menuitems_staging %(path)s/static/images/menuitems", fail=abort)

# rotate "staging" symlinks
    run('rm %(releases_path)s/staging.rollback', fail='warn')
    run('mv %(releases_path)s/staging  %(releases_path)s/staging.rollback', fail='warn')

# staging sym to new destination
    run('ln -s %(path)s %(releases_path)s/staging', fail='abort')
    
    # server is hup'd
    invoke(hup)

def rm_cur_rev():
    config.svn_version   = svn_get_version()
    run('rm -rf %(path)s', fail='abort')

def hup():
    sudo('/etc/init.d/apache2 restart')
    sudo('/etc/init.d/nginx restart')
    
    
def svn_get_version():
    from subprocess import Popen, PIPE
    output = Popen(["svn", "info", "mysite.com"], stdout=PIPE).communicate()[0]
    return output.partition('Revision: ')[2].partition('\n')[0]

config.fab_hosts = ['mysite.com']
config.fab_user = 'builder'
config.releases_path = '/var/www_apps/mysite.com'
config.path          = '%(releases_path)s/releases/$(svn_version)'

Final Thoughts

There's room for improvement, but this is a start. virtualenv looks like another avenue I might want to explore (especially when it comes time to upgrade Django, python, MysqlDB or PIL). This also doesn't take care of priming the server with necessary packages - again, something virtualenv could make easier for me.

However, this file was not too difficult to create. I sat down, thought about the ideal solution, implemented the steps and iterated on that until the script worked. I have pushed 19 revisions of code so far, mostly to test this process, but also to make necessary adjustments. I've probably saved countless hours of logging in and doing things manually.

How do you deploy your apps?

Resizing Image on upload in Django

2009-02-21T00:00:00-08:00

I had trouble wrapping my head around Django ORM's handling of Images.

The first hurdle was realizing that there is no easy way to store images in the database. I couldn't be too upset with that, it was a flawed concept at best. I wrote a dump script to fix this, and converted my app to read from disk. One of those "good" problems.

When it came to uploading new content... I seemed to hit every roadblock. So I thought this guide might be of use for people (other than future-me). What I want to do is upload an image, resize it per the requirements of my app, and save the resized image.

The Template

The template was fairly easy, yet I forgot to specify the enctype attribute. Here's what you need:

<h2>Upload a new photo</h2>

<form action="" method="POST" class="cmxform" enctype="multipart/form-data">
  <fieldset>
    <ul>
      <li>
        
        <label for="id_image">Photo</label>
        
      </li>

    </ul>
  </fieldset>
  <input type="submit" value="Submit" class="submit"/>
</form>

Resizing an image

Resizing an image was straightforward. The Python Imaging Library is very easy to work with, and very similar to the PHP GD Library. There was one caveat...

def handle_uploaded_image(i):
    # resize image
    imagefile  = StringIO.StringIO(i.read())
    imageImage = Image.open(imagefile)

    (width, height) = imageImage.size
    (width, height) = scale_dimensions(width, height, longest_side=240)

    resizedImage = imageImage.resize((width, height))

    imagefile = StringIO.StringIO()
    resizedImage.save(imagefile,'JPEG')
    # ...

I'm using OS X and didn't have libjpeg installed. The recommended way was to install it via fink.

Generating a filename

I like using MD5 to generate filenames for images because MD5 almost guarantees uniqueness ("unique enough").

   ...
    filename = hashlib.md5(imagefile.getvalue()).hexdigest()+'.jpg'
    ...

Saving the file via the model

The model is supposed to put the files in the right spot and populate the meta data.

You can supposedly do this via:

   ...
    my_object = MyDjangoObject()
    my_object.photo.save(filename, content)
    ...

That will also take care of my_object.save(). The trick is that content needs to be a django.core.files.File object or things will never work. Further more, you can't use a StringIO for your File object. A true file object is best. Hence rewriting the JPEG:

    imagefile = open(os.path.join('/tmp',filename), 'w')
    resizedImage.save(imagefile,'JPEG')
    imagefile = open(os.path.join('/tmp',filename), 'r')
    content = django.core.files.File(imagefile)

Note: I also had to reopen the imagefile.

All together now

@login_required
def my_view(request, slug, item_slug):

    if request.method == 'POST':
        form = NewImageForm(request.POST, request.FILES)

        if form.is_valid():
            handle_uploaded_image(request.FILES['image'])
            return HttpResponseRedirect(elsewhere)
    else:
        form       = NewImageForm()

    return render_to_response(somewhere, locals(),
                              context_instance=RequestContext(request))

def handle_uploaded_image(i):
    # resize image
    imagefile  = StringIO.StringIO(i.read())
    imageImage = Image.open(imagefile)

    (width, height) = imageImage.size
    (width, height) = scale_dimensions(width, height, longest_side=240)

    resizedImage = imageImage.resize((width, height))

    imagefile = StringIO.StringIO()
    resizedImage.save(imagefile,'JPEG')
    filename = hashlib.md5(imagefile.getvalue()).hexdigest()+'.jpg'

    # #save to disk
    imagefile = open(os.path.join('/tmp',filename), 'w')
    resizedImage.save(imagefile,'JPEG')
    imagefile = open(os.path.join('/tmp',filename), 'r')
    content = django.core.files.File(imagefile)

    my_object = MyDjangoObject()
    my_object.photo.save(filename, content)

I'm sure there's room to optimize this flow. But this gets the job done. I do wish I could have used a StringIO instead of an actual file to save the image. Writing to disk twice seems silly.

Database versus files for Images

2009-02-18T00:00:00-08:00

This is a dead topic for sure, but one of the bad web development habits I had picked up was that storing everything in a database made things easier.

I actually put some effort into thinking this through. For me it was a case of management. I didn't want to have to worry about two data stores for application-generated data. In otherwords, the data in the database was all generated via my web application (either by myself or others who worked on the product, or by end users). Having to also remember that there were select files as well seemed like a disaster.

Most frameworks, correctly assume you'll upload files to a filesystem. I never fully understood this until I thought about how I'd speed things up when the time comes to speed things up. Django forces you to think that way.

Almost from the start Django encourages you to:

Upload any binary content to the filesystem with pointers in a database
Have a separate server, or even machine serve static content.

Furthermore in a cached environment or even an environment that utilizes a CDN, putting static content in one spot, including user-generated content, was a big win.

I've been porting an app from symfony to django for some time, and I had been serving images via the database. Immediately when I switched to the filesystem I saw a huge benefit. Not just a drop in database connections, but overall "zippiness" in the site.

We'll see how well this performs in the real world, but I am quite sure that I learned my lesson.

Versioning Django Models

2009-02-13T00:00:00-08:00

In symfony, versioning a model was not terribly difficult. I had my own specialized brute-force way of doing this.

In my experience it's been a lot easier to write python code than PHP code, so naturally I figured this would be an easy task. It was not. I suspect that it was not easy because I have a naive understanding of Django and my symfony knowledge was fairly well grounded.

I tried looking for a generic implementation of django model versioning, but failed. So I came up with a specific solution.

How I think of versions

For my app I chose to use a sparse versioning system. I'd have one interface for interacting with a model. For example, I have a Restaurant model and a RestaurantVersion model.

Restaurant would directly store immutable elements like name or rating or approved and it would encapsulate versioned elements like description by proxying to RestaurantVersion.

This means I only need to interface with one class and let the versioning happen behind the scenes.

The Django implementation of `RestaurantVersion`

RestaurantVersion held the data that I thought might suffer from corruption, and therefore require reversion. Therefore there's nothing surprising in this table:

The only thing of note is that I'm enforcing a 1:M relation between Restaurant and RestaurantVersion.

The Django implementation of `Restaurant`

Restaurant is where the magic happens. It needs to do a few things:

Do what a normal object does.
Proxy attributes to the corresponding attribute of a RestaurantVersion
Generate a new RestaurantVersion on demand.
(optional) Manage which version is "active".

I say "4" is optional because I don't do this myself. I built versioning into a few models because it was easier to do upfront than worry about it down the road.

Here's the code I use:

class Restaurant(models.Model):
    name           = models.CharField(max_length=765, blank=True)
    stripped_title = models.CharField(max_length=384, blank=True)
    approved       = models.IntegerField(null=True, blank=True)
    version        = models.ForeignKey(RestaurantVersion, related_name="the_restaurant")
    updated_at     = models.DateTimeField(auto_now=True)
    created_at     = models.DateTimeField(auto_now_add=True)
    new_version    = None
    
    def save(self, force_insert=False, force_update=False):
        if not self.stripped_title:
            self.stripped_title = slugify(self.name)
        
        super(Restaurant, self).save(force_insert, force_update)
        if self.new_version:
            self.new_version.restaurant = self
            self.new_version.save()
            self.version = self.new_version
            super(Restaurant, self).save(force_insert, force_update)        
        
    def __setattr__(self, name, value):
        if name == 'description':
            self.get_new_version().description = value
        
        elif name == 'url':
            self.get_new_version().url = value
        
        else:
            object.__setattr__(self, name, value)

def __getattr__(self, name):
        try:
            if name == 'description':
                return self.version.description

elif name == 'url':
                return self.version.url
        except RestaurantVersion.DoesNotExist:
            return ''
            
        models.Model.__getattribute__(self, name)
    
    def get_new_version(self):
        if self.new_version == None:
            try:
                rv    = self.version
                rv.id = None
            except RestaurantVersion.DoesNotExist:
                rv = RestaurantVersion()
            
            self.new_version = rv

return self.new_version
            
    class Meta:
        db_table     = u'restaurant'

It's not rocket science, but it wasn't necessarily easy either. Let's look at requirements 2 and 4 in a bit more detail.

Proxy attributes to the corresponding `RestaurantVersion` attributes

Proxying attributes is a way of masking the whole versioning infrastructure from the developer. We do this with __getattr__ and __setattr__ methods.

The __getattr__ determins if you are looking for description or url attributes of a Restaurant. If you are then it uses the current RestaurantVersion's attribute.

__setattr__ is very similar, except since we're changing a versionable attribute we're going to make a request to a method, get_new_version(), which we will detail later. It does what it says, though, it gets the "new" RestaurantVersion of the Restaurant.

Generate a new `RestaurantVersion` on demand

As you can see from the above code, I am "auto-versioning". This is done mostly via get_new_version(). We also have to do a few tricks to make sure the bidirectional relationship gets maintained on save.

get_new_version() either returns the current "new version", a brand new "new version" or a "copy" of the current version.

The "copy" is done simply by setting the id attribute of the new version to None. Django takes care of assigning it a new id on save, thus preserving the old version.

Note that when we create a brand new RestaurantVersion, we don't immediately set it's restaurant attribute. That's because we haven't saved the current restaurant yet. It's a "chicken and the egg" problem that gets solved in our save() method:

We first save the Restaurant, then we save a new version if there is one. If we save a new version we want to update the Restaurant a second time. This ensures that there's a 1:1 relationship between Restaurant and the current RestaurantVersion and it also establishes a 1:M relationship between Restaurant and all RestaurantVersions.

Conclusion and Drawbacks

While, I think that this setup works in my particular situation, I feel like it has some major flaws.

The code is quite messy and very specific to this model. My goal was to take care of this quickly, not necessarily in a reusable manner. I also was not familiar enough with the Django model to create some sort of extension.

This way of versioning does not allow for versioned attributes to be set upon instantiation of the model:

r=Restaurant(description="Great place")

won't work. You'll have to do:

r=Restaurant()
r.description="Great place"

I figured this was acceptable.

Lastly I'm not entirely happy with having to explicitly save the Restaurant model twice, but I think my bidirectional relation requires this.

All in all this works in my particular situation. I'm hoping that this can be simplified. I'm curious to hear about other versioning schemes.

Dear Banks, Stop Encouraging Bad Security

2008-12-16T00:00:00-08:00

I use an online personal finance site that connects to all my financial accounts and aggregates my transaction history. I love it, it's very useful, and it keeps me financially organized.

The part that annoys me is that most of these personal finance sites require you to supply your username and password for all your bank accounts. For some banks it also requires your social security number, the last five people you've slept with, your home town, your favorite color, etc, etc. Basically all the pesky sign in questions your bank might ask you when you log in.

This is a cruel necessity for companies like Geezeo, Mint, Ameriprise and Quicken Online in order to provide this aggregation service and a scary proposition for people like us who use these services. You're giving full unfettered access to companies you may not have ever heard of to all your finances.

Security questions, and personalized security questions are the wrong way to fix bank security.

People want online personal finance sites. They want all their data in a single place without having to jump through a bazillion hoops for each and every 401K, savings account, checking account, online stock trading system and mortgage account. They will gladly sacrifice security for a chance to better their financial management capabilities.

Banks need to create APIs so third-party software can access transaction data. The authentication for this should be secure, limited and revokable. Meaning, I may authorize Mint to see my Bank of America account, but at any time I can log on to BoA and deny Mint's ability to see my transaction data. OAuth may be one mechanism to achieve this.

This will achieve a few things: * People won't give out their passwords online to anybody but their bank. * Getting data into these aggregating sites will be reliable and secure. * At any time you can see who has access to your transaction data and revoke it.

Please banks, do your part to keep the internet secure. Mint, Geezeo and anybody else, please do your part of turning up the pressure on financial institutions and when the time comes... please start using these APIs.

PHPs strengths: array_count_values

2008-12-07T00:00:00-08:00

I always like to think of what different interpreted programming languages bring to the table.

Perl is great with string manipulation. I tend to use perl if I need to rewrite/reformat text.

Python makes it easy to write clean, organized code and has a lot of syntactical sugar like list comprehensions.

For PHP one of my coworkers suggested that maybe it's ease is what it brings to the table. I've accepted that wryly at first, but after some thought it's actually a real strength.

PHP has positioned itself to be easily adoptable. When I learned PHP/FI 2.0 it was because I had exhausted server-side includes and needed something better. PHP/FI was a great solution because it was easy to install (my ISP had it) and documented easy things that I could do. After all it was just some additional changes to my templates.

I never for once thought that I'd be paid at a company to code in it... but then the internet just got big.

The other way php positioned itself is the library of functions it has out of the box. Today I found myself looking for a python equivalent to array_count_values.

Its a function I wouldn't use every day, but there are rare instances where having a set of data aggregated by count would be very useful. That's exactly what this function does.

Even python's lists, sets and dictionaries, which can do with simple operations what PHP needs an army of array_ functions to do, didn't reveal anything useful. So I will use the following:

Python String Formatting

2008-12-06T00:00:00-08:00

Python 2.6 (and Py3K) introduce a new way to format strings. Perviously you did this:

"%s, Here is my string, %s!" % ('Salutations', 'Dave')

Now you do this:

"{0}, Here is my string, {1}!".format('Salutations','Dave')

At first glance this looks like syntactical-sugar. But now you can do some clever things (which you could have done before, but I think are more intuitive now). You can easily repeat strings:

def tag(name, value):
    return "<{0}>{1}</{0}>".format(name, value)

Mimic propel's update_at and created_at in Django models

2008-11-29T00:00:00-08:00

One "trick" that propel offers you is tables with fields created_at and updated_at.

The fields are self explanatory and within Propel you just define them and they just do their business.

To simulate this in Django just do:

updated_at = models.DateTimeField(auto_now=True)
created_at = models.DateTimeField(auto_now_add=True)

auto_now automatically sets a field to the current time, and auto_now_add only does this if the object is being added (not updated).

Django models: saving markdown

2008-11-29T00:00:00-08:00

I love markdown. I write my blogs in markdown. For most user text areas in my web apps I support markdown and save both the markdown and the formatted text into my data store.

To do this I had to install markdown for python. For my Django projects I prefer downloading (or externally linking) to source code versus easy_install.

Then you override your save model like so:

from markdown import markdown

class MyNote(models.Model):
    def save(self, force_insert=False, force_update=False):
        self.html_note = markdown(self.note)
        super(MyNote, self).save(force_insert, force_update)

Easy.

Custom error messages for Django forms

2008-11-28T00:00:00-08:00

For some reason, it was difficult for me to find the documentation for this. If your Django form field is required you'll by default get an error stating that 'This field is required.' You can easily replace that when defining your form like so:

class ReviewForm(forms.Form):
    rating = forms.ChoiceField(choices=STARS, error_messages={'required': 'Please choose a star rating'})
    note   = forms.CharField(widget=forms.Textarea(),)

error_messages is just a simple dictionary of validation messages that override the default.

While this is properly documented it was not quickly searchable. But the django form fields reference page documents this well.

sfGuardUser -> django.contrib.auth

2008-11-20T00:00:00-08:00

If you find yourself moving from symfony to Django, here's how you sf_guard_user's user table to django.contrib.auth user table:

INSERT INTO 
    auth_user (id, username, password, is_active, last_login, date_joined)
SELECT 
    id, 
    username, 
    CONCAT(algorithm,'$', salt, '$', password), 
    1, 
    last_login, 
    created_at 
FROM sf_guard_user;

DROP TABLE sf_guard_user;

Luckily, django uses a similar salting and encryption strategy as symfony, so it's easy to go back and forth.

Picking up loose ends: Position Pieces

2008-11-17T00:00:00-08:00

I'm picking up a pet project of mine that I was working on a few months back. It's written in Django and is openID enabled... but I didn't used django openID and I have no clue why... I wrote my own code.

My best guess is I wrote my own code, just so I could understand it better, and that perhaps django openID wasn't mature enough or documented well enough.

Whatever the reason, in the future when I take the non obvious route... that I actually document the "why?" This will save me the tedious task of determining if I need to swap out my code and replace it with something that's more maintained.

In any case, it's certainly refreshing to tool around with Django some more... it's a lot more pleasant than the PHP that I'm often subjected to...

Django Admin and Cookies

2008-11-15T00:00:00-08:00

I was dusting off an old Django project and everything was working except the admin site:

Looks like your browser isn't configured to accept cookies. Please enable cookies, reload this page, and try again.

The culprit was SESSION_COOKIE_DOMAIN = "onyxfoundation.org" being set.

The project wasn't loading into Development mode (it's triggered by hostname) so it set the cookie domain to what I require in production. Once remedied, everything worked as expected.

Google Apps: In search of a worthy email system

2008-10-12T00:00:00-07:00

Email, in a lot of ways, is one of the most critical applications for me.

There was a point in my life where I made sure all my email was kept together indefinitely, but as time grew, so did this task.

For the longest time, I had my own hosted @davedash.com, @ftmax.com addresses. These were hosted diligently at pair Networks. pair introduced the qmail system which gave birth to my trend of using "minus addressing" (if your email is user@domain.com you can use user-anything@domain.com). It was my way of seeing which sites spammed me (so far I only outed one company using this trick) and just to organize my inbox.

Gmail...

In 2004, I decided to experiment with Gmail. I had a lot of email to manage and Gmail's ajax-enabled features made its webmail service far superior to any competitors and even made desktop apps cower in shame.

The ideology of search don't sort, and archive don't delete, made Gmail a perfect storage system for all my emails. I never had to lose track of email again. Just about any conversation, or discussion I've had over email - I can pull up very quickly using Gmail.

I'd say after 4 years of solid use, the experiment is over, I was sold on Gmail.

Switching hosts.

I did, however, move my domain management over to Dreamhost. This is where it still resides, and that was a problem. Minus addressing was not a part of Dreamhost. I had to painstakingly create procmail files that simulated this. Inevitably pushing anything and everything to my gmail account if it matched the syntax I chose.

I knew Gmail supported plus addressing (user+anything@gmail.com), but I never took the plunge. My habit, was and is still minus style.

Google Apps

The Dreamhost account I used has been tossed from one server to the next, unfortunately there's a lot of hand-holding I need to do in order to make sure procmail and all my other tweaks I've made to my accounts work. I finally threw in the towel today and settled on Google Apps.

Both my wife and I use Gmail (with aliased emails @katiebonn.com and @davedash.com respectively), so I enabled an app and did the painstaking migration process.

While it's not that complicated, you don't ever realize how entangled your Google account can be. We both use GTalk and Google Calendar and we both have large Gmail accounts. Mine has 70K+ addresses and is slowly being migrated to my Google Apps email address.

In the end, I think this struggle will be worth it, since I'll have my addresses managed in a clean efficient way.

This will also pave the path for me to move more of my web sites off of Dreamhost and onto Slicehost.

Wish me luck.

Creating a new project in Mercurial versus SVN

2008-07-22T00:00:00-07:00

One of the most annoying things about creating new SVN projects is the new project dance:

$ svnadmin create $SVNREP_DIR/newproject
$ svn mkdir -m "layout creation" file:///$SVNREP_DIR/newproject/trunk file:///$SVNREP_DIR/newproject/tags file:///$SVNREP_DIR/newproject/branches
$ mkdir newproject
$ cd newproject
$ svn import -m "initial import" . file:///$SVNREP_DIR/newproject/trunk

Oh... yeah, no this isn't that bad.

It's this part that's bad:

$ cd ..
$ mv newproject newproject.tmp
$ svn co file:///$SVNREP_DIR/newproject/trunk newproject
$ # do sanity check here...
$ rm -rf newproject.tmp

That's a lot of work, and it's actually hard to remember unless you do it a lot.

Here's the mercurial equivalent:

$ mkdir -p newproject/trunk
$ cd newproject
$ mkdir branches tags
$ hg init

That's it. It's not apples to apples of course. Mercurial keeps its repository locally, and does very simple push/pull commands for synchronization. But the dreaded dance of removing a project after it's been imported and then checking out the project is really a peeve of mine.

Windows Mobile 6.1 ROM for Mogul

2008-07-06T00:00:00-07:00

A leaked ROM for the Sprint Mogul featuring Windows Mobile 6.1 was circulating the internet and many of my issues with Windows Mobile were pushing me to upgrading. For me it fixes a few things with the GPS and improves the usability of texting.

GPS

The biggest issue I was having with the Sprint Mogul was the GPS would crash continuously... forcing me to restart my device. Restarting any phone tends to take forever, so I was eager to have this problem minimized. The issue was that if an app still had the GPS open, and the machine went to sleep, upon waking the GPS driver would crash.

Something similar still happens in 6.1, but its way more resilient. Instead of restarting every day, I find myself restarting once or twice every several weeks - and that's after heavy GPS usage.

SMS Notifications

The Mogul used to vibrate and alert that your SMS message has sent. There was a registry hack to disable this for versions prior to 6.1, but that's no longer necessary.

The downside to the hack was you wouldn't have any sort of feedback that a message was sent. However, Pocket Outlook has adopted a chat style interface for displaying text messages. So now texts look like conversations - which make it very easy to have tiny conversations with people.

Do it...

Flashing the ROM is easy since Sprint (apparently) gives you a Rom Update Utility (Windows executable unfortunately) that does all the dirty work. I backed up my data using Missing Sync and then started the update process in a virtual machine.

Restoring my default settings and apps was a bit of a pain, but overall it's a decent upgrade and fixes my biggest gripes.

The Sprint SERO plan

2008-07-02T00:00:00-07:00

...

So originally I was thinking of getting the new iPhone, because they had a competitive rate plan, but they later dropped that with the new 3Gness. Simple math tells you it's not that great of a deal over two years.

Looking at the comments in the Gizmodo article, I learned about Sprint SERO and proceeded to find out as much as I could about it. SERO is the Sprint Exclusive Referral Offer which is a data-enabled plan with 500 minutes and unlimited texts for $30/mo. That's less than what I pay with a family plan now (with no data or texts). There are slight variations if more minutes are needed. 500 was enough for me, so I was on board.

Sprint SERO

The SERO plan is simple to get, you just need to enter an email address of a Sprint employee to order phones and plans from the web site or by calling Sprint SERO by phone. It's fairly easy to find these by doing some simple navigating on Sprint.com or ask around you're bound to know someone who knows someone at Sprint.

Dealing with Sprint customer service can be a mixed bag. Unlike T-mobile not everybody seems helpful. Many people there seem to be reading off a script and not deviating a bit, so it's hard to get a real conversation. But some people really know their stuff and can help you in a pinch and really want to help you.

When you activate your phone make sure you write down every detail they tell you. You'll get an NSL code that lets you re-activate your device over and over if you have to flash it, without having to call them up.

Also, if you're porting your phone number do it ASAP.

Porting nightmare

I had a bit of a porting nightmare with my phones. I had two SERO lines on one account and they ported both my numbers to the same phone line. After several email exchanges and several phone calls everything was restored. Basically of WLNP goes awry they can fix it. I was without my 612 number for a day or so.

Everything is restored now, but in the future I would have either ported both lines at once (I waited a day or two between porting) or created two separate accounts and then combined them at a later time.

Other discounts

You can dial #733 and enter in a current Sprint customer and get a $25 gift card (and the referrer also gets one). If you need a Sprint number, I can give you mine.

Generally Sprint will not give you much more beyond this since they view SERO as a huge discount (and it is in comparison to other providers).

Get it!

Seriously, as much as I hate not having a GSM phone, Sprint's SERO makes a lot of sense money-wise. The data has been excellent and the customer service has been good, even if they do make mistakes.

Open season for headhunting Yahoos

2008-06-07T00:00:00-07:00

None of my resumes floating around on the internet have my current job listed because I get enough calls and emails (mostly from Minneapolis, but some from the SF Bay Area) from recruiters attempting to help me find a job.

Here's an email my coworker received that I thought I'd translate. I removed the name of my coworker and the firm who wrote the email. I really don't want backlash from them, since they harass me enough as it is.

Hi [Name removed],

Since you've communicated with [Company name removed] in the past, I wanted to touch base and get caught up with you on your potential career advancement.

Since your name appeared on a job board in the last 5 years, or we bought an email list of job seekers we wanted to see if we could be the custodians of your resume and make a few grand by getting you to quit your job and start somewhere else.

We’ve been placing many of your Yahoo colleagues lately, and are currently working with many Yahoo employees on their passive job searches, both at the individual contributor level and Management/Director level.

We've placed two or more people from Yahoo at other jobs, can we count on you're resume pulling in some big money too?

Based on these dynamics and the information we have gathered, it looks like the timing is looking better and better to get out of Yahoo! Even if Yahoo is trying to bribe you into staying with promotions, counteroffers, or severance packages, don’t be fooled. The smartest people will leave now. One example: we had a candidate making 160k at Yahoo that would have gotten a 1 year severance package if he stayed till after the Microsoft acquisition, but he still left and is happily placed at a much better company.

Consider this: Before the last round of layoffs, we were usually able to get Yahoo candidates significant pay raises and multiple offers. After the latest round, Yahoo candidates’ value has gone down because there are more of them available in the market and Yahoo has continued to lose respect in the industry. We foresee this trend continuing through the Microsoft Acquisition, when there will probably be another round of layoffs and voluntary departures of your colleagues.

It's simultaneously getting better and better to leave Yahoo! while at the same time, it's becoming not-so "better and better" to leave Yahoo! The market economy is confusing!

This all means that the sooner you move from Yahoo, the better off you’ll be.

Let's face it. We want you to leave Yahoo! now. The sooner you leave, the sooner we can make bank by placing you elsewhere. If you stay elsewhere for at least a year, we make commission and then we can find you a new job.

Please contact me as soon as possible so we can show you some better alternatives with much hotter companies/jobs. When you email me, let me know what’s new since the last time we talked, and what type of opportunities you’d like us to get you. The job market is very hot right now!

I need to fill out the details on your record in my CRM software, that way the next time I contact you I can pretend to know intimate details about you.

Also, feel free to view my profile on LinkedIn, where you can read testimonials of people I’ve worked with (some from Yahoo), and verify that I am indeed well connected in the industry.

Finally a social network for recruiters.

I look forward to working with you

$$$ HOT HOT HOT $$$

[Name Removed]
Manager, Sr. Technical Recruiter

I'm more than just a phone jockey, I'm a deal maker.

[Company Removed]
405 El Camino Real #356
Menlo Park, CA

We live on El Camino Real, you know, between the adult bookstore and the Indian buffet.

Smart pricing for iphone

2008-05-12T00:00:00-07:00

Going to NYC really made me want a new phone. Primarily just to have Google Maps telling me where stuff is. So I've been looking at prices (theoretically for my wife and myself) more closely. Despite both devices being different, I'm treating them essentially the same:

	iPhone	Blackberry (AT&T)	Blackberry (T-Mobile)
Cost of two	$800	Free＊	$600
Ammortized over 24 months	$34	$25
Monthly rate	$109	$60	$70
Data plan	Included	$60	$40 ($30＊＊)
Total Monthly cost over 2 years	$143	$120	$135 ($125＊＊)

＊ If you go through Amazon.com

＊＊ My wife is likely to use this mostly for email and less-so for web surfing, and Blackberry has a plan for just that.

The way I see it, I'm likely to have a smart phone last me two years — at least. It's a stretch since most of my phones go about a year (til I get bored of them), but I'm hoping for better results with a smart phone. So initial costs can be spread out over time.

So I get a few things out of this chart:

Amazon/AT&T's free Blackberry is a white elephant
The iPhone doesn't cost much more than a Blackberry.
I will jump at a refurb iPhone since that will be cheaper in the end or a free Blackberry for T-mobile. Unfortunately I am currently with T-mobile so I don't get a lot of the awesome discounts that you'd find on Amazon.

Screen

2008-05-06T00:00:00-07:00

Screen is AWESOME!

I've been doing some data auditing at Delicious and we have a lot of data, and I work off of a laptop connected to other machines.

So I caved in and started diving into screen.

If your not familiar, screen is a multi-tasking terminal that you can connect and reconnect to. Everything you run in screen keeps running as if you were there. Think of it as VNC for the command line.

I keep my screen session on a linux box at work and now I can do this:

ssh -t myawesomecomputer screen -x

and be connected to my 10 different terminal sessions all at once. I'm connected to any database I might need, and I have several long running tests in others, or even just an open bash prompt at the ready on the right server. Very awesome!

Surgical Precision versus Brute Force

2008-04-11T00:00:00-07:00

Wireless networking can be troublesome... usually I never have problems, but every now and then I'll go to a coffeeshop or a café where I can't just open up my laptop and immediately start surfing.

The problem solver in me wants to figure out why, what setting needs enabling, etc.

The person who wants to cut the crap restarts the machine.

Restarting almost always is the best course of option. If it still doesn't work then diagnose it.

unixtimestamp conversion

2008-03-06T00:00:00-08:00

My coworker taught me this:

dash$ date -r 1204748227
Wed Mar  5 12:17:07 PST 2008

Yahoo Pipes and reducing information overload

2008-02-28T00:00:00-08:00

I've been suffering from some information overload. I subscribed to Engadget and Gizmodo because I wanted to keep up with some home based network devices like homeplug/powerline and wireless routers. But Engadget and Gizmodo are overwhelming.

Finally I decided... to write a pipe. If you're not familiar with Yahoo's Pipes it is the best thing to come out of Yahoo! (Flickr and del.icio.us are acquisitions originally). Pipes is what Yahoo! should be doing:

It's niche oriented: It appeals highly to geeks who understand pipes, yet it's very learnable, so you don't need to know how to program to use it.
It's got a great UI. The user interface is great, you drag and drop inputs and outputs - again very learnable.
It's very useful. As I'll show you, below.
It harnesses the technology that a large company that Yahoo! can provide that might otherwise be difficult: Pipes has to store a lot of data from RSS feeds, to screen scrapes, to anything that you feed it. Using Yahoo! technology it's trivial to obtain and process all that.
It's not a clone or a "me too" of other products: This is completely original, and very appealing.

So my pipes take a regexp filter (e.g. "delicious" or if your clever "delicious|Yahoo|Google") and give me items that match in a selection of RSS feeds. So it aggregates things like Engadget and Gizmodo and gives you just what you want to see:

Take a look: Gadget Filter

I also made a copy called Technology Filter which aggregates the SF Chronicle and TechCrunch.

Try them out, remember you can customize the filters... not everyone likes what I like ;)

Smarter excerpts: the art of the semi-automatic CMS

2008-02-28T00:00:00-08:00

I was browsing the TED site, since it's all up in my blogospheres and ran across this:

If you look at the excerpts, they are piss poor descriptions:

Frans Lanting is one of the greatest nature p...

Nature p-what? Nature painter? Nature preserver? If you click around enough... the answer is nature photographer. Many of the excerpts failed at giving me any indication of who these people are or even enticing me to find out. I am a web 2.0 person, if I have to click on a link and load a new page, then it better be good. When I'm doing an information binge, I am not going to have that level of patience.

One day at del.icio.us I decided to test the limits of every profile field... and I learned that every now and then you do need to truncate user generated content.

This page however, is trusted users who have edited structured content (to borrow a term from the Django Book). That means, that someone had to enter in a description for this person. Someone who is trusted at TED. Chances are the page was rendered to trim the excerpt at 46 characters.

There's two easy solutions to this:

DHTML

This remaining excerpt could have been hidden away, and a button that said "show all" or "expand" (or whatever is the usable phrase/icon) could show the full except. This prevents me from having to jump around.
Use Semi-Automatic excerpts

We are talking about "trusted users" meaning, we can trust them to do things that humans do better than machines (there's my del.icio.us humans-do-some-things-better mentality). When they enter the description for a person, their administration interface should be able to suggest an excerpt and they, the "trusted user" should be able to make adjustments or tweaks.

The former solution is far easier, but if you're scanning a site, especially one as extensive as TED's speakers listing, the last thing you want to do is click "expand" buttons everywhere to read everything. The latter solution is smarter. Of course, the problem could also be solved by upping the 46 character limit, but not everything should be fully automated.

FriendFeed: Pure Genius

2008-02-28T00:00:00-08:00

FriendFeed is pure genius.

I had this idea that I was going to write an aggregation service that took your sources from multiple locations and aggregated them into once nice RSS'able feed so your friends could follow your footprints on the web.

FriendFeed seems to do just that. I've listed:

Amazon.com wishlist
spindrop
davedash.com
livejournal
del.icio.us
digg
flickr
katie's flickr (since I hardly post)
gTalk
last.fm
netflix
Pandora
Twitter
reddit
pownce
youtube
upcoming
linked in
yumbo

That's a lot! I'm prolly on a few more sites than that, but having one central source where I can point people is great. What'll be better is if they can do a more Facebook newsfeed style system, where you're not pummeled with everything at once, unles you want it, and it learns what info you are interested (e.g. I am interested in Dave's flickr photos, but not so much his Pandora listening habbits). Of course that's always stage two, I imagine.

Try it out if you're everywhere on the web. Here's my feed.

Building a triangle toggler in YUI

2008-02-25T00:00:00-08:00

If someone knows the more common name for triangle-toggle menu's similar to this. Let me know.

There is a widget where an element toggles the display of a secondary set of elements. The toggle shows an arrow pointing down or right depending on the visibility of the elements. I wanted to build that in YUI.

The problem is two-fold:

Build a triangle that toggles from right to down.
Show and hide content.

However, with some clever CSS we can do this all in a single class change on an element.

Build that triangle

There's a number of ways to show or hide this triangle. Because it is flexible, I'm going to opt for using a background image. Also to save on HTTP requests, I'll use a sprite-d image. This may get merged with other icons on the system.

The particular site I'm working on uses a black background with white text, therefore a white triangle toggle seems appropriate.

This is what I use. I leave a little space, because the background will only clip horizontally based on the size of the element.

Add the HTML/CSS

The following HTML:

and the following CSS:

Provide the two states we require for the triangle. If you are new to sprites, rather than changing the background image entirely we just shift the background image up or down appropriately to show a new background.

The Javascript

The javascript is rather simple, but we put in some magical tricks here and there.

If you're unfamiliar with this style of Javascript, here's what's going on. Everything is done in the MA namespace as to not conflict with other javascript.

The toggler is fairly generic and expects a similar HTML structure for any toggle-able element. That means this code only needs to be written once, and anytime we use the toggle class, toggling in this fashion will occur.

We only run the init function. init says when the DOM is available then run the setup function. setup adds an event handler to doc4. doc4 just happens to be the id we use on our body tag.

Note that we're listening for clicks everywhere. This means we have to define only one event handler, regardless of how many toggle-able items there are. The event handler checks to make sure we clicked on a relevant element and then applies the toggle function to the grandparent element (switching the class from off to on as appropriate).

Note that this style of event handling means you need to carefully apply your class names. Also note, that this code could be optimized a little bit more, I haven't put this code into production, and therefore haven't optimized it for the YUI compressor.

Try this out if you want, I'm sure it can be trimmed down overtime. If you have trouble with it, let me know.

-d

Why I hate the Apple Store and so can you!

2008-02-23T00:00:00-08:00

So if you remember, I went to the apple store over a week ago to learn that Apple gave me a goofy battery and it needed replacing. Here's what happened:

I waited over the 72 hours maximum I was told I'd wait for a phone call.
I went to the store today and was told that I should have made yet another appointment just to pick up a battery. They could have told me that on the phone.
Instead of getting a new battery right away, I was told it would take an hour. Good god, Apple... this is ridiculous.
I spent a nasty rainy day at a mall trying to blow away gift cards. California malls are outdoor ones mostly.
I came in after an hour and was told to wait even longer.
Finally someone came out with an invoice for $110? Wait, I thought you were replacing a defective battery. Here's a hint. If a customer has been waiting for a week, and then comes in and is told to wait an hour, read your notes that say, this is a replacement for a battery recall.
The guy went back and got new and improved paperwork that says the battery is free.

Seriously.. I wasted a lot of time today. In the future I will resign to calling Apple Care, because at least I can work while on speakerphone. It does suck that all these batteries are proprietary so there's generally one place to get them taken care of.

I'm going to see how long I can hold off on buying any new apple stuff. I'd rather not shell out $50 to apple for a keyboard, but it's been the most comfortable keyboard I've used (I have one at work).

Update: There's a lot of fanboys out there, so I guess I need to break it down. For a process that should take 5 minutes (me bringing in a battery and getting a replacement) it took over a week of waiting and additionally 2 hours on two separate days to actually assess that I need a new battery and to finally get it in my hand. This is incompetence on Apple's part. This post serves as an alert for people who actually value their time.

py vs php: stemming

2008-02-16T00:00:00-08:00

I've been porting some PHP to python during SuperHappyDevHouse and was amazed at how little code I needed to write since python makes list manipulation a breeze.

Today I was working on stemming (ala Porter Stemming algorithm). reviewsby.us uses stemming in the search engine to make queries:

Stemming turns hello everybody how are you guy's into a collection 'everybodi', 'gui', 'hello'. To produce this in php I do the following:

With some magic python:

The magic here is list mappings. Learning about them, they don't seem that great, but as soon as you start coding you stop using a lot of for loops.

I'm sure my PHP can be cleaned up and reduced as well, but its fun exploiting the magic of languages.

unix timestamp in python

2008-02-14T00:00:00-08:00

I spent far too much time learning that:

gives me the nice numeric representation of the unix timestamp.

iPhone and MacBook Air

2008-02-13T00:00:00-08:00

I had to wait quite some time at the Apple Store only to find out my MacBook Pro had a battery that had been recalled.

So while Apple was wasting my time, I looked at the Macbook Air and iPhone. Yes, this is my blog where I'll give my un-expert opinion on gadgets that people have already written about at length.

iPhone

First the bad news. I think I have fat fingers. I finally "got" how it's supposed to work, but keyboard interface and the gesturing was not to my liking.

If someone gave me an iPhone, I definitely wouldn't mind, and I'd probably grow to like it. But at the moment... It really doesn't do much for me. I like that I could access the web, and play with Google Maps, but the interface just doesn't click for me.

I do want a smart phone, I think there's a few nice alternatives out there. I think I can wait on the iPhone. But then again, maybe I don't really need a new toy.

Macbook Air

So here's the good news. Despite it's lack of things, it really seems appealing. My biggest problem with my laptop is when I take it somewhere. Granted I bike with my Y! laptop short distances almost daily, but every now and then, I am toting my personal laptop, and the weight starts to drag me down.

It seemed reasonably fast, although with little effort I can bring any machine to a grinding halt. I think if one of our laptops go out, we might replace it with a MacBook Air. Although... both my MacBook Pro, and Katie's Powerbook are pretty sturdy.

Redo The Web

2008-02-12T00:00:00-08:00

Sometimes it's nice to read a blog the "normal" way. You go to the site and just read the entries in reverse chronological order ... versus the RSS-crazy aggregated with everything else.

François Zaninotto's Redo The Web is a great blog for that if you do any work with symfony or web development in general. François wrote most of the documentation for symfony including The Definitive Guide to Symfony and his blog manages to keep that same easy-to-follow tips and tutorials.

Definitely check it out.

Keep a second web server around for luck...

2008-02-12T00:00:00-08:00

I had one of those mid-day "what's going on with my server" heart-atacks. I have a service that emails me when reviewsby.us is down. On my old server if it went down, I could just restart the server and it'd be back up. That was big old apache, running out of memory or something.

Reviewsby.us is a medium sized site. It gets a fair amount of traffic at a steady pace. Even in this case I decided I was in need for a new server, so I looked into nginx. It's fast and it can serve static content well and pass things to fastcgi. Joshua Schachter explains the proxy in front concept pretty well.

Back to my web developer heart attack...

Well this setup had been holding up for a better part of a month fairly well... then I saw that a lot of the pages just lagged. I restarted fastcgi and nginx (it was a fastcgi issue). Rather than try to debug something I couldn't, I quickly installed apache2 and setup the server the tried and tested way.

This all took place in a half an hour. Not the end of the world, but not elegant either. In the future, I'll revert to using nginx (possibly nginx+apache versus nginx+fastcgi) but I'll keep my other configurations around when all hell breaks loose.

Unicode or How to deal with f'd up text in Django

2008-02-11T00:00:00-08:00

So I've been going out of my mind trying to figure out why something like:

Pham's would look like Phamâ€™s

After searching as much as I could into django and unicode, I found this article which worked.

As near as my feeble USA-centric "cafes not cafés" understood it, my data was encoded incorrectly, and PHP's mysql ignores encoding and expects UTF-8 whereas django tries to understand the encoding.

the magic of django: get_callable

2008-02-08T00:00:00-08:00

symfony is an amazing framework simply because of the uphill battle it has with PHP. There's a lot of behind the scenes magic that ameliorates the frustrations one might have with PHPs lack of syntactical magic.

A lot of work is done to make sure certain things are autoloaded and that certain things behave the way you expect.

I hadn't had a chance to peek under Django's hood too much myself, but it appears that despite the inherent magic of python, there were some much needed additions. My savior for today was get_callable.

I have a very particular vision of interacting with OpenID. Basically I like to decouple as much of the OpenID interactions away from the inner workings of my web site. I do, however, want a hook where I can supply some of my own logic after OpenID has been verified.

It's roughly how I implimented the sfOpenIDPlugin. Luckily this time, I was able to rely on the OpenID 2.1 libraries from OpenIDEnabled.

I did ditch their example code, because I wanted a sexy package. The package isn't quite ready for primetime, but you can have a gander (and yes, I do welcome collaborators and am open to merging my work with the myriad of work that's out there).

Getting to the point

Anyway, get_callable is what allowed me to write my "hook" into my OpenID logic.

In my settings.py I defined:

OPENID_SUCCESS = 'myproject.myapp.views.openid_handler'

The way my OpenID library works, is it does it's business and when it verifies a url belongs to someone it then delivers he request to the OPENID_SUCCESS hook. Here's how:

if settings.OPENID_SUCCESS:
    view = get_callable(settings.OPENID_SUCCESS)
    return view(request, result)

get_callable works some python magic by splitting myproject.myapp.views.openid_handler into the module: myproject.myapp.views and the attribute openid_handler. It uses this to produce a callable object. In the example above I gave it parameters of request and result as they were required by my app.

sfGuardUser to django.contrib.auth.models.User

2008-02-07T00:00:00-08:00

Let's pretend that your assignment was to convert a symfony app that used sfGuardAuth to a django-based system. Wouldn't it be great if someone just gave you a bunch of SQL that you could use to convert the symfony sf_guard_user table to a django auth_user table?

Here you go:

REPLACE INTO 
  auth_user

SELECT 
  p.id AS id, 
  replace(username, 'http://', '') AS username,
  null AS first_name,
  null AS last_name,
  email,
  concat(algorithm,'$',salt,'$',password),
  0 as is_staff,
  1 as is_active,
  0 as is_superuser, 
  last_login,
  u.created_at as date_joined


FROM 
  `sf_guard_user` u, profile p 

WHERE
  p.userid = u.id

rsnapshot to the rescue

2008-02-05T00:00:00-08:00

Sometimes I wonder what convinced me to buy a ReadyNAS and use rsnapshot to create incremental backups.

Oh yeah, I know, because when I, like an idiot, accidentally hit "Remove" in svnX and realize there's no undo... I've got options.

SVN revert, and then open up the file back in TextMate, find today's daily snapshot and drag it onto my editor window. Verify everything and save and commit.

I like that my past-self is friends with my current-self.

Dear Bank

2008-02-04T00:00:00-08:00

I can't do simple things because they inevitably make me ranty.

Dear Small Bank in MN,

Your online banking system is difficult to log into and yet insecure.

I have to jump through so many hurtles trying to log into the [Small Bank in MN] eBiz > bank, yet it still isn't that secure.

Instead of entering in my password, it's easier to just reset my password... but the scary thing is... resetting my password means you email me my old password in the clear! Instead of just providing a temporary link, you give me my old password. That's crazy! That means anybody snooping into my email could find it out. Or worse, someone could steal my computer and have all my online banking info.

My bank information should be secure even when my email gets compromised. And it shouldn't be so hard to log into. Look at ING, they have a sufficiently secure and accessible system.

Also... your HTML is inaccessible. checkboxes and radiobuttons should have <label> tags.

Sincerely,

Dave Dash

I hate online banking. It's so frustrating. Each company has a weird set of rules on login ids:

A random Number that they provide
A username which you choose must have at least two numbers

I get it... but really? Is it all that necessary. All it means for me is I only check those bank accounts when I have to, or I write those account numbers in an easy to remember place, like on the palm of my hand. Or it means I call customer support like an idiot and waste more money and time.

Then there's the magic questions:

Where were you born?
Who's your daddy?
Who's your mommy?
Which of your two cats do you like better?
What's your best friends name? (I know "friends" should be "friend's" but that's not what the bank knows)
What is the date that your second girlfriend started holding your hand?

Okay... less annoying, but seriously, how many times do I have to answer them to log into my account?

And then finally passwords:

Passwords must contain numbers, letters, bears, tigers
Passwords cannot be any of the passwords you've used in the last 9 months
Passwords cannot be your daughter's name

And then the inevitable... I give up and click forgot password... and without question it emails my email account with the password I couldn't remember... no questions asked... just a simple request to change my password again.

Um... yeah that's real secure. Thanks Bank!

Oh and never mind that their forms are hard to use because they don't know proper HTML. If you don't use <label>s it's like trying to pee in a Cheerio using your mouse.

Django circular model references

2008-02-03T00:00:00-08:00

I'm used to circular references in my model. Often I do a versioning of an Item with an ItemVersion. Item will link to the latest ItemVersion and ItemVersion will link to the relevant Item.

Here's how you can define your appropriate django models:

Note in the first model, Item, we reference ItemVersion in quotes because ItemVersion is not yet defined.

Why Django Templating is awesome and why I get smarty again

2008-02-01T00:00:00-08:00

I get Smarty thanks to django... yeah, it's weird.

Back to my original comment about templating, smarty really is trying to limit the scope of PHP in a good way. Too often I see a lot of heavy-lifting in the templates. It's so bad it makes my MVC'd brain explode.

Django templates are very limited, based on a philosophy of keeping view logic and only view logic in the templates.

This is what smart tries to do and it's a reasonable solution to the fact that PHP is a templating language with a kitchen sink. It's saying, okay... well let's treat PHP as a programming language, and keep Smarty as the template.

symfony of course says (well not really, or in any official capacity, but would you believe... frameworks talk to me), some PHP is view and some is model/controller. We'll suggest which ones are which, but really we're not getting in the way of making your life a living hell by sticking complicated business logic inline.

Girl Geek Dinner

2008-02-01T00:00:00-08:00

Katie and I went to Girl Geek Dinner. The best thing about being a Geek Husband is having a Geek Wife to bring you as a guest.

The panel talk was pretty good. Featuring:

Irene Au, Director of User Experience, Google
Rashmi Sinha, CEO, SlideShare
Leah Culver, Lead Developer & Co-Founder, Pownce
Sumaya Kazi, Entrepreneur & Social Media Manager, Sun Microsystems
Katherine Barr (moderator), Partner, Mohr Davidow Ventures

What really made the talk was it had a lot of good advice in general. Writing in a blog for example is how a lot of people got their credibility. It's how I got mine, and why I now live in the bay area and ultimately why I work on del.icio.us.

The talk was really reaffirming of my move out here. It also makes me want to code-like-mad.

Following the Django theme... Pownce was written in Django ;)

django experiment: day 2 templates, and that's it?

2008-01-28T00:00:00-08:00

[tags]django, symfony[/tags]

[Note: This really does have something to do with symfony.]

So I finished the tutorials on Django, but the Django tutorial is nothing compared to the Askeet Advent Calendar. Askeet was 24 one hour tutorials that go in depth to each nook and cranny of symfony, whereas I still feel like there's got to be more to Django.

I get Templating now, and I really like the flexibility offered in Django. The one major difference is the layout structure.

symfony has this nice structure of defining a general layout, and then all following templates automatically inherit from the default layout (unless otherwise specified). Django on the other hand requires you to explicitly extend other layouts and define regions within those layouts.

I like the symfony style because I inherit almost always from the general template.
The Django style however offers the flexibility of having subsections (which inherit from the base style) which you can inherit from. E.g. you may have a site which has 1. a blog and 2. a gallery. Both the blog and gallery sections share common elements from a main template, but all the pages within blog all have some unique bloggish features that the gallery lacks.

I've got a lot more to learn with Django, but I am enjoying it so far.

I...

~~still have to learn how to create templates.~~
~~want better understanding of the available field types.~~
want to find a good CMS package
want to know if there's the idea of environments in Django (like in symfony)
want to override .save() on my models
want to learn how to have self-referencing models (E.g. a hierarchical category)

Serving Firefox xpi extensions over ssl/https

2008-01-22T00:00:00-08:00

[tags]firefox, cache, xpi, https[/tags]

Apparently serving XPI extensions over HTTPS is a headache. What happens is you may have a link to an XPI served over HTTPS, but Firefox will ask intercept the download and ask you to add your domain to the allowed hosts. After allowing it, you can't re-download the file (it gets cached, incorrectly, even if you request it to not do so).

So the workaround is this:

Don't link directly to the XPI file. Link to an action (e.g. /getxpi). Have this action link to the actual XPI file with a random GET parameter, e.g. https://mydomain.com/xpi/myextension.xpi?ts=123456.

This will result in any XPI downloaded from /getxpi to not be cached.

This is a cruddy work-around for a problem with Firefox, but unfortunately I'm not a XUL hacker.

-d

nginx and 'www' free urls

2008-01-19T00:00:00-08:00

[tags]nginx, rewrite[/tags]

I'm not a fan of 'www' in urls and I've been flirting with nginx (pronounced Engine-X as much as I want to call it N-jinx).

Here's a one-liner that'll rewrite your urls:

# rewrite hostname
server {
  server_name www. mydomain.com;
  rewrite ^/(.*) http://mydomain.com/$1 permanent;
}

I put this at the end of the definition for mydomain.com.

ReadyNAS redux

2008-01-04T00:00:00-08:00

[tags]infrant, readynas, nv+, redux, harddrive, crash, gear[/tags]

So... before I left for MSP (I live in Silicon Valley, btw) my solitary disk in my Infrant ReadyNAS NV+ crashed... damn.

I lost backups (you know the backups I like to take just in case something happens to my laptop while in transit) and some non-essentials, like all the music I used to have.

So, I ordered some more drives and returned my old one to Newegg.com (I had to pay the shipping).

The upside is I got to learn the lesson fast and make sure everything is redundant right away. I have 3 250GB drives giving me almost half a gig of redundant storage, plus I have a replacement 500GB drive that'll add some more.

I worry that I may have been hammering the drive by doing a lot of backups all at once, but I'll give it another try. I figure if another drive crashes, I can get it replaced fairly easily.

ReadyNAS is awesome, Netgear is not so awesome

2007-12-14T00:00:00-08:00

After much debate I decided to get the ReadyNAS NV+ with a single 500GB disk. It's great so far. I even got ssh to work which means I have rsnapshot working (I can post details if necessary).

The problem is... I wouldn't have needed rsnapshot at all if the CD they shipped me wasn't cracked. I called Netgear (who is headquartered down the road, literally) and they said they can't do anything and that I should call Newegg (the vendor) or call EMC the provider of the backup software (and my next door neighbor at work). Newegg could only give me a $25 credit which is fine, but as someone at my work pointed out... that's not the same as a 5 seat Mac/Windows license for a backup software.

Of course... it's impossible to hunt down an actual support email address for Netgear, so this might not get resolved.

FYI, Here's some good links about readyNAS that I've found.

Affordable IM/Phone-based consulting

2007-08-20T00:00:00-07:00

[tags]symfony, spindrop, services, consulting[/tags]

I'm now available for chat-based consulting. If you are stuck in a rut and find yourself spending a lot of time in chat rooms or on forums to no avail trying to understand symfony, php, or javascript (AJAX), this is the solution for you.

I can give you expert help on best practices and get you out of snags quickly. The types of problems I can help you solve are:

symfony/PHP white-screens-of-death
other 404/500 errors
symfony exceptions
best practices
apache configuration
AJAX/Javascript questions

I'm available by chat, or email or phone during pre-arranged times. I can even remote into your server via VNC and show you (literally) how to fix your problems.

If you're interested in this service please contact me.

Affordable IM/Phone-based consulting

2007-08-20T00:00:00-07:00

[tags]symfony, spindrop, services, consulting[/tags]

I can give you expert help on best practices and get you out of snags quickly. The types of problems I can help you solve are:

symfony/PHP white-screens-of-death
other 404/500 errors
symfony exceptions
best practices
apache configuration
AJAX/Javascript questions

I'm available by chat, or email or phone during pre-arranged times. I can even remote into your server via VNC and show you (literally) how to fix your problems.

If you're interested in this service please contact me.

Symfony Camp: Ajax and Zend, what would you like to know?

2007-08-16T00:00:00-07:00

[tags]symfonyCamp, symfony, netherlands, ajax, zend search lucene, zsl, jquery[/tags]

I've been asked to speak at SymfonyCamp (symfony['camp']) next month (you should all go if you can) and I thought I'd present as well as I could on Ajax and the Zend Framework Bridge (including Zend Search Lucene).

If you're attending the camp and/or would like to hear about these topics please let me know any specific questions you might have about "symfony and Ajax" and "symfony and Zend" and I'll try to address them in my presentations.

If you are unable to go fear not, I'll try to post my notes on this site.

symfony and the .htaccess file

2007-08-15T00:00:00-07:00

One performance boost that can be garnered from a symfony app (or any app for that matter) is disabling .htaccess. .htaccess does not need to be parsed on each visit to your app. Disabling .htaccess is trivial in your VirtualHost or another relevant part of your apache configuration place:

AllowOverride None

symfony does require the Rewrite recipes in .htaccess to properly give you pretty urls, so place those statements right after AllowOverride None.

In my specific case I used this:

<VirtualHost *>
  ServerName contest.spindrop.sf
  DocumentRoot    /Users/davedash/Sites/spindrop/web/
  DirectoryIndex  frontend_dev.php

  <Directory /Users/davedash/Sites/spindrop/web/>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    <IfModule mod_rewrite.c>
      RewriteEngine On

      RewriteCond %{REQUEST_URI} \..+$
      RewriteCond %{REQUEST_URI} !\.html$
      RewriteRule .* - [L]

      RewriteRule ^$ index.html [QSA]
      RewriteRule ^([^.]+)$ $1.html [QSA]
      RewriteCond %{REQUEST_FILENAME} !-f

      RewriteRule ^(.*)$ frontend_dev.php [QSA,L]
    </IfModule>
  </Directory>
</VirtualHost>

Now all these rewriting rules are loaded when the server is restarted and the .htaccess is not examined upon each request.

FBML and CSS background images

2007-08-11T00:00:00-07:00

I've been tediously building out the new reviewsby.us app for Facebook and I ran into some problem when I tried to embed this CSS:

The CSS worked fine, just now background image. The fix is mentioned in this post which is to repeat the background statement:

Voila, it works. Hopefully this hack won't be necessary in later versions of the Platform.

doctine and getState()

2007-07-10T00:00:00-07:00

[tags]doctrine, php, symfony, sfDoctrine, database,errors[/tags]

I tend to have models with a field called state. Doctrine offers a few ways of getting to the state field:

$obj->get('state');
$obj['state'];

$obj->getState() however conflicts with Doctrine_Record::getState() from which all objects inherit. Use one of the above alternatives.

TextMate + YUI = YUI snippets!

2007-04-30T00:00:00-07:00

I do a lot of YUI grid layouts and I love the nestable grids:

<div class="yui-g$1"> 
    <div class="yui-u first">
        $2
    </div>
    <div class="yui-u">
        $3
    </div>
</div>

There's a tab stop after yui-g in case you want to use one of the variants (yui-gb, yui-gc, etc).

I'm working on a site that uses two equal width columns... a lot... so this comes in quite handy. So long tables.

Creative Friends

2007-04-10T00:00:00-07:00

[tags]photography, design, pretty[/tags]

A college friend of mine swims some creative waters:

corvail photographic agency has some interesting. This picture was pretty awesome. If we could get people to upload delicious photos like that for reviewsBy.us... we'd all be fat and broke ;)
The full screen pictures on striel were also fun. I especially liked 8/2005, 10/2005, 11/2005, 2/2006, 5/2006 and 3/2007.
Grasshopper is pretty... pretty awesome!

Finding a co-founder

2007-04-05T00:00:00-07:00

[tags]startups[/tags] A friend of mine directed me to Why to Not Not Start a Startup:

Not having a cofounder is a real problem. A startup is too much for one person to bear. And though we differ from other investors on a lot of questions, we all agree on this. All investors, without exception, are more likely to fund you with a cofounder than without.

This is probably the single largest road-block in the way of starting a start-up. One solution is to move to where there are more interested parties (e.g. Silicon Valley), but that's not always an easy option (although I'm warming up to it, Minneapolis isn't exactly heaven for several months out of the year).

Another option is to find someone online. Mike Subelsky created Founder Finder as an attempt at addressing this very issue.

Unfortunately, deep down inside I think that is a difficult proposition as well. I have a hard time finding good programmers, I've worked as a one-man island most of my life, and haven't maintained good contact with my fellow CS major's from UIUC.

Believe it or not, social networking is helping in some ways, because I'm able to reconnect with people I knew at one time or another who are very much start-up driven and really have the drive.

A better network of developers needs to exist, not just people we scrape out of monster.com and jobster... how's that for a cheap idea?

MarsEdit

2007-02-24T00:00:00-08:00

[tags]MarsEdit,software,blogs[/tags]

I recently read about the acquisition of MarsEdit by Red Sweater Software so I decided to check it out. I'm quite glad I did it does make my workflow a lot easier. What I had been doing up until today was this:

I'd come up with an idea.
Open up my writings project in TextMate
Open a new file and start typing ideas, etc.
Find the right blog to post it on (e.g. Spindrop, davedash.com, yumbo the reviewsBy.us blog or Metroblogging).
Login to the blog.
Cut and paste titles, categories, tags, etc.

It was easy to drop my blogging habbit ;)

Now, I can do this: 1. Come up with my idea. 2. Select which blog I want to post it in. 3. Write my idea. 4. When it's ready to go live, just hit submit to blog.

The one slight problem I have is I use a tags feature of wordpress in addition to categories (not sure why I don't just use categories, but that's another story), currently there's no easy way to deal with WordPress plugins' custom fields that I know of. So there's an optional 5th step of examining the post on the site and possibly adding tags.

Still, these 4 steps are easy and all done from one app. It also features a preview of your post that you can customize per blog with a custom HTML template. Which makes it easy to just drop your blog's style sheet and see if your post looks right.

Definitely looks like a keeepr!

Vim Undo

2006-09-20T00:00:00-07:00

So apparently you can "time-travel" in vim. Commands like :earlier 10m and :later 5m will move your document to points 10 minutes ago and from their 5 minutes after.

Is it just me, or is this acceptable?

vim phd_thesis.txt

:later 5y

Cropping Images using DHTML (Prototype) and symfony

2006-09-16T00:00:00-07:00

Note: Like many of my tutorials, you don't need symfony, just PHP. However, I develop in symfony and take advantage of the MVC-support that it offers.

Years ago when I was working on a photo gallery for davedash.com I got the art of making tumbnails down fairly well. It was automated and didn't allow for specifying how the thumbnail should be made. With dozens of photos (which was a lot back then), when would I find that kind of time.

Flashback to today, for my company... we want users with avatars... but nothing too large. Maybe a nice 80x80 picture. Well the coolest UI I've seen was Apple's Address Book which let you use this slider mechanism to crop a fixed sized image from a larger image.

Here's a demo.

Overview

The front-end GUI is based on code from digg which is based on the look and feel (as near as I can tell) from Apple.

The GUI provides a clever visual way of telling the server how to chop the image. The gist is this, sliding the image around and zooming in and out change a few form values that get passed to another script which uses this data to produce the image.

Frontend: What would you like to crop?

In this tutorial, we're going to be cropping an 80x80 avatar from an uploaded image. The front-end requires the correct mix of Javascript, CSS, HTML and images. The Javascript sets up the initial placements of the image and the controls. The CSS presents some necessary styling. The images makeup some of the controls. The HTML glues everything together.

HTML

Let's work on our HTML first. Since I used symfony, I created a crop action for a userpics module. So in our cropSuccess.php template:

<div id="ava">
    <?php echo form_tag("userpics/crop") ?>
        <div id="ava_img">
            <div id="ava_overlay"></div>
            <div id="ava_drager"></div>
            <img src="<?php echo $image ?>" id="avatar" />
        </div>
        <div id="ava_slider"><div id="ava_handle"></div></div>
        <input type="hidden" id="ava_width" name="width" value="80" />
        <input type="hidden" id="ava_x" name="x" value="100" />
        <input type="hidden" id="ava_y" name="y" value="100" />
        <input type="hidden" id="ava_image" name="file" value="<?php echo $image ?>" />
        </div>
        <input type="submit" name="submit" id="ava_submit" value="Crop" style="width: auto; font-size: 105%; font-weight: bold; margin: 1em 0;" />
    </form>
</div>

Right now a lot of this doesn't quite make sense. If you attempt to render it, you will just see only the image. As we add the corresponding CSS and images it will make some more sense.

CSS and corresponding images

We'll go through each style individually and explain what purpose it serves in terms of the GUI.

#ava is our container.

#ava {
    border: 1px solid gray;
     width: 200px;
}

#ava_img is the area that contains our image. Our window for editing this image 200x200 pixels. If we drag out image out of bounds we just want the overflowing image to be clipped. We also want our position to be relative so any child elements can be positioned absolutely with respect to #ava_img.

#ava_img {
    width: 200px;
    height: 200px;
    overflow: hidden;
    position: relative;
}

#ava_overlay is a window we use to see what exactly will be our avatar. If it's in the small 80x80 window in the center of the image, then it's part of the avatar. If it's in the fuzzy region, then it's getting cropped out. This overlay of course needs to be positioned absolutely.

#ava_overlay {
    width: 200px;
    height: 200px;
    position: absolute;
    top: 0px;
    left: 0px;
    background: url('/images/overlay.png');
    z-index: 50;
}

#ava_drager is probably the least intuitive element (Heck, I'm not even sure if I've even got it right). In our demo you're not actually dragging the image, because you can drag anywhere within the #ava_img container and move the image around. You're using dragging an invisible handle. It's a 400x400 pixel square that can be dragged all over the container and thusly move the image as needed.

#ava_drager {
    width: 400px;
    height: 400px;
    position: absolute;
    z-index: 100;
    color: #fff;
    cursor: move;
}

#avatar is our image, and since it will be moving all around the window, it requires absolute positioning.

#avatar {
    position: absolute;
}

#ava_slider and #ava_handle are our slider components. They should be self-explanatory.

#ava_slider {
    width: 200px;
    height: 27px;
    background: #eee;
    position: relative;
    border-top: 1px solid gray; 
    background: url('/images/slider_back.png');
}
#ava_handle {
    width: 19px;
    height: 20px;
    background: blue;
    position: absolute;
    background: url('/images/handle.png');
}

Internet Explorer

PNG do not work so well in Internet Explorer, but there is a small trick, adding these components into a style sheet that only IE can read will make things work:

#ava_overlay {
  background: none;
  filter:progid:DXImageTransform.Microsoft.AlphaImageLoader(src='/images/ui/cropper/overlay.png', sizingMethod='crop');
}

#ava_handle {
  background: none;
  filter:progid:DXImageTransform.Microsoft.AlphaImageLoader(src='/images/ui/cropper/handle.png', sizingMethod='crop');
}

The Javascript

The Javascript is actually not as complicated as you'd expect thanks to the wonder of prototype. This framework provides so much so easily. You'll need to include prototype.js and dom-drag.js.

So let's take a look.

<script type="text/javascript" language="javascript" charset="utf-8">
// <![CDATA[
function setupAva() {
    if ($("avatar")) {
        var handle = $("ava_handle");
        var avatar = $("avatar");
        var drager = $("ava_drager");
        var slider = $("ava_slider");
        var ava_width = $("ava_width");
        var ava_x = $("ava_x");
        var ava_y = $("ava_y");
        // four numbers are minx, maxx, miny, maxy
        Drag.init(handle, null, 0, 134, 0, 0);
        Drag.init(drager, avatar, -100, 350, -100, 350);
        var start_w = avatar.width;
        var start_h = avatar.height;
        var ratio = (start_h / start_w);
        var new_h;
        var new_w;
        if (ratio > 1) {
            new_w = 80;
            new_h = (80*start_h)/start_w;
        } else {
            new_h = 80;
            new_w = (80*start_w)/start_h;
        }
        // these need to be set after we init
        avatar.style.top = '100px';
        avatar.style.left = '100px';
        avatar.style.width = new_w + 'px';
        avatar.style.height = new_h + 'px';
        avatar.style.margin = '-' + (new_h / 2) + 'px 0 0 -' + (new_w / 2) + 'px';
        handle.style.margin = '3px 0 0 20px';
        avatar.onDrag = function(x, y) {
            ava_x.value = x;
            ava_y.value = y;
        }
        handle.onDrag = function(x, y) {
            var n_width = (new_w + (x * 2));
            var n_height = (new_h + ((x * 2) * ratio));         
            avatar.style.width = n_width + 'px';
            avatar.style.height = n_height+ 'px';
            ava_width.value = n_width;  
            avatar.style.margin = '-' + (n_height / 2) + 'px 0 0 -' + (n_width / 2) + 'px';
        }
    }
}
Event.observe(window,'load',setupAva, false);
// ]]>
</script>

If this isn't exactly crystal clear, I can explain. If you're new to prototype, $() is the same as doucment.getElementByID() (at least for our purposes).

We need to initialize two draggable elements, one is our slider for zooming and the other is our avatar itself. We initialize the draggers using Drag.init(). We specify what to drag, if another element should be used as a handle and then the range of motion in xy coordinates. In the second call we use that #dragger to move around the image in this manner.

Drag.init(handle, null, 0, 134, 0, 0);
Drag.init(drager, avatar, -100, 350, -100, 350);

We want to initialize the the size and placement of the avatar. We do that using maths. First we want it in our 80x80 pixel box. So it should be roughly 80x80. I've set the math up so that the smallest side is 80 pixels (there's reasons for doing this the other way around).

    if (ratio > 1) {
        new_w = 80;
        new_h = (80*start_h)/start_w;
    } else {
        new_h = 80;
        new_w = (80*start_w)/start_h;
    }

We then place the avatar element. We initialize it to be in the center of the screen (top: 100px;left:100px) and then nudge the image using margins.

    avatar.style.top = '100px';
    avatar.style.left = '100px';
    avatar.style.width = new_w + 'px';
    avatar.style.height = new_h + 'px';
    avatar.style.margin = '-' + (new_h / 2) + 'px 0 0 -' + (new_w / 2) + 'px';

We also use margins to place the handle.

    handle.style.margin = '3px 0 0 20px';

#ava_x and #ava_y tell us where the center of the avatar is. So when the avatar is moved we need to set these again:

    avatar.onDrag = function(x, y) {
        ava_x.value = x;
        ava_y.value = y;
    }

That was easy. Slighly more complicated is the zoomer function. We are basically adjusting the width and the height proportionately based on roughly where the slider is. Note that we're still using that ratio variable that we calculated earlier. We basically take the new x-coordinate of the handle and allow our image to get just slightly larger than the #ava_image container.

    handle.onDrag = function(x, y) {
        var n_width = (new_w + (x * 2));
        var n_height = (new_h + ((x * 2) * ratio));         
        avatar.style.width = n_width + 'px';
        avatar.style.height = n_height+ 'px';
        ava_width.value = n_width;  
        avatar.style.margin = '-' + (n_height / 2) + 'px 0 0 -' + (n_width / 2) + 'px';
    }

We want to load initialize the slider right away when the page loads: Event.observe(window,'load',setupAva, false);

Not terribly hard or complicated. Once these elements are all in place you have a working functioning slider. It returns the x and y coordinates of the center of the image with respect to our 200x200 pixel #ava_image. It also tells us the new width of our image. We feed this information into a new script and out should pop a new image which matches exactly what we see in our GUI.

Processing the crop

Initially I was frustrated with the data that was being sent. I knew the center of the image in relation to this 200x200 pixel canvas and its width... but what could I do with that. Well I could just recreate what I saw in the GUI. I needed to create a 200x200 pixel image first, place my original avatar resized (and resampled) at the precise coordinates and then cut out the center most 80x80 pixels to become the final avatar image.

If you note in our template above for cropSuccess.php we submit our form back to the crop action. Let's look at the action:

public function executeCrop()
{
    if ($this->getRequestParameter('file')&&$this->getRequestParameter('width')) {          // we are saving our cropped image
        // Load the original avatar into a GD image so we can manipulate it with GD
        $o_filename = $this->getRequestParameter('file');  // we'll use this to find the file on our system
        $o_filename = sfConfig::get('sf_root_dir').'/web' . $o_filename;
        $o_im = @imagecreatetruecolor(80, 80) or die("Cannot Initialize new GD image stream");
        $o_imagetype = exif_imagetype($o_filename); // is this gif/jpeg/png

        // appropriately create the GD image
        switch ($o_imagetype) {
            case 1: // gif
                $o_im = imagecreatefromgif($o_filename);
                break;
            case 2: // jpeg
                $o_im = imagecreatefromjpeg($o_filename);   
                break;
            case 3: // png
                $o_im = imagecreatefrompng($o_filename);
                break;
        }

        // Let's create our canvas
        $im = @imagecreatetruecolor(200, 200) or die("Cannot Initialize new GD image stream");
        imagecolortransparent ( $im, 127 ); // set the transparency color to 127
        imagefilledrectangle( $im, 0, 0, 200, 200, 127 ); // fill the canvas with a transparent rectangle

        // let's get the new dimension for our image

        $new_width = $this->getRequestParameter('width');
        $o_width = imageSX($o_im);
        $o_height = imageSY($o_im);

        $new_height = $o_height/$o_width * $new_width;

        // we place the image at the xy coordinate and then shift it so that the image is now centered at the xy coordinate
        $x = $this->getRequestParameter('x') - $new_width/2;
        $y = $this->getRequestParameter('y') - $new_height/2;

        // copy the original image resized and resampled onto the canvas
        imagecopyresampled($im,$o_im,$x,$y,0,0,$new_width,$new_height,$o_width,$o_height); 
        imagedestroy($o_im);

        // $final will be our final image, we will chop $im and take out the 80x80 center
        $final = @imagecreatetruecolor(80, 80) or die("Cannot Initialize new GD image stream");
        imagecolortransparent ( $final, 127 ); // maintain transparency

        //copy the center of our original image and store it here
        imagecopyresampled ( $final, $im, 0, 0, 60, 60, 80, 80, 80, 80 );
        imagedestroy($im);

        //save our new user pic
        $p = new Userpic();
        $p->setUser($this->getUser()->getUser());
        $p->setGD2($final);
        $p->save();
        imagedestroy($final);
        $this->userpic = $p;
        return "Finished";
    }


    $this->getResponse()->addJavascript("dom-drag");
    $this->getResponse()->addJavascript('/sf/js/prototype/prototype');
    $this->getResponse()->addJavascript('/sf/js/prototype/effects');
    $this->image = '/images/userpics/originals/' . $this->getRequestParameter('file');
}

It's doing exactly what the paragraph above explains when the image dimensions are given. The code is well commented so it should be easy enough to follow.

GD image functions in PHP are fairly robust and can help you do a lot of tricks with image data. Note the code to save the image, we'll cover it in detail soon.

The Model

$p = new Userpic();
$p->setUser($this->getUser()->getUser());
$p->setGD2($final);
$p->save();

First some clarification the second line. myUser::getUser() gets the User object associated with the currently logged in user. The third line, however, is where the magic happens. Before we look at it, let's have a quick look at our model:

userpic:
 _attributes: { phpName: Userpic }
 id:
 user_id:
 image: blob
 thumb: blob
 created_at:
 updated_at:

We have an image attribute and a thumb property to our Userpic object. This is where we store PNG versions of each icon and their 16x16 thumbnails respectively. We do this in Userpic::setGD2():

public function setGD2($gd2_image)
{
    //convert to PNG
    ob_start();
    imagepng($gd2_image);
    $png = ob_get_clean();
    //save 16x16
    $gd2_tn = @imagecreatetruecolor(16, 16) or die("Cannot Initialize new GD image stream");
    imagealphablending( $gd2_tn, true );
    imagecolortransparent ( $gd2_tn, 127 );

    imagecopyresampled ( $gd2_tn, $gd2_image, 0, 0, 0, 0, 16, 16, 80, 80 );
    ob_start();
    imagepng($gd2_tn);
    $tn = ob_get_clean();

    $this->setImage($png);
    $this->setThumb($tn);
}

We capture the output of the full size PNG, then we scale it again and capture the output of the thumbnail and set them.

Conclusion

When it comes to web apps, having a relatively simple GUI for people to resize images can go a long way in terms of adoption rate of avatars and custom user pictures by non technical users.

Enjoy, and if you found this useful (or better implemented it) let me know.

Restaurants can be (tagged) spicy too!

2006-07-27T00:00:00-07:00

We had a change of heart on tagging. Tag everything! Initially I felt that only menu items should be tagged. "People want to know what dishes are spicy"... true, but people also want to know what restaurants are in Uptown or are Mexican or are closed on mondays. So... we now have restaurant tags. Login and tag something!

Menu item versions

2006-07-24T00:00:00-07:00

We make a lot of mistakes, and by we, I don’t mean myself, but I mean we as humanity. That’s why we like to keep multiple versions of things in our restaurant review database. That also means menu items. Did you describe the Pesto Chicken as having pisto and not pesto? Problem solved, now any user can log in and change it. It’s like a wiki…. for restaurant menus.

Safari Fixes

2006-07-11T00:00:00-07:00

Safari interprets /* */s differently than FireFox or IE. FF and IE will ignore a unmatched /* or */, whereas Safari will ignore parts of code if there's a lone */. Once I found that out, I was able to get the list items that are used throughout the site to render properly.

Spindrop objectives

2006-06-05T00:00:00-07:00

I had wanted to write about this later, but Darren at ProBlogger started this group writing project on "Blog Goals." So I am jumping in quickly. This blog was created for two reasons, one, to document any technical things that I've learned, code samples, best practices, strategies, etc. as they pertain to web development and open source. The second is to serve as a site to record updates to any of my projects.

I'm still in my infancy for this blog, but I've seen a lot of little things that keep me optimistic.

The objectives I have for Spindrop are both internal and external. Internal goals are things that I can change myself. For example, the style of the site, linking to other places, posting more content, changing the way ads are presented, etc.

External goals depend on readers like you. I can do my best to make this site be relevant to a lot of people, but I can't make people click on my site, comment, or any of that. I can still make goals for them, and that will subconsciously get me to position myself better.

They are both related. If I achieve my internal objectives, I'm better suited to getting external objectives done. If I get my external objectives done, it encourages my behavior of making the site better.

What am I willing to do

My "9-5" is a fairly demanding job as a lead web developer for a health and wellness website. On top of that, I'm getting married this month, so I'm very pressed for time when it comes to a side project like this. Luckily I'm very well organized, and do wake up early and spend time writing.

Writing

In an OmniOutliner file, I keep a detailed list of what I'll be writing as well as other "to-do list" items for this site. I've already taken into account that I'm not going to be pushing anything useful for the week or two surrounding my wedding. Hopefully, I can still muster two or three articles this month. Some of those articles are on software choices, strategies for using propel and symfony and general work habbits.

By spending an hour (or more) a day on writing, I'm generally looking over things and making sure I did a halfway decent job. I prefer not to have typos, spelling errors, etc, I prefer to have a readable article, and I prefer to throw in images, when useful. Of course, I do this early in the morning, so there are mistakes. But, I've noticed a lot of heavy hit articles on del.icio.us and digg have typos too.

Non-content changes

I'm willing to tweak ad placement, and practice all the other ad-voodoo that is involved with blogging. I'm also willing to announce the site at appropriate moments, communicate with people via forums and other blogs. I'm willing to listen and execute on others advice. The best advice I've heard is to stick with it.

What can I get at the very least?

With all my efforts, at the very least, I know I'll have something to show for. For one, I tend to do things over and over again. I'm a web developer, and the bulk of my blog is about how to do things I've done before. So if I have to make another map or migrate a blog, I can see how I did it before. Now, anything else this blog achieves is gravy.

External goals

Some of my hopes for the site are out of my control. I want what a lot of other blogs want: traffic, revenue and community. Additionally I want traffic to be sent to my other sites, like the reviewsby.us site so that they can share the success.

Traffic

The most important goal for me is traffic. I want this site to get a lot of traffic. For the last two weeks, I've averaged 112/users a day. Of course, part of that is due to a spike because of last Friday's article on Editing CSS from Firefox which had an anomalous amount of hits due to getting on reddit, del.icio.us popular, digg.¹

Traffic is important because I without it I can't expect to have revenue or community. There's also a sense of validation that you have something useful to say, and sites like del.icio.us make you feel that not only was it useful, it's worth hanging on to.

Revenue

I'd like to earn enough to actually make blogging make a significant change in my lifestyle. Right now it's a hobby. A dedicated hobby, but it's still a hobby. I love writing, I love to see myself improve, I love what I write about. I'd like to continue to do that full time. I'm sure with some tweaks to adsense and some other forms of advertising, I can kick that up. But content, which attracts traffic, will be the biggest driver of them all. My less-than-"adsense optimized" restaurant review site generates a lot more money (seriously, we're talking in small terms, like the cost of an iced tea), but it has a lot of content and a lot of useful ads that get generated from adsense.

Last month (May 2006) I made just shy over $2.00 from the web site. That's not much, but it's better than $0. I'd like to make $4 in a given month. On one hand it's a doubling of earnings, but on another, it's not very much money. I'm not expecting to double the earnings this month, although I might easily do that, but I am expecting to hit that $4 mark eventually. It's an easily obtainable goal, but for me easy goals are good. I feel just as good when I hit them, as I would if I got a raise or a bonus at my "9 to 5."

Community

I'd like for there to be some level of interaction with me. I don't mean community in the LiveJournal sense of the word, I just want some interaction with me the poster, and my commenters. The last week or two has seen a few "real comments," which is promising. I hope that trend continues over time. It's an opportunity to improve myself if I get good feedback, and it's an opportunity to further express or clarify a point. I also like to help people out.

Linking out to my other sites

Lastly, I do want this site to assist my other sites. With a decent amount of traffic, this site can be a great link in to my other sites, and bring them up to speed.

Conclusion

Overall, I'm happy where I am, and I'm happy where I intend to go. Each week brings in a new useful content which is a nice record of how I do things, and proves some degree of usefulness to others.

I hope to expand on those strengths. This body of content might be small now, but it only gets bigger. My long term goal is to be able to generate the traffic and revenue I need so I can justify dedicating more time to this project.

Perhaps it's not anomalous, but if you saw my statistics you'd understand why I might think so. ↩

Dave Dash

Alphabetical sorting in Sphinx

Test Driven Confidence

Finding the most common Firefox issues

A few weeks in Chrome

Making our tests run thrice as fast

Why speed matters

Testing Sphinx search with TransactionTestCase

Making our own TransactionTestCase

Fast tests are good

λ^2: safely doing class based views in Django

λ

λ λ

django-fixture-magic: Testing issues with real data.

Serializing alone isn't enough.

Too much serial

Retrieving elements in a specific order in Django and mySQL

Palm Pré: A retraction, I really like it now

Google Chrome Extensions Puzzle

Django: Model Inheritance or Related Tables wrt AMO

Palm Pre: Always hot

AMO Search: Powered by Sphinx

A bit about addons

The technical details of addons.mozilla.org

Why search matters

Improve Search

Complex Data

Using Sphinx

The challenges

Wins

Building a platform

Getting started with pipe viewer

DjangoCon wrapup

Day one

Keynote - Avi Bryant

UR doing it wrong - James Bennet

Testing - Eric Holscher

Deploying Django -

Day 2

Keynote - Ian Bicking

Using Django in Non-standard ways - Eric Florenzano

Real-time web and other Buzzwords - Chris Wanstrath

Pluggable, Reusable Django Apps: A Use Case and Proposed Solution - Shawn Rider and Nowell Strite

Day 3

Keynote - Ted Leung - Sun

Scaling Django Mike Malone

Gearman - working later - Chris Heisel

Snow Leopard for Macports and Mysql users

Fun with HTML5, Blockchalk, Bookmarklets and Google Maps

git svn rebase... forever?

V is for Version Hell

Delicious keeps you in the know

Have unique descriptive page titles

Comprehensive list of international dialing codes

Question: Building a Better Search Engine

From Delicious to Mozilla

Reading urlopen and probably any file-ish things in python fast

Python Generators

Fixing the DiggBar with Delicious

Designing a tagging system: what is tagging

Tagging is search

Optimizing via YSlow

Begin: D - 61

YUI Combo-loading: Sneaking up to D - 62

Concatenation and Minification: Instant C- - 72

Google: They want you to fail

Content Delivery Network (CDN)

Gzip files (still at 72)

Upgrading from urchin.js (up to 73)

Move Javascript to the bottom (still at 73)

Conclusions

AppEngine is not a free CDN

Automating Django Redirects

A stitch in Fabric saves time

The task

Directory Structure

Subversion

The fabfile.py

Final Thoughts

Resizing Image on upload in Django

Testing Sphinx search with `TransactionTestCase`

Making our own `TransactionTestCase`

Upgrading from `urchin.js` (up to 73)

The `fabfile.py`

The Django implementation of `RestaurantVersion`

The Django implementation of `Restaurant`

Proxy attributes to the corresponding `RestaurantVersion` attributes

Generate a new `RestaurantVersion` on demand