Dave Dash

Trimming Whitespace in Django Forms

2010-08-18T00:00:00-07:00

I've been using frameworks for a number of years. So I expect a lot of things to happen "for free" in Django. One is whitespace removal. In Delicious we had a lot of data in our database with leading and trailing whitespace. On the frontend we moved to symfony (actually ysymfony) and that prevented a lot of this.

So I was quite surprised that this is not the case with Django. So I decided we could solve this at the form level, and released a ridiculously simple library. After some googling, I found that I was not the first to do this.

Feel free to use this, fork it, submit pull requests, etc. I suspect in the future we'll handle other global form filtering - like stripping high order Unicode since MySQL is often not a fan.

The Perils of One Giant Fixture

2010-08-12T00:00:00-07:00

A while back, I thought it would be good to consolidate all the data used in testing the django-layer of AMO into a single data fixture. Unfortunately we have 600 tests, which were now loading and unloading large amounts of data each time the test would run. This made our tests take 20 minutes.

I decided to cut this down quite a bit, by using smaller fixture files. Each fixture file attempts to be a singular primary object (e.g. an Addon or a Collection or a User) and its associated supporting objects. It's far from perfect, but it's achieved tests that run in under 10 minutes.

The other side effect is tests will be simpler. They'll only include the addons needed to generate an effect, and if something can't be done easily with the fixtures in place, we can always alter the data during the test.

The Python textcluster Package

2010-07-08T00:00:00-07:00

Earlier I wrote about finding the most common Firefox issues. I had wanted to automate that process and continually find these issues. Unfortunately I never had time to do this.

When they announced Firefox Input, I thought about doing this again... just with Firefox Input data but then I went on paternity leave and time kind of crept away. But I mentioned the idea this week and it piqued some interest.

So I found myself with a bit of time to work on it. The first stage was releasing a python library called textcluster.

textcluster takes the work I did earlier and makes it a bit more general purpose. The idea is I can do something like this:

docs = (
        'Every good boy does fine.',
        'Every good girl does well.',
        'Cats eat rats.',
        "Rats don't sleep.",
        )

c = Corpus()
for doc in docs:
    c.add(doc)

print c.cluster()

Which results in:

[
    (
        "Rats don't sleep.",
        {'Cats eat rats.': 0.21353467285253394}
    ),
    (
        'Every good girl does well.',
        {'Every good boy does fine.': 0.32030200927880093}
    )
]

The number is the "similarity" between the strings relative to the entire document corpus.

My next trick is to see if I can run this memory-intensive calculation over a data-set of 25,000 opinions submitted. If I can we can get some interesting data about what people think of the new Firefox beta.

Firefox Input, powered by Sphinx

2010-07-06T00:00:00-07:00

Thursday, I decided to take a half-day for my sanity, but saw an email about how Whoosh wasn't going to cut it for Firefox Input. I was CC'd about this and there was mention that Sphinx might be possible.

Sphinx is my hammer, and everything is a nail. So I said, let's do this. That translated into me spending my weekend, soothing my newborn and working on Sphinx. Luckily this was easy, since AMO and SUMO are both running Sphinx in a similar Django environment.

In order to move quickly, I copied code from the Zamboni project to Firefox Input. Even our deployment into staging and production wasn't done by our usual "Sphinx guy" in IT. Ultimately, everything landed in place.

So try it out and file bugs or let me know if searches don't go as planned.

Your objects, and all their friends

2010-06-25T00:00:00-07:00

This is complicated.

In my ever-evolving quest to get data out of the AMO database for tests, I found myself not just extracting a single object, but a list of complicated requirements in order to fully replicate behavior in production in a testing environment.

For AMO we can use fixture magic to dump a single add-on and all of it's database dependencies so that it will insert safely into a test-database. But we need more than just valid data. We need some supporting data. For an add-on to be browsable and searchable it needs to have a valid version and the version needs to have a valid file.

In our app we can check for these things by using this:

my_addon.current_version.files.all()[0]

Of course we need to check that my_addon.current_version exists and that files.all() has at least one object. This ends up being a lot of work if you just know the id of the add-on object.

So what I want is something simple, like:

./manage.py custom_dump addon 3615

And it should get me everything I need to test add-on 3615, including a Version object and any files associated with the version.

Turns out this just works. It works if you define the following settings:

## Fixture Magic
CUSTOM_DUMPS = {
    'addon': {  # ./manage.py custom_dump addon id
        'primary': 'addons.addon',  # This is our reference model.
        'dependents': [  # These are items we wish to dump.
            # Magic turns this into current_version.files.all()[0].
            'current_version.files.all.0',
        ],
        'order': ('app1.model1', 'app2.model2',),  # stuff gets sorted
        'excludes': {
            'app1.model1': ('fields', 'to', 'hide',),
        },
    }
}

Using this we're able to find out that addon means an addons.addon object and that you want the addon object with an id of 3615. From there we'll try looking for dependent objects. Using some black magic we can turn: current_version.files.all.0 into addon.objects.get(pk=3615).current_version.files.all()[0]. This gives us a file.

If we mimic our dump_object command we can get the file into the database and everything that the file needs to be valid. This in turn gives us enough data (usually) to begin testing a single addon.

So have fun with this, if you're database is remotely complicated, this can save you some time replicating it during testing.

Also note, that you can re-order models and exclude certain fields. This can make your fixtures very easy to load.

So your Wordpress has been hacked

2010-05-19T00:00:00-07:00

Last week, someone informed me that my blog had been hacked:

I'm not quite sure what the vector was. Wordpress wasn't very secure and I didn't take too many measures to harden it. A coworker of mine (on our security team) decided it might be fun to have a look at the infected Wordpress Installation.

Here's how the hack works

Your blog appears normal to you and your visitors.
Some rogue PHP code detects if Google is crawling your site and modifies the text and links so it looks like your website is a Viagra pharmacy.
The links go to other infected blogs and thus builds up page rank for this ring of blogs. So the upside is that your blog may be a top result... for VIAGRA.

Prevention

Here are some tips for prevention, but you can find a lot more by googling for Wordpress hacks. My solutions are more technical:

Don't use Wordpress - I recently switched to Jekyll since it was conceptually easier to understand, and it's coder-friendly.
Remove all users other than your own.
Change your password.
Check your code into git so you can see what files have changed.
Prevent Wordpress from writing to your webroot.

Restoration

Here's what you'll need to do to de-spam yourself:

Verify that you are still spammed by using Google Webmaster Tools|Labs|Fetch as Googlebot.
Back up your blog and database.
Move your Wordpress installation to a new directory.
Install Wordpress from scratch.
Remove all users except for yourself.
Change your password.
Copy your theme to your new installation.
Install only the plugins you need.

By step 4, you should be able to verify, using Fetch as Googlebot, that your website is no longer an online pharmacy.

Good luck.

Test Driven Confidence

2010-04-20T00:00:00-07:00

If you're already testing your web applications, you can skip this post.

One of the bugs I am working for AMO on involves porting a small, but moderately complicated checkbox from our PHP site and rewriting it for Django.

I decided to look at the existing implementation and found it to not work correctly at all. This was frustrating, especially since I verified that my own code worked, and that QA verified that it worked as well.

This is frustrating on many levels. Chances are some minor assumption I made changed, and thus broke this functionality. Discovering regressions is never fun, and fixing them is can be long and tedious if you can't automatically verify that everything is working correctly.

Lucky for me, coming up with tests is easy, you just do what you would do to verify the code satisfies the requirements and then code it. Sometimes the tests can take longer than writing the actual code, but ultimately you can ship with confidence. You can be confident that your feature won't break in the future without immediate notice, and you can be confident that your new code won't break anything else.

Finding the most common Firefox issues

2010-03-18T00:00:00-07:00

Cheng Wang of the Mozilla Support team, a few months back, decided to present on some design ideas for Firefox Support. One of the issues he noted was that there are a lot of repeated issues and that it would be useful to group them. Grouping them lets you see how often something occurs, and secondly let's you see how urgent it might be.

Luckily grouping and clustering text is something computers can do. So I wrote this utility that does just that.

I ran this script over a sampling of data from the last week:

Firefox won't start after update. (65 related issues)
- 5.6: Firefox updated, Gmail not delivering mails
- 5.6: How to change My Profile when Firefox won't load?
- 7.5: Once I close firefox, cannot start firefox again except system restart
- 5.6: When intalling updates Firefox uninstalls itself
- 16.8: firefox won't start after update 3.6
- 11.2: Upgraded to Firefox 3.6 and now it won't start
- 14.9: Firefox won't start with most extensions
How do I add a bookmark to more than one folder? (64 related issues)
- 8.9: How do I get my bookmarks on the bookmarks toolbar to show up as an icon only with no text?
- 7.5: Bookmarks lost after upgrade and cannot save new bookmarks
- 7.5: why do i have to add the .com now to addy's?
- 8.7: When I open sidebar to edit bookmarks, I only see the folder for Bookmarks Toolbar. I do not see a folder just called Bookmarks nor do I see my list of bookmarks, that separately appear under bookmarks menu at top of screen
- 7.5: All my impoted bookmarks go to the same webpage
How do I remove the \ask toolbar\"?" (50 related issues)
- 14.9: How do I remove an unwanted toolbar?
- 5.6: how to remove temporary video files from computer
- 7.5: I have no Toolbars or searchbar and i cant bring them back
- 7.5: nowhere says how to REMOVE a toolbar - only how to add or modify one
not able to open youtube videos (45 related issues)
- 5.6: Cannot open bookmark/history sidebar
- 5.6: After working well for years Firefox will now not open
- 6.7: opening bookmarks do not open in new tab
- 5.6: I can't watch videos on youtube with firefox, but on internet explorer i can
I cannot download Firefox 3.6. I've tried erasing the download file. I cannot get beyond logging out of Firefox. (44 related issues)
- 8.4: when downloading files firefox download manager will freeze and i will have to start over the file download
- 5.6: Firefox will not let me download anything! Can someone help?
- 6.3: cannot download epixHD.com: not compatible with firefox 3.6
- 5.0: Several tabs are coming up when i try to downloads things
- 5.0: Firefox wont open since I downloaded the 3.6 update.

The number on the right of the related issue is a score of how strongly it relates to the main issue.

The full sample is 352 clusters from an original 3000+ issues. That's a lot less stuff to go through. We can tune this to have either less clusters, and more related issues in a cluster, or we can make more clusters of issues and that might result in more accuracy.

Despite the inaccuracy of clustering we can make some general observations:

Firefox not starting is a big issue.
Bookmarks are either confusing or broken.
People don't like toolbars
Opening things is hard
Downloading things or Firefox is hard

Hopefully we can fine tune these reports and have them run regularly... maybe automatically posting to Tumblr?

A few weeks in Chrome

2010-03-17T00:00:00-07:00

A number of weeks ago I got annoyed with Firefox and decided to use Chrome for a while. This reminded me of the olden days where I used Netscape for a while, and then IE6 came out, and then Phoenix came out all the while I'd keep switching to the newest shiniest thing (note: I'm not sure about the timeline of all the browsers either).

My browser of choice since Firefox was released has been Firefox. For some time - nothing shiny in browser-land was coming out. Little UI things in Safari kept me away (and the lack of extensions), but Chrome finally showed promise. WebKit, out of process plugins, process separated tabs and now extensions. This was great.

I immediately felt like I was going to really love Chrome, and be that guy at the office (I work at Mozilla) who insists on using Chrome (just like I was that guy at Yahoo! who used Google for everything). I also wanted to answer the question as to why so many people really like Firefox in spite of Chrome's amazing speed -- even many Googlers will admit to preferring Firefox.

Overall I'm happy with Chrome, but I'm switching back to Firefox for now. Here's some things I observed:

"/" in Firefox let's you search. Which to me seems more natural than Ctrl-F. I am pleased that Chrome supports a lot of Firefox's shortcuts, like Cmd-1..9 for switching tabs, or Cmd-Shift-T for reopening a closed tab.
No titlebar... I kind of miss it.
XML is way easier to work with in Firefox. It's collapsable and always looks pretty.
Certain sites don't work well in Chrome, like Rypple or the AmericanExpress web site. Rypple surprisingly enough is built using the Google Web Toolkit. I really wish their was a "FirefoxTab" that would open certain sites in Firefox instead.
There's a number of Jetpacks and Extensions that only exist for Firefox or they are severely lacking.
- The Jetpacks for Mozilla's Bugzilla instance are awesome.
- The Delicious and AdBlock extensions on Chrome aren't nearly as good as the ones for Firefox.
Firebug is much better than the Chrome developer tools. For example, you can adjust css values instantly.
Extensions die... and don't come back without restart and appear to never have been installed unless you remember them crashing.
AwesomeBar (the location bar in Firefox) queries your history much better than the OmniBar (the location bar in Chrome)
- At first I thought this was because of a sparse history, but after several weeks I still have a hard time finding sites I've been to.
- Chrome will show you a handful of results, and then let you know there are more results, but that takes you to a new screen which is a jarring UI.
Chrome can be slow. The extensions can take a while, and even switching between tabs can be slow. At this point startup time can be a moot point.

Overall this was a healthy exercise, since I really like to be up on new browsers, and Chrome really seems like it's can be a good browser for many people. I'll probably try it again after the next major Chrome update.

Making our tests run thrice as fast

2010-03-16T00:00:00-07:00

I've written a faster version of TransactionTestCase and packaged it with test_utils. It's mysql specific since it relies on SET FOREIGN_KEY_CHECKS=0 to flush the database.

The long story...

Why speed matters

We're closing in on 300 tests for Zamboni. As of yesterday, to run our entire test suite it would have taken approximately 5 minutes. If you run tests before code-reviews, during a code-review, and before you push to master - you've spent about 15 minutes doing tests for a single feature or bug-fix. We have about 5 developers, so this cycle happens many times in a work day. In that time many sandwiches can be made and consumed.

Even shortcuts, like running a subset of tests will only go so far, and ultimately we do want to validate that all our tests pass for any code-change.

Testing Sphinx search with `TransactionTestCase`

Django recently sped up testing by running tests in a transaction. However, this means that data never gets committed to the database and therefore external tools, like the Sphinx indexer, will never see any of that data. So we resort to TransactionTestCase which will commit the data.

Unfortunately TransactionTestCase is painfully slow. The accepted practice is to only use TestCase if you want your tests to be fast. So, I decided to complain to one of our new hires and he and I decided to tinker in mysql to figure out what was slow. We discovered the following:

delete from [table] is slow
truncate [table] is slow
... unless you SET FOREIGN_KEY_CHECKS=0

So we decided we should do our own tear down. After some tinkering with cProfiler I discovered that TransactionTestCase does a (slow) database flush on setup for a test case. This wouldn't do.

Making our own `TransactionTestCase`

I decided to make our own TransactionTestCase and it would just run SET FOREIGN_KEY_CHECKS=0 and TRUNCATE on each table at tear down time. It would also not do a flush on set up.

We write our tests with the idea that they clean up after themselves. Rather than having them cleanup after the last test. This is a requirement for us since django-nose doesn't reorder tests (nor should it) and a standard django.test.TestCase assumes a clean database.

Looking at a single test test_sphinx_indexer, using django.test.TransactionTestCase took ~30 seconds. Using our new TransactionTestCase it takes ~4 seconds!

Fast tests are good

We can now run our 275 tests in ~100 seconds versus the ~300 seconds it used to take. Furthermore, skipping our sphinx tests (which are the only tests that use TransactionTestCase) only saves us ~10seconds. That's not a lot of overhead for better coverage.

This took me the better part of a day, but solving this now, means we're going to more often than not run our sphinx tests all the time rather than skip them. Our QA team will assure you that search is probably the most regression prone part of our site, so running these tests are vital to quality.

If you need to use TransactionTestCase in mysql, give ours a try.

λ^2: safely doing class based views in Django

2010-03-09T00:00:00-08:00

When I started rewriting the API for addons.mozilla.org, my views were mostly the same: get some data and render it as either JSON or XML. I also wanted all my API methods to take an api_version parameter, so I decided class based views would be best. This way my classes could just inherit from a base class.

To do this I had to implement a __call__ method. This works fine, except I wanted to store things into the class -- after all the whole point of my use of classes was to keep the code a bit more compact, and cleaner. So, why pass the api_version around everywhere? Unfortunately thread-safety comes to play, and you need a separate instance of your class for each request.

λ

Django's urlpatterns expects a callable object. So you can't give it an instance of AddonDetailView(). But you could give it a callable that creates an instance of AddonDetailView() and passes it *args and **kwargs. Luckily python has lambda functions. You can note how we solved that in our urlpatterns.

λ λ

But wrapping all your urls with lambda is tedious and remembering to pass *args and **kwargs is error prone.

So let's make a lambda function that returns... a lambda function that turns an instance of our class into a callable.

We can now return to coding and not think about thread safety.

λλλ

django-fixture-magic: Testing issues with real data.

2010-03-05T00:00:00-08:00

I just released Fixture Magic.

When dealing with legacy data, you'll run into all kinds of edge cases. Perhaps, an object might not display correctly unless it has the right parameters, or if it has null parameters it might not display at all. So when testing Django, it's nice to actually use non-dummy data.

Luckily Django has a way of pulling real data out of your database using django.core.serializers:

from addons.models import Addon
a = Addon.objects.get(id=3615)
from django.core.serializers import serialize
jsonize = lambda a: serialize("json", a, indent=4)
jsonize([a])

This solution runs well in a Django shell and can be lots of fun for the whole family... until things get complicated.

Serializing alone isn't enough.

Serializing a fixture with foreign keys means you'll have an un-loadable fixture unless you serialize the dependent fixtures. Even for one or two foreign keys, this can be a pain. For addons.mozilla.org, we have a spidery-web of dependencies: Files need a Version which needs an Addon which need Translations.

Thus begat the dump_object management command. Give it an app, model name and a pk and it will give you not only a serialized JSON of that object, but all the objects that it requires.

Example:

./manage.py dump_object files.file 64874 64876 > my_new_fixture.json

This looks for the File model in the files app and pulls out of the database Files instances with pks of 64874 and 64876. It then recursively searches for any required objects.

Too much serial

If you create a lot of fixtures, you'll eventually have overlapping serialized objects. In addons.mozilla.org we have Addons, Versions (which depend on Addons) and AddonCategorys (which depend on Addons and Categorys). If we wanted to get serialize a specific Addon, it's dependent Versions and AddonCategorys it makes sense to start with dump_objecting the related Version and then dump_objecting the AddonCategory. Both dump_object commands will fetch the Addon in question, resulting in duplicated data.

To combat this we can use merge_fixtures to dedupe our fixtures:

./manage.py dump_object versions.version 64874 > 1.json
./manage.py dump_object categories.addoncategory > 2.json
./manage.py merge_json 1.json 2.json > happy_fixture.json

This should make creating test data slightly less painful. So give it a try.

Palm Pré: A retraction, I really like it now

2010-01-11T00:00:00-08:00

So before I went on trip to Minnesota last month, I decided maybe I would give the Palm Pré another shot. After all, my parents have no internet access, so having the Pré... if I could overcome my issues, might be a welcome distraction.

Before I packed it, I updated to WebOS 1.3.x (a few days later I updated to 1.3.5) and I was blown away. The horsepower was increased by utilizing the GPU. The following problems were fixed:

* The device was no longer hot all the time
* Shutdown and startup were long, but not nearly as long as before.
* Render times were quicker
* All the elements usually rendered quickly in an app
* Network was fairly steady
* Phone calls also seemed fairly drop-free.

All these improvements helped me get over

* The tiny keyboard... not so bad in practice.
* No soft keyboard - I missed it, but I could deal without it.

Overall the device was great, it was fast enough to use, and most of the errors were annoying, but things I could deal with. Cut and paste could be improved, and I wish the USB connector was the same as the one for HTC devices (I can't keep micro or mini USB types straight).

So I love the device, and Murphy's Law dictates if work gives you a phone you don't like you get to keep it... until you start liking it again. So I sent the phone back into rotation for other people at Mozilla to try. Have at it.

Google Chrome Extensions Puzzle

2010-01-06T00:00:00-08:00

I went to Add-on-Con some weeks back to represent my employer, the Mozilla Corporation.

One of the goodies you get as a registrant was a jigsaw puzzle from the Google Chrome Extensions team.

Perfect, my wife and I love solving jigsaw puzzles. We finally finished a few days ago. Anybody who has started at all will realize the puzzle is of a QR-code. The QR-code is an extension that will eventually lead you to a prize. It was a bit of a mini-puzzle not nearly as difficult as finding the QR code.

Although finding a QR code scanner was a bit difficult, I had to borrow a HTC Magic from Justin Scott and installed a decent barcode scanner.

Palm Pre: Always hot

2009-11-19T00:00:00-08:00

So I borrowed a Palm Pré that we had at Mozilla to see what it was like. I was at first very excited, I remember before the Pre was released there was a lot of talk about how awesome-fantastic it was going to be. The stories of awesomeness sort of died, and I had thought nothing of it.

Immediately upon using the Pre I figured out why. In short, it's a crappy phone. It makes a very good attempt to do a lot, but it does them with such piss-poor performance, that nothing good is noticed.

I am disappointed. It's not even in the same class as an iPhone - maybe a future generation of Palm devices will be, but not this one. I was hoping WebOS would be a good alternative to the iPhone. It looks like Google will be doing that, though their phones haven't impressed me much either. I am hoping that maybe this phone is just a dud.

Here's what I didn't like: * The Palm was always hot. * The first run experience is painfully slow. * The first run was an indicator of things to come, startup and shutdown are ridiculously slow. * Every application is slow to render. * Not all elements of an app render. * The keys are too small. Some people aren't migrating from a Treo and aren't used to mini keys. * No soft keyboard. * The palm website doesn't let you use plus-style addressing * Media Mode was not self explanatory - and forced the phone to not work. * Network would constantly drop out. Couldn't use a lot of the data features. * Phone calls didn't work so great. * Did I mention it was ass slow, even the dialing program was slow. * The battery dies quickly * I could only cut/paste when composing, but I couldn't cut a string of text from an email. * Felt too much like an old palm

Despite the sadness there were a few good things:

When it did fetch email, and other notices, it displayed them nicely
The unification of Facebook and Gmail was pretty cool - it also made me want to trim some of those friends from highschool off my facebook - I ain't ever gonna call em.
The Icons were pretty.
The card interface was interesting.
The travel charger could be modified to work in non US chargers fairly easily.

All in all, I'm glad that I had a chance to try out this device. It showed me, that user interfaces above all need to be very fast and responsive. Furthermore, everything you try to do should be done exceptionally well. I'm hopeful that software updates can alleviate some of the problem, but I think the root of the problem is slow hardware.

AMO Search: Powered by Sphinx

2009-09-30T00:00:00-07:00

Last night, I gave a talk at the Addons Meetup at Threadless HQ in Chicago on the new search engine powering addons.mozilla.org. I'll recap the technical portion of the talk and give a bit more details.

First, I'd like to thank Harper and Threadless. It was a great location in the greatest city in the universe. Before and after the meetup, Harper was just an all-around great guy to hang with and the threadless headquarters was a nice hangout place for meeting people interested in addons.

Shortly after my talk, our Engineering Ops team deployed the new AMO 5.1 complete with a new Sphinx powered search engine.

So let's talk about search. Note: parts of this are a rehash of my talk, so feel free to skip around.

A bit about addons

Addons is a huge growing space. Arguably it's Mozilla's best kept secret. Sure readers of this blog probably know what Addons are, but ask people who aren't as web-savvy. Most people don't know what a browser is - and it's hard to explain it to people without getting technical.

We can just skip that step. Because Addons are small things that people can easily "get".

"It's an easy way to customize the internet when your surfing."

While perhaps not technically correct, its one way of explaining it to people. Maybe a better way is just showing people what they can do with addons.

On my flight out to Chicago, I talked to a person on the plane who didn't know what a browser was, but after showing her AMO she was really intrigued.

If everyday non-technical people can realize the potential of addons, it's only a matter of time before they start knocking down the doors to AMO.

So we better be prepared to handle them, and get them what they want.

The technical details of addons.mozilla.org

Everytime you open Firefox, it pings AMO to see if there's any updates to any of the addons that happen to be installed. Over a third of the people using Firefox have at least one addon, and Firefox is roughly 22% of the browser market. That means roughly 7% of people opening their browsers are pinging our servers for updates.

Needless to say it's a lot of traffic, and to support it we need a fair amount of hardware. AMO is clearly the largest site in the Mozilla universe in both respects.

Some stats:

1 mySQL master
4 mySQL slaves
2 memached servers
2 Sphinx indexer/search daemons
24 Web Frontend
Multiple Zeus ZXTM clusters all

Most of this is standard, we'll talk about Sphinx later, but Zeus is amazing. I didn't know what Zeus was until earlier this year when I interviewed with Mozilla's VP of Engineering Operations. All our requests get cached so much of our hits actually hit our Zeus cluster and not our web servers.

To see just how amazing they are read our mrz's ops blog.

Why search matters

If you have any kind of custom content and unique meta data a custom search solution is a must. Browsing through a site isn't going to cut it. Browsing is dead. Search is how you find things on a web site. On AMO you may see an addon that's featured somewhere, or you might want to see what's out there, but the right search query will find you the right addon in two clicks.

Improve Search

So my first job on AMO was to improve addons search. It was a vague request and born out of frustration with what we had. It wasn't a problem that certain things were indexed, or unicode didn't work, or results weren't sorted. We may have had all those problems, but as a product search needed to be replaced.

To me it meant that we needed some framework that would allow developers to quickly debug and fix any future search calamities at a moments notice.

So here were the goals I made for myself:

Do something that sucks less than what we’ve got
Do something that makes it easier to suck less in the future
Do something that’s easy to use for our operations team, web developers and most importantly, end-users
Reduce strain on our databases, developers and operations teams

Complex Data

Our data set is small (we have 5,000 addons), but there's a lot of secondary meta data about the addons that we track:

Addons work in 1 or more locales (e.g. en-US, fr, de, etc)
Addons are optionally platform specific (Linux, OS X, etc)
Addons work with one or more products (Firefox, Thunderbird, Seamonkey, Sunbird or Fennec)
Addons come in multiple flavors (extensions, themes, dictionaries and more)

We want to index all this data. Unfortunately to get at much of this data it involves either numerous queries, or numerous joins which put a strain on mysql. How much strain?

At peak we get about 10 search queries per second. If we do something smarter this won't have to cause a lot of strain.

Using Sphinx

Sphinx is an open source search indexer and daemon. It's used by Craigslist, the Pirate Bay and Mozilla Support. It was very easy to use and despite a complicated set of data and business logic, Sphinx was up to the task.

The challenges

We needed to search for addons in several languages. So indexing just addons wouldn't work, we need to make sure we have every translation of every addon indexed. For those counting, we have 5,000 addons, but 18,000 translations of addons.

All the joining and filtering that needed to be done for our old search still needs to be done, but we can do this all in one shot by using a mysql view. This view is a flat list of each translated addon as well as all meta data associated with it. This then gets fed into the sphinx indexer.

Along the way we ran into some issues which used to be dealt with outside of mysql, such as comparing versions. It was gross and quite a hack, so we turned the variety of acceptable version strings into integers.

We also learned that stemming wasn't a good idea as we assumed it would be. Stemming was great for searching through lots of text, but a great deal of addon searches were really just searches for product names, so we opted for substring searches. We'll see how that fares. There is probably room for improvement.

Much of this, however involved knowing our data, and knowing how it will be used by our users. Once we got that down, we could hammer it all out using Sphinx.

Wins

So Sphinx gains us a bit architecturally. We have a complicated query, but it only gets run once every 5 minutes versus the 180,000 times it was run "on demand."

Indexing happens rather quickly, just over a minute.

The API was a breeze to work with, and was easy to drop into our own codebase.

Because of our relatively small data set, and quick indexing, we're able to scale this simply by cloning and load balancing. Meaning, we just need to scale for traffic, but addon growth (which is slower than traffic growth) we can safely not worry about for a while.

Our ops team can monitor the sphinx clusters and just deploy additional nodes as needed.

Building a platform

What we've done is built a foundation for search. Not all the problems are gone, but a lot of the problems that our QA team finds are able to be resolved quickly. We have a nice pile of unit tests as well that help us keep our results in check when we start tweaking dials.

We even have the groundwork for some nifty advanced search syntax, that hopefully we can inject into future releases of AMO.

Enjoy. And if you find anything, let me know.

Getting started with pipe viewer

2009-09-16T00:00:00-07:00

Despite working on slimming the addons.mozilla.org database through dieting and exercise - I still have to occasionally do long running database tasks. So I finally tried out pipe viewer. As someone who's impatient this has been awesome. Here's some quick examples:

[root@ml-db10 sun]# pv -cN source < addons_remora.2009.09.15.sql.gz | gunzip|pv -cN gunzip > addons_remora.2009.09.15.sql
   gunzip: 10.1GB 0:06:48 [25.5MB/s] [   <=>                                  ]
   source: 3.47GB 0:06:48 [8.72MB/s] [======================>] 100%

Here we are calling pipe viewer with an argument that says to title this progress meter as source, and feeding it the gzip'd file. Pipe viewer will output two things the progress, and the actual file. We pipe that file into gunzip to unzip it, and back into another instance of pipe viewer (again with a title, of gunzip) and the standard output gets redirected to our destination file.

Now a simpler example is checking the progress of loading a large sql file into mysql:

[root@ml-db10 sun]# pv -cN sql < addons_remora.2009.09.15.sql | mysql -uroot addons_remora -p$PWD
      sql: 2.55GB 0:18:19 [5.68MB/s] [=====>                  ] 25% ETA 0:54:30

We could have probably combined all this, however:

[root@ml-db10 sun]# pv -cN source < addons_remora.2009.09.15.sql.gz | gunzip|pv -cN gunzip | mysql -u root addons_remora -p$PWD

Armed with this knowledge you can determine whether to grab a soda, a sandwich or a 2-hour lunch.

DjangoCon wrapup

2009-09-15T00:00:00-07:00

I went to DjangoCon this past week for work. Django is one of my favorite frameworks. I dropped PHP and the symfony framework to learn python and Django and I haven't looked back. I think for Mozilla's webdev team it would be the framework of choice. We have 100s of sites in many frameworks, but not a lot of resuability. Django apps are built to built to be reusable. If you build correctly you don't have to refactor, it's already done.

Here's a collection of notes I collected through the conference.

Day one

Keynote - Avi Bryant

Frameworks lock us into RDBMS = bad

This keynote mentioned the limits of modern frameworks and modern web development. Essentially frameworks are great for getting started, but as a site grows, the framework gets replaced little by little. Sometimes it can get in the way - such as with limitation of database choices.

UR doing it wrong - James Bennet

James outlined a few key problems that many Django developers run into:

learning python as you go
- doesn't work unless you know some programming upfront
- do the python tutorial
- read python in a nutshell or dive into python
Things you should know:
- subclasses
- super()
- slides went too fast... hopefully they'll be posted

All in all RTFM for python and Django :)

Learn about other py packages... like twisted. If Twisted Matrix was implemented in Ruby it would be advertised as the second coming of Christ.

Bennet's Django App review smoketests:

installable via pip, easy_install or setup.py
- read distutils-guide
- stay away from setuptools
have a README
INSTALL file list deps
Write DOCUMENTATION
- use sphinx.pocoo.org
- store it in your package and upload package docs
LICENSE (most Django apps use BSD)
Write unit tests
django-lint - to look over code (like pep8.py)

pro-django is a decent book, but not written by Bennet.

Testing - Eric Holscher

Django 1.1 encourages you to test by auto-creating tests.py.
Support for:
- Unittests
- Doctest
- Tests done in a db transacation
Test Driven Documentation (TDD + DDD)
Doctest
- easy
- can't use PDB
- Hides certain failures
Unittests via Django TestCase
- XUnit
- setup/Teardown
- adds db fixtures
- assertions
- mail testing/inbox testing
- url testing
TestCase
- Browserless Request/Response testing
- Similar to sfBrowser in symfony
Google Summer of Code (for Django 1.2)
- Coverage reports!
I need to learn PDB

Deploying Django -

Run mod_wsgi in daemon mode.

Day 2

Keynote - Ian Bicking

GNU Manifest:

I consider that the golden rule requires that if I like a program I must share it with other people who like it. Software sellers want to divide the users and conquer them, making each user agree not to share with others. I refuse to break solidarity with other users in this way. I cannot in good conscience sign a nondisclosure agreement or a software license agreement. ...

So that I can continue to use computers without dishonor, I have decided to put together a sufficient body of free software so that I will be able to get along without any software that is not free.

GNU manifesto was the idea of sharing software amongst friends
GNU has purpose - BSD, etc is just a rule - free to share
Free is not just the absense of copyright
Free is not a reaction to existing rules, but a golden rule
Not just a fight against MS
Need to find morality (the why) within the practical (the law, or what you can do)
Open sourcing closed source code isn't building open source
This might apply to Mozilla... as webkit has taken off more than Gecko.
Open source is person to person not company to company - despite sponsorship.

Using Django in Non-standard ways - Eric Florenzano

Django loosely coupled
Replace templating with Jinja 2
Copy Django methods into djangoext to easily customize Django behavior
Not using django.contrib.auth
- reasons: writing a fb app - no auth needed
- no shoehorning needed - saves time - less overhead
skip the orm?
- legacy dbs
- non standard or db (or non-relational database)
- no database
wsgi middleware has some cool shit
- repose.bitblt: autoscales images
- repose.squeeze: will concat js/css on the fly based on statistical analysis
non standard Django based apps
- YARDBird - IRCBot framework
- djng micro framework
- Jngo- singlefile cms
using admin in a nonstandard way is hard/impossible coupled with ORM and auth

Real-time web and other Buzzwords - Chris Wanstrath

more than just getting your rss feeds faster
push vs. pull
1 persisting connection vs polling
comet/flash-xml/or html5 web socket
orbitted - open source python comet server
zeddicus - does the business logic
orbitted has its own js libs - its a simple port/socket thing for your server code to deal with - not request/response.
all connections are persisting browser/orbitted orbitted/zeddicus
You can even use orbitted to connect straight to IRC and write a client in JS
Jetty also is good for comet

Also: * see webhooks * see pubsubhubub

Pluggable, Reusable Django Apps: A Use Case and Proposed Solution - Shawn Rider and Nowell Strite

PBS moved from perl to django - build a lot of reusable apps
convincing your superiors
- need a good story -
- existing base of python helped
- With Django easy to do things right without doing things slow
- be really good...
built a lot of apps to be very reusable, and pluggable based on requirements PBS had

Day 3

Keynote - Ted Leung - Sun

Django jobs are a growing market
Preferred by startups
Bespin/wave - cool
APIs are big... still
Physically impossible to create purely server-side interactions that are usable enough - rely on rest/comet/ajax/etc to bridge gap

Scaling Django Mike Malone

MM from Pownce (now sixapart)
Slides started out as "Building Scalable Web Applications"
Django didn't get in the way too much when it came to scaling
Django had tons of caching support
Cached objects by hand (memcached) and object ID lists
Use memache for sessions too
use signals to signal cache invalidation
race conditions...
Queue shit... gearman, rabbit mq, etc.
Memecached incr/decr operators are awesome
See gh/mmalone/django-caching
See gh:.../django-multidb
to combat slavelag use a memcache key to alternate between master or slave

Gearman - working later - Chris Heisel

Gearman - a work later alt to rabbit mq
Makes the most sense for something like cesium, with a bazillion worker ~~bees~~ foxes feeding off a single queue

Also at the con, I talked to someone about rebuilding large apps... and they took a PHP app and used URL rewriting to and a lot of PHP/Python glue code to build a seamless transitory app. The rule is, all new functionality was done up in python while the old app was in maintenance mode.

More talks here!

Snow Leopard for Macports and Mysql users

2009-09-02T00:00:00-07:00

I use mysql and macports on OSX and both were broken when I upgraded to Snow Leopard.

Mysql was a quick fix:

ln -s /usr/local/mysql-5.1.35-osx10.5-x86 /usr/local/mysql

(you're installed version might be different). It turns out a symlink was removed during the Snow Leopard upgrade.

As for MacPorts, I had to install Xcode from the Snow Leopard CD, install the Snow Leopard version of MacPorts and then follow this migration guide.

Fun with HTML5, Blockchalk, Bookmarklets and Google Maps

2009-09-01T00:00:00-07:00

I've been filing away neat things that I've learned. Like:

Firefox supports GeoLocation (which varies in accuracy, but is really accurate for me)
Stephen Hood released the Blockchalk API
Google Maps lets you use GeoRSS feeds as a term

This solves the problem I had with BlockChalk, which is I wanted a way that I could see what's going on near where I am - and I don't have an iPhone.

So I wrote two bookmarklets:

Blockchalk Me which will list Blockchalk listings near you

and

Blockchalk this Google Map which only works if Google Maps is open. It will load Blockchalks that are near the center of the open Google Map.

Unfortunately Blockchalk doesn't have a lot of data yet, and will return no results if there's nothing within a mile radius. Hopefully a radius parameter will be included for the API call.

So there's no guarantees on the first bookmarklet, but the second bookmarklet should yield nice results for this location.

Enjoy.

git svn rebase... forever?

2009-08-12T00:00:00-07:00

While working on addons.mozilla.org I ran into an issue of git svn rebase continually asking me to merge a file, over and over.

I had a branch open for a bug. In that branch I wrote a library. While that bug was under review, I had to use that library in a new branch for another bug - and had to develop on it a bit.

$ git co -b bug1 master
$ vi libs/mylib.php # make the lib
$ git add .
$ git commit -m "my new lib"
$ git checkout -b bug2 master
$ git checkout bug1 libs/mylib.php # copies this file from one branch to the next
$ git commit -m "lib copied over"
$ vi libs/mylib.php # hack on the lib 
$ git commit -m "awesomized lib"
$ git svn dcommit # push it up
$ git checkout bug1
$ git svn rebase #... oh shit

So the rebase was happening. This is git trying to merge your changes in bug1 and bug2 and play them together in realtime nicely, asking you each step of the way to merge things manually. I thought something weird was happening since "libs/mylib.php" kept needing manual merging. Then I noticed that git is applying a series of patches, and that eventually this will resolve and your site will be rebased.

Don't lose hope, git svn rebase will finish.

Have unique descriptive page titles

2009-07-31T00:00:00-07:00

One of my internet pet-peeves is people using the same page title for every page on their web site. Take a look at this search for Sphinx. As you can see virtually all the links for Sphinx are titled "Sphinx - Free open-source SQL full-text search engine" which blows for usability when it comes to searching, or even managing the various pages you might have open in your web browser.

To get an idea of the page I want I need to look at the abstract which may or may not give me a clue. Even the forum posts which usually have subjects, have their <title> set to the site-wide default.

The first step in solving this, is identifying you have a problem in the first place. So I wrote a tool in python to determine how unique the page titles you have are:

% python measure.py sphinxsearch.com
1050 titles found for sphinxsearch.com
483 unique titles found for sphinxsearch.com
46% of the pages on sphinxsearch.com have unique titles

This is fairly telling. It means over half the pages on sphinxsearch.com have a generic title.

This site faired a bit better:

% python measure.py spindrop.us
988 titles found for spindrop.us
822 unique titles found for spindrop.us
83% of the pages on spindrop.us have unique titles

So please, think about people trying to use the information on your site. Design your templates in such a way that you can come up with unique titles.

Feel free to expand on this tool, it could easily output the offending pages or titles.

Comprehensive list of international dialing codes

2009-07-01T00:00:00-07:00

I get bored with mundane tasks. So I create little adventures for myself. I had to create a list of countries and country codes to use on the Firefox mobile home page. The first few lists were incomplete, so I made my own by parsing a list provided by the International Telecommunication Union. I stripped it down to simple forms of the country names and removed codes that are very rare (satellite phones).

Well this could be a boring task for most people, so I placed it on [github][http://github.com/davedash/International-Dialing-Codes/]. Feel free to use this in your own projects, I even include a perl one-liner in the README to convert this into a drop down HTML list.

Question: Building a Better Search Engine

2009-06-18T00:00:00-07:00

So I finally have one of those jobs where I can tell people almost every little detail about what I'm doing and I'm encouraged to talk to people on the intar-webs and solicit opinions.

Uh - this is more or less how I've operated at previous jobs, just now I can be overt about it.

So my new task is to work on improving the addons.mozilla.org search engine. I've built various "search engines" over time in PHP, powered by Lucene and most recently in python using an inverted index.

One tool that I've been looking at briefly is Sphinx. While my record count is low (5-10K), Sphinx basically bakes in a lot of the things I would want in a search engine. Indexing, merging, etc.

Since I'm fairly new to the add-ons team I'm still understanding the basics of what we need:

Fast automated indexing of addons for Firefox, Thunderbird and any other Mozilla product
Quick result sets
Easy deployability
Extendible
Customized ranking
Filtering (e.g. by Firefox version, etc).
Basics: Stemming and stop-words

Whether it's Sphinx, Lucene or some home grown solution, I have all that to support. But this should be fairly straight forward. What are people's thoughts?

From Delicious to Mozilla

2009-05-26T00:00:00-07:00

Today I said my good-byes to Delicious.com and Yahoo! and tonight I went to the Addons Meetup @ Mozilla to get a sneak peak at what I'll be working on in less than two weeks.

I was thrilled. I had no idea how many people to expect, but the Mozilla living room was packed - and most people were there the whole time. Real developers with really cool addons giving feedback to addons.mozilla.org directly. No matter how many blog comments, forums answered, customer care emails I responded to at Delicious - nothing beats the real insight and instant feedback you get from meeting a group of users face to face.

My brain, because of Delicious is always in data mining and analysis mode so through each presentation and each question asked, my brain was churning through things that I could build to bring some level of utility to the community.

I'm also happy to be joining an organization where everything is open sourced and available for comment. So I'm hoping to post a lot more on some of the cool tricks I do at Mozilla.