Dave Dash

Your objects, and all their friends

2010-06-25T00:00:00-07:00

This is complicated.

In my ever-evolving quest to get data out of the AMO database for tests, I found myself not just extracting a single object, but a list of complicated requirements in order to fully replicate behavior in production in a testing environment.

For AMO we can use fixture magic to dump a single add-on and all of it's database dependencies so that it will insert safely into a test-database. But we need more than just valid data. We need some supporting data. For an add-on to be browsable and searchable it needs to have a valid version and the version needs to have a valid file.

In our app we can check for these things by using this:

my_addon.current_version.files.all()[0]

Of course we need to check that my_addon.current_version exists and that files.all() has at least one object. This ends up being a lot of work if you just know the id of the add-on object.

So what I want is something simple, like:

./manage.py custom_dump addon 3615

And it should get me everything I need to test add-on 3615, including a Version object and any files associated with the version.

Turns out this just works. It works if you define the following settings:

## Fixture Magic
CUSTOM_DUMPS = {
    'addon': {  # ./manage.py custom_dump addon id
        'primary': 'addons.addon',  # This is our reference model.
        'dependents': [  # These are items we wish to dump.
            # Magic turns this into current_version.files.all()[0].
            'current_version.files.all.0',
        ],
        'order': ('app1.model1', 'app2.model2',),  # stuff gets sorted
        'excludes': {
            'app1.model1': ('fields', 'to', 'hide',),
        },
    }
}

Using this we're able to find out that addon means an addons.addon object and that you want the addon object with an id of 3615. From there we'll try looking for dependent objects. Using some black magic we can turn: current_version.files.all.0 into addon.objects.get(pk=3615).current_version.files.all()[0]. This gives us a file.

If we mimic our dump_object command we can get the file into the database and everything that the file needs to be valid. This in turn gives us enough data (usually) to begin testing a single addon.

So have fun with this, if you're database is remotely complicated, this can save you some time replicating it during testing.

Also note, that you can re-order models and exclude certain fields. This can make your fixtures very easy to load.

Alphabetical sorting in Sphinx

2010-04-21T00:00:00-07:00

Sphinx 0.9.9 is great at searching full text, but treating actual strings as attributes takes some work.

Initially I employed the strategy of indexing my full text fields and storing them as attributes. E.g.:

sql_query = SELECT name, name AS name_ord FROM documents
sql_attr_str2ordinal = name_ord

This stores each attribute in lexical order. Meaning if your name's are Apple, Aardvark, Button, Choco-room they would be given the ordinal 2, 1, 3, 4 respectively.

However, this is case-insensitive. So trying this approach:

sql_query = SELECT name, UPPER(name) AS name_ord FROM documents
sql_attr_str2ordinal = name_ord

Will allow for case-insensitive alphabetical sorting in Sphinx.

Test Driven Confidence

2010-04-20T00:00:00-07:00

If you're already testing your web applications, you can skip this post.

One of the bugs I am working for AMO on involves porting a small, but moderately complicated checkbox from our PHP site and rewriting it for Django.

I decided to look at the existing implementation and found it to not work correctly at all. This was frustrating, especially since I verified that my own code worked, and that QA verified that it worked as well.

This is frustrating on many levels. Chances are some minor assumption I made changed, and thus broke this functionality. Discovering regressions is never fun, and fixing them is can be long and tedious if you can't automatically verify that everything is working correctly.

Lucky for me, coming up with tests is easy, you just do what you would do to verify the code satisfies the requirements and then code it. Sometimes the tests can take longer than writing the actual code, but ultimately you can ship with confidence. You can be confident that your feature won't break in the future without immediate notice, and you can be confident that your new code won't break anything else.

django-fixture-magic: Testing issues with real data.

2010-03-05T00:00:00-08:00

I just released Fixture Magic.

When dealing with legacy data, you'll run into all kinds of edge cases. Perhaps, an object might not display correctly unless it has the right parameters, or if it has null parameters it might not display at all. So when testing Django, it's nice to actually use non-dummy data.

Luckily Django has a way of pulling real data out of your database using django.core.serializers:

from addons.models import Addon
a = Addon.objects.get(id=3615)
from django.core.serializers import serialize
jsonize = lambda a: serialize("json", a, indent=4)
jsonize([a])

This solution runs well in a Django shell and can be lots of fun for the whole family... until things get complicated.

Serializing alone isn't enough.

Serializing a fixture with foreign keys means you'll have an un-loadable fixture unless you serialize the dependent fixtures. Even for one or two foreign keys, this can be a pain. For addons.mozilla.org, we have a spidery-web of dependencies: Files need a Version which needs an Addon which need Translations.

Thus begat the dump_object management command. Give it an app, model name and a pk and it will give you not only a serialized JSON of that object, but all the objects that it requires.

Example:

./manage.py dump_object files.file 64874 64876 > my_new_fixture.json

This looks for the File model in the files app and pulls out of the database Files instances with pks of 64874 and 64876. It then recursively searches for any required objects.

Too much serial

If you create a lot of fixtures, you'll eventually have overlapping serialized objects. In addons.mozilla.org we have Addons, Versions (which depend on Addons) and AddonCategorys (which depend on Addons and Categorys). If we wanted to get serialize a specific Addon, it's dependent Versions and AddonCategorys it makes sense to start with dump_objecting the related Version and then dump_objecting the AddonCategory. Both dump_object commands will fetch the Addon in question, resulting in duplicated data.

To combat this we can use merge_fixtures to dedupe our fixtures:

./manage.py dump_object versions.version 64874 > 1.json
./manage.py dump_object categories.addoncategory > 2.json
./manage.py merge_json 1.json 2.json > happy_fixture.json

This should make creating test data slightly less painful. So give it a try.

Django: Model Inheritance or Related Tables wrt AMO

2009-12-15T00:00:00-08:00

When I attended DjangoCon this year, I lamented that our flagship web property was difficult to test, and not fun to develop. I figured DjangoCon was a way to placate me, and Django might mean something for some of the smaller projects at Mozilla. However, Wil Clouser, our lead web developer, announced development changes for addons.mozilla.org (AMO) that says we'll be moving to Django.

Wil was open to Django and knew that's what we in the dev team wanted. Jeff spawned our foray into a new AMO with Zamboni. I've been working on some grunt-work tasks inside and outside of Django.

One of those tasks is building a transparent layer in Django to keep users logged in from our PHP-based site. That kind of problem almost immediately forces you to ask one of the most fundamental questions you ask when using any framework:

How much do I change my app, in order to accommodate the framework?

More specifically:

Should I use the django.contrib.auth User module, and to what extent?

The more we looked into what features of Django we might want to use, django.contrib.auth was heavily tied into other things we wanted, so it made sense for us to use it. The next question is whether we try the inheritance approach or do we treat our legacy users table as a sort of User Profile and utilize the User module using the related table approach?

Using model-inheritance seems real nice, because we can pretend that our legacy user is the same thing as a djaango.contrib.auth User - but this isn't true:

Looking at our users table more closely:

mysql> explain users;
+-------------------------+---------------------+------+-----+---------------------+----------------+
| Field                   | Type                | Null | Key | Default             | Extra          |
+-------------------------+---------------------+------+-----+---------------------+----------------+
| id                      | int(11) unsigned    | NO   | PRI | NULL                | auto_increment |
| email                   | varchar(255)        | YES  | UNI | NULL                |                |
| password                | varchar(255)        | NO   |     |                     |                |
| firstname               | varchar(255)        | NO   |     |                     |                |
| lastname                | varchar(255)        | NO   |     |                     |                |
| nickname                | varchar(255)        | YES  | MUL | NULL                |                |
| bio                     | int(11) unsigned    | YES  | MUL | NULL                |                |
| emailhidden             | tinyint(1) unsigned | NO   |     | 0                   |                |
| sandboxshown            | tinyint(1) unsigned | NO   |     | 0                   |                |
| homepage                | varchar(255)        | YES  |     | NULL                |                |
| display_collections     | tinyint(1) unsigned | NO   |     | 0                   |                |
| display_collections_fav | tinyint(1) unsigned | NO   |     | 0                   |                |
| confirmationcode        | varchar(255)        | NO   |     |                     |                |
| resetcode               | varchar(255)        | NO   |     |                     |                |
| resetcode_expires       | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notifycompat            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| notifyevents            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| deleted                 | tinyint(1)          | YES  |     | 0                   |                |
| created                 | datetime            | NO   | MUL | 0000-00-00 00:00:00 |                |
| modified                | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notes                   | text                | YES  |     | NULL                |                |
| location                | varchar(255)        | NO   |     |                     |                |
| occupation              | varchar(255)        | NO   |     |                     |                |
| picture_type            | varchar(25)         | NO   |     |                     |                |
| averagerating           | varchar(255)        | YES  |     | NULL                |                |
+-------------------------+---------------------+------+-----+---------------------+----------------+

You can very easily argue that this is a profile table, which happens to have credential information thrown in.

I can see overtime, I'll just struggle to keep our legacy User to act like a Django User, whereas a UserProfile is fairly standard.

Had I been writing this app from scratch, I would have chosen the UserProfile route. This is extra data which takes up a lot of space, and changes far more often than user credentials. Changing 4M+ rows sucks, by making users our UserProfile table, any changes to that table, don't tie up the table used for sign-ins.

I'm curious what other people who port their apps to Django have done.

AMO Search: Powered by Sphinx

2009-09-30T00:00:00-07:00

Last night, I gave a talk at the Addons Meetup at Threadless HQ in Chicago on the new search engine powering addons.mozilla.org. I'll recap the technical portion of the talk and give a bit more details.

First, I'd like to thank Harper and Threadless. It was a great location in the greatest city in the universe. Before and after the meetup, Harper was just an all-around great guy to hang with and the threadless headquarters was a nice hangout place for meeting people interested in addons.

Shortly after my talk, our Engineering Ops team deployed the new AMO 5.1 complete with a new Sphinx powered search engine.

So let's talk about search. Note: parts of this are a rehash of my talk, so feel free to skip around.

A bit about addons

Addons is a huge growing space. Arguably it's Mozilla's best kept secret. Sure readers of this blog probably know what Addons are, but ask people who aren't as web-savvy. Most people don't know what a browser is - and it's hard to explain it to people without getting technical.

We can just skip that step. Because Addons are small things that people can easily "get".

"It's an easy way to customize the internet when your surfing."

While perhaps not technically correct, its one way of explaining it to people. Maybe a better way is just showing people what they can do with addons.

On my flight out to Chicago, I talked to a person on the plane who didn't know what a browser was, but after showing her AMO she was really intrigued.

If everyday non-technical people can realize the potential of addons, it's only a matter of time before they start knocking down the doors to AMO.

So we better be prepared to handle them, and get them what they want.

The technical details of addons.mozilla.org

Everytime you open Firefox, it pings AMO to see if there's any updates to any of the addons that happen to be installed. Over a third of the people using Firefox have at least one addon, and Firefox is roughly 22% of the browser market. That means roughly 7% of people opening their browsers are pinging our servers for updates.

Needless to say it's a lot of traffic, and to support it we need a fair amount of hardware. AMO is clearly the largest site in the Mozilla universe in both respects.

Some stats:

1 mySQL master
4 mySQL slaves
2 memached servers
2 Sphinx indexer/search daemons
24 Web Frontend
Multiple Zeus ZXTM clusters all

Most of this is standard, we'll talk about Sphinx later, but Zeus is amazing. I didn't know what Zeus was until earlier this year when I interviewed with Mozilla's VP of Engineering Operations. All our requests get cached so much of our hits actually hit our Zeus cluster and not our web servers.

To see just how amazing they are read our mrz's ops blog.

Why search matters

If you have any kind of custom content and unique meta data a custom search solution is a must. Browsing through a site isn't going to cut it. Browsing is dead. Search is how you find things on a web site. On AMO you may see an addon that's featured somewhere, or you might want to see what's out there, but the right search query will find you the right addon in two clicks.

Improve Search

So my first job on AMO was to improve addons search. It was a vague request and born out of frustration with what we had. It wasn't a problem that certain things were indexed, or unicode didn't work, or results weren't sorted. We may have had all those problems, but as a product search needed to be replaced.

To me it meant that we needed some framework that would allow developers to quickly debug and fix any future search calamities at a moments notice.

So here were the goals I made for myself:

Do something that sucks less than what we’ve got
Do something that makes it easier to suck less in the future
Do something that’s easy to use for our operations team, web developers and most importantly, end-users
Reduce strain on our databases, developers and operations teams

Complex Data

Our data set is small (we have 5,000 addons), but there's a lot of secondary meta data about the addons that we track:

Addons work in 1 or more locales (e.g. en-US, fr, de, etc)
Addons are optionally platform specific (Linux, OS X, etc)
Addons work with one or more products (Firefox, Thunderbird, Seamonkey, Sunbird or Fennec)
Addons come in multiple flavors (extensions, themes, dictionaries and more)

We want to index all this data. Unfortunately to get at much of this data it involves either numerous queries, or numerous joins which put a strain on mysql. How much strain?

At peak we get about 10 search queries per second. If we do something smarter this won't have to cause a lot of strain.

Using Sphinx

Sphinx is an open source search indexer and daemon. It's used by Craigslist, the Pirate Bay and Mozilla Support. It was very easy to use and despite a complicated set of data and business logic, Sphinx was up to the task.

The challenges

We needed to search for addons in several languages. So indexing just addons wouldn't work, we need to make sure we have every translation of every addon indexed. For those counting, we have 5,000 addons, but 18,000 translations of addons.

All the joining and filtering that needed to be done for our old search still needs to be done, but we can do this all in one shot by using a mysql view. This view is a flat list of each translated addon as well as all meta data associated with it. This then gets fed into the sphinx indexer.

Along the way we ran into some issues which used to be dealt with outside of mysql, such as comparing versions. It was gross and quite a hack, so we turned the variety of acceptable version strings into integers.

We also learned that stemming wasn't a good idea as we assumed it would be. Stemming was great for searching through lots of text, but a great deal of addon searches were really just searches for product names, so we opted for substring searches. We'll see how that fares. There is probably room for improvement.

Much of this, however involved knowing our data, and knowing how it will be used by our users. Once we got that down, we could hammer it all out using Sphinx.

Wins

So Sphinx gains us a bit architecturally. We have a complicated query, but it only gets run once every 5 minutes versus the 180,000 times it was run "on demand."

Indexing happens rather quickly, just over a minute.

The API was a breeze to work with, and was easy to drop into our own codebase.

Because of our relatively small data set, and quick indexing, we're able to scale this simply by cloning and load balancing. Meaning, we just need to scale for traffic, but addon growth (which is slower than traffic growth) we can safely not worry about for a while.

Our ops team can monitor the sphinx clusters and just deploy additional nodes as needed.

Building a platform

What we've done is built a foundation for search. Not all the problems are gone, but a lot of the problems that our QA team finds are able to be resolved quickly. We have a nice pile of unit tests as well that help us keep our results in check when we start tweaking dials.

We even have the groundwork for some nifty advanced search syntax, that hopefully we can inject into future releases of AMO.

Enjoy. And if you find anything, let me know.

V is for Version Hell

2009-08-07T00:00:00-07:00

Versioning is quite difficult to deal with. Versions are nearly-numbers, but you can't quite sort them using standard numerical algorithms.

While the following is true:

1.1 < 1.2

The following is also true:

1.2 < 1.18 < 1.20

The "." is not a decimal point but a separator.

Mozilla uses a modestly complicated versioning system that involves stars, plusses, and sometimes "x".

I found a very convoluted way to translate these versions into large integers. The versions for applications in the AMO database have four parts at most, they are potentially alpha or beta and potentially a pre-release. In some cases we have multiple versions represented with .*, .x or + at the end.

The Toolkit docs let us translate "+" to mean "pre-release of the next version". E.g. 1.0+ is 1.1pre0. Since my primary purpose of all this is for sorting, .* and .+ may as well just be a very large "version part." Since all the version parts I deal with are a maximum of 2-digits, I turned .* and .+ into .99.

For example:

3.5+ => '03'+'05'+'99' => 030599

We also need to deal with versions that may be alpha, beta or not. If everything else is equal:

3.5a < 3.5a5 < 3.5b < 3.5b2 < 3.5 < 3.5+

We assign a single integer to represent a version's "non-alphaness":

a => 0
b => 1
non alpha/beta => 2

We assume that 3.5a = 3.5a1. Therefore:

'3.5a => 3.5.0a1 => '03'+'05'+'00'+'0'+'01' => 030500001

Similarly if it's a pre-release we assign a 0 or 1 to represent "non-pre-releaseness":

'3.5a pre2 => 3.5.0a1pre2
=> '03'+'05'+'00'+'0'+'01'+'0'+'02
=> 030500001002

So what does this get us? Integers which we can use for comparison, sorting, etc. It's a one time calculation for each version and we can do some nice SQL statements in AMO like:

mysql> SELECT version,version_int FROM appversions WHERE application_id = 1 ORDER BY version_int LIMIT 15;
+---------+--------------+
| version | version_int  |
+---------+--------------+
| 0.3     |  30000200100 |
| 0.6     |  60000200100 |
| 0.7     |  70000200100 |
| 0.7+    |  80000200000 |
| 0.8     |  80000200100 |
| 0.8+    |  90000200000 |
| 0.9     |  90000200100 |
| 0.9.0+  |  90100200000 |
| 0.9.1+  |  90200200000 |
| 0.9.2+  |  90300200000 |
| 0.9.3   |  90300200100 |
| 0.9.3+  |  90400200000 |
| 0.9.x   |  99900200100 |
| 0.9+    | 100000200000 |
| 0.10    | 100000200100 |
+---------+--------------+
15 rows in set (0.00 sec)

I can now index these integers using Sphinx and do some very easy searches for addons based on version number.