Dave Dash

Testing Redis in Django

2010-12-07T00:00:00-08:00

For the Firefox Add-ons we've been using redis here and there mostly for cache, but lately for a few things we'd love to persist.

Unfortunately relying on redis does mean we need to be able to test it. Since redis touches some of our core components of the site, we can't just raise a SkipTest like we would for Sphinx search related tests. I also don't want to rely on our developers to have redis installed in order to run the test-suite.

So I built a simple Mock Redis client. It's part of our redisutils.py that handles connections to redis. If a test's setUp method calls mock_redis you'll get this phony object that can do a few minimal redis-like operations.

It works great for our specific cases, but feel free to fork it and make it better.

Note: This MockRedis is specifically designed to work with django.

Pythonic string formatting in Javascript

2010-11-19T00:00:00-08:00

We do a lot of string manipulation on the Firefox Addons site. A lot of it has to do with localization so one thing that comes up is being able to format strings. Here's a little snippet to give yourself python like string formatting:

    /* Python(ish) string formatting:
     * >>> format('{0}', ['zzz'])
     * "zzz"
     * >>> format('{x}', {x: 1})
     * "1"
     */
    function format(s, args) {
        var re = /\{([^}]+)\}/g;
        return s.replace(re, function(_, match){ return args[match]; });
    }

Faceted Search on Input

2010-10-29T00:00:00-07:00

So one trick with Sphinx search is faceted search. It's somewhat crudely implemented, by batching queries together, but does the job well. In the case of Firefox Input it can reduce quite a bit of queries (our search result pages take one batched sphinx query, and one database query now instead of 5 database queries).

Faceted search is search with filters to help narrow down a result set. I'll give you three examples. Firefox Add-ons which I wrote, Sitter City which gives you a lot of ways on narrowing down on the perfect baby sitter and ebay which lets your narrow down on auction items.

For Input we ask for the following when we do a search:

How many opinions match the term for which we are searching taking into account any preferences we have already specified (feeling, locale, operating system, date range, etc).
How many opinions show a positive sentiment, and how many show a negative sentiment?
What is the breakdown of languages for the opinion results. (I.e. how many are en-US, de, fr, etc).
How many people are on Mac, Linux or Windows.

We can batch these four queries into a single Sphinx request.

Here's our implementation.

Having done this twice, I do recognize that there is a lot of room for making the code a bit more reusable. But overall it runs fairly well.

Trimming Whitespace in Django Forms

2010-08-18T00:00:00-07:00

I've been using frameworks for a number of years. So I expect a lot of things to happen "for free" in Django. One is whitespace removal. In Delicious we had a lot of data in our database with leading and trailing whitespace. On the frontend we moved to symfony (actually ysymfony) and that prevented a lot of this.

So I was quite surprised that this is not the case with Django. So I decided we could solve this at the form level, and released a ridiculously simple library. After some googling, I found that I was not the first to do this.

Feel free to use this, fork it, submit pull requests, etc. I suspect in the future we'll handle other global form filtering - like stripping high order Unicode since MySQL is often not a fan.

The Perils of One Giant Fixture

2010-08-12T00:00:00-07:00

A while back, I thought it would be good to consolidate all the data used in testing the django-layer of AMO into a single data fixture. Unfortunately we have 600 tests, which were now loading and unloading large amounts of data each time the test would run. This made our tests take 20 minutes.

I decided to cut this down quite a bit, by using smaller fixture files. Each fixture file attempts to be a singular primary object (e.g. an Addon or a Collection or a User) and its associated supporting objects. It's far from perfect, but it's achieved tests that run in under 10 minutes.

The other side effect is tests will be simpler. They'll only include the addons needed to generate an effect, and if something can't be done easily with the fixtures in place, we can always alter the data during the test.

Django: Model Inheritance or Related Tables wrt AMO

2009-12-15T00:00:00-08:00

When I attended DjangoCon this year, I lamented that our flagship web property was difficult to test, and not fun to develop. I figured DjangoCon was a way to placate me, and Django might mean something for some of the smaller projects at Mozilla. However, Wil Clouser, our lead web developer, announced development changes for addons.mozilla.org (AMO) that says we'll be moving to Django.

Wil was open to Django and knew that's what we in the dev team wanted. Jeff spawned our foray into a new AMO with Zamboni. I've been working on some grunt-work tasks inside and outside of Django.

One of those tasks is building a transparent layer in Django to keep users logged in from our PHP-based site. That kind of problem almost immediately forces you to ask one of the most fundamental questions you ask when using any framework:

How much do I change my app, in order to accommodate the framework?

More specifically:

Should I use the django.contrib.auth User module, and to what extent?

The more we looked into what features of Django we might want to use, django.contrib.auth was heavily tied into other things we wanted, so it made sense for us to use it. The next question is whether we try the inheritance approach or do we treat our legacy users table as a sort of User Profile and utilize the User module using the related table approach?

Using model-inheritance seems real nice, because we can pretend that our legacy user is the same thing as a djaango.contrib.auth User - but this isn't true:

Looking at our users table more closely:

mysql> explain users;
+-------------------------+---------------------+------+-----+---------------------+----------------+
| Field                   | Type                | Null | Key | Default             | Extra          |
+-------------------------+---------------------+------+-----+---------------------+----------------+
| id                      | int(11) unsigned    | NO   | PRI | NULL                | auto_increment |
| email                   | varchar(255)        | YES  | UNI | NULL                |                |
| password                | varchar(255)        | NO   |     |                     |                |
| firstname               | varchar(255)        | NO   |     |                     |                |
| lastname                | varchar(255)        | NO   |     |                     |                |
| nickname                | varchar(255)        | YES  | MUL | NULL                |                |
| bio                     | int(11) unsigned    | YES  | MUL | NULL                |                |
| emailhidden             | tinyint(1) unsigned | NO   |     | 0                   |                |
| sandboxshown            | tinyint(1) unsigned | NO   |     | 0                   |                |
| homepage                | varchar(255)        | YES  |     | NULL                |                |
| display_collections     | tinyint(1) unsigned | NO   |     | 0                   |                |
| display_collections_fav | tinyint(1) unsigned | NO   |     | 0                   |                |
| confirmationcode        | varchar(255)        | NO   |     |                     |                |
| resetcode               | varchar(255)        | NO   |     |                     |                |
| resetcode_expires       | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notifycompat            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| notifyevents            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| deleted                 | tinyint(1)          | YES  |     | 0                   |                |
| created                 | datetime            | NO   | MUL | 0000-00-00 00:00:00 |                |
| modified                | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notes                   | text                | YES  |     | NULL                |                |
| location                | varchar(255)        | NO   |     |                     |                |
| occupation              | varchar(255)        | NO   |     |                     |                |
| picture_type            | varchar(25)         | NO   |     |                     |                |
| averagerating           | varchar(255)        | YES  |     | NULL                |                |
+-------------------------+---------------------+------+-----+---------------------+----------------+

You can very easily argue that this is a profile table, which happens to have credential information thrown in.

I can see overtime, I'll just struggle to keep our legacy User to act like a Django User, whereas a UserProfile is fairly standard.

Had I been writing this app from scratch, I would have chosen the UserProfile route. This is extra data which takes up a lot of space, and changes far more often than user credentials. Changing 4M+ rows sucks, by making users our UserProfile table, any changes to that table, don't tie up the table used for sign-ins.

I'm curious what other people who port their apps to Django have done.

AMO Search: Powered by Sphinx

2009-09-30T00:00:00-07:00

Last night, I gave a talk at the Addons Meetup at Threadless HQ in Chicago on the new search engine powering addons.mozilla.org. I'll recap the technical portion of the talk and give a bit more details.

First, I'd like to thank Harper and Threadless. It was a great location in the greatest city in the universe. Before and after the meetup, Harper was just an all-around great guy to hang with and the threadless headquarters was a nice hangout place for meeting people interested in addons.

Shortly after my talk, our Engineering Ops team deployed the new AMO 5.1 complete with a new Sphinx powered search engine.

So let's talk about search. Note: parts of this are a rehash of my talk, so feel free to skip around.

A bit about addons

Addons is a huge growing space. Arguably it's Mozilla's best kept secret. Sure readers of this blog probably know what Addons are, but ask people who aren't as web-savvy. Most people don't know what a browser is - and it's hard to explain it to people without getting technical.

We can just skip that step. Because Addons are small things that people can easily "get".

"It's an easy way to customize the internet when your surfing."

While perhaps not technically correct, its one way of explaining it to people. Maybe a better way is just showing people what they can do with addons.

On my flight out to Chicago, I talked to a person on the plane who didn't know what a browser was, but after showing her AMO she was really intrigued.

If everyday non-technical people can realize the potential of addons, it's only a matter of time before they start knocking down the doors to AMO.

So we better be prepared to handle them, and get them what they want.

The technical details of addons.mozilla.org

Everytime you open Firefox, it pings AMO to see if there's any updates to any of the addons that happen to be installed. Over a third of the people using Firefox have at least one addon, and Firefox is roughly 22% of the browser market. That means roughly 7% of people opening their browsers are pinging our servers for updates.

Needless to say it's a lot of traffic, and to support it we need a fair amount of hardware. AMO is clearly the largest site in the Mozilla universe in both respects.

Some stats:

1 mySQL master
4 mySQL slaves
2 memached servers
2 Sphinx indexer/search daemons
24 Web Frontend
Multiple Zeus ZXTM clusters all

Most of this is standard, we'll talk about Sphinx later, but Zeus is amazing. I didn't know what Zeus was until earlier this year when I interviewed with Mozilla's VP of Engineering Operations. All our requests get cached so much of our hits actually hit our Zeus cluster and not our web servers.

To see just how amazing they are read our mrz's ops blog.

Why search matters

If you have any kind of custom content and unique meta data a custom search solution is a must. Browsing through a site isn't going to cut it. Browsing is dead. Search is how you find things on a web site. On AMO you may see an addon that's featured somewhere, or you might want to see what's out there, but the right search query will find you the right addon in two clicks.

Improve Search

So my first job on AMO was to improve addons search. It was a vague request and born out of frustration with what we had. It wasn't a problem that certain things were indexed, or unicode didn't work, or results weren't sorted. We may have had all those problems, but as a product search needed to be replaced.

To me it meant that we needed some framework that would allow developers to quickly debug and fix any future search calamities at a moments notice.

So here were the goals I made for myself:

Do something that sucks less than what we’ve got
Do something that makes it easier to suck less in the future
Do something that’s easy to use for our operations team, web developers and most importantly, end-users
Reduce strain on our databases, developers and operations teams

Complex Data

Our data set is small (we have 5,000 addons), but there's a lot of secondary meta data about the addons that we track:

Addons work in 1 or more locales (e.g. en-US, fr, de, etc)
Addons are optionally platform specific (Linux, OS X, etc)
Addons work with one or more products (Firefox, Thunderbird, Seamonkey, Sunbird or Fennec)
Addons come in multiple flavors (extensions, themes, dictionaries and more)

We want to index all this data. Unfortunately to get at much of this data it involves either numerous queries, or numerous joins which put a strain on mysql. How much strain?

At peak we get about 10 search queries per second. If we do something smarter this won't have to cause a lot of strain.

Using Sphinx

Sphinx is an open source search indexer and daemon. It's used by Craigslist, the Pirate Bay and Mozilla Support. It was very easy to use and despite a complicated set of data and business logic, Sphinx was up to the task.

The challenges

We needed to search for addons in several languages. So indexing just addons wouldn't work, we need to make sure we have every translation of every addon indexed. For those counting, we have 5,000 addons, but 18,000 translations of addons.

All the joining and filtering that needed to be done for our old search still needs to be done, but we can do this all in one shot by using a mysql view. This view is a flat list of each translated addon as well as all meta data associated with it. This then gets fed into the sphinx indexer.

Along the way we ran into some issues which used to be dealt with outside of mysql, such as comparing versions. It was gross and quite a hack, so we turned the variety of acceptable version strings into integers.

We also learned that stemming wasn't a good idea as we assumed it would be. Stemming was great for searching through lots of text, but a great deal of addon searches were really just searches for product names, so we opted for substring searches. We'll see how that fares. There is probably room for improvement.

Much of this, however involved knowing our data, and knowing how it will be used by our users. Once we got that down, we could hammer it all out using Sphinx.

Wins

So Sphinx gains us a bit architecturally. We have a complicated query, but it only gets run once every 5 minutes versus the 180,000 times it was run "on demand."

Indexing happens rather quickly, just over a minute.

The API was a breeze to work with, and was easy to drop into our own codebase.

Because of our relatively small data set, and quick indexing, we're able to scale this simply by cloning and load balancing. Meaning, we just need to scale for traffic, but addon growth (which is slower than traffic growth) we can safely not worry about for a while.

Our ops team can monitor the sphinx clusters and just deploy additional nodes as needed.

Building a platform

What we've done is built a foundation for search. Not all the problems are gone, but a lot of the problems that our QA team finds are able to be resolved quickly. We have a nice pile of unit tests as well that help us keep our results in check when we start tweaking dials.

We even have the groundwork for some nifty advanced search syntax, that hopefully we can inject into future releases of AMO.

Enjoy. And if you find anything, let me know.

Question: Building a Better Search Engine

2009-06-18T00:00:00-07:00

So I finally have one of those jobs where I can tell people almost every little detail about what I'm doing and I'm encouraged to talk to people on the intar-webs and solicit opinions.

Uh - this is more or less how I've operated at previous jobs, just now I can be overt about it.

So my new task is to work on improving the addons.mozilla.org search engine. I've built various "search engines" over time in PHP, powered by Lucene and most recently in python using an inverted index.

One tool that I've been looking at briefly is Sphinx. While my record count is low (5-10K), Sphinx basically bakes in a lot of the things I would want in a search engine. Indexing, merging, etc.

Since I'm fairly new to the add-ons team I'm still understanding the basics of what we need:

Fast automated indexing of addons for Firefox, Thunderbird and any other Mozilla product
Quick result sets
Easy deployability
Extendible
Customized ranking
Filtering (e.g. by Firefox version, etc).
Basics: Stemming and stop-words

Whether it's Sphinx, Lucene or some home grown solution, I have all that to support. But this should be fairly straight forward. What are people's thoughts?