Dave Dash

Faceted Search on Input

2010-10-29T00:00:00-07:00

So one trick with Sphinx search is faceted search. It's somewhat crudely implemented, by batching queries together, but does the job well. In the case of Firefox Input it can reduce quite a bit of queries (our search result pages take one batched sphinx query, and one database query now instead of 5 database queries).

Faceted search is search with filters to help narrow down a result set. I'll give you three examples. Firefox Add-ons which I wrote, Sitter City which gives you a lot of ways on narrowing down on the perfect baby sitter and ebay which lets your narrow down on auction items.

For Input we ask for the following when we do a search:

How many opinions match the term for which we are searching taking into account any preferences we have already specified (feeling, locale, operating system, date range, etc).
How many opinions show a positive sentiment, and how many show a negative sentiment?
What is the breakdown of languages for the opinion results. (I.e. how many are en-US, de, fr, etc).
How many people are on Mac, Linux or Windows.

We can batch these four queries into a single Sphinx request.

Here's our implementation.

Having done this twice, I do recognize that there is a lot of room for making the code a bit more reusable. But overall it runs fairly well.

Counting Sphinx groupBy Queries

2010-10-15T00:00:00-07:00

I quickly implemented Sphinx on Input, while revisiting it, I saw that we try to answer this type of question:

Of the results displayed:

How many are happy and how many are sad?

How many are for Windows, Linux or Mac?

How many are for English, French or Japanese

Finding these involve using faceted search. Unfortunately this is a bit awkward to do using Sphinx. For the first example, happy or sad you would have to run the query like such:

Take the query, remove any filters on happiness and do a group by on happy opinions
Restore any filters on happiness and run the query as normal.
Return both the results, and the aggregate data from step 1.

Doing the group by is easy, but you only get to know how many feelings there are and what they were. In our case: happy and sad. What we really want is how many of our original search were happy and how many were sad?

I assumed something like this would work:

sphinx.SetSelect('feeling, @count')

@count is one of those magic variables that Sphinx uses. Unfortunately this doesn't work. COUNT(*) doesn't work either. Here's what did:

sphinx.SetSelect('feeling, SUM(1) AS count')

Not the straight forward mysqlish syntax I've come to expect from Sphinx, but it works.

Firefox Input, powered by Sphinx

2010-07-06T00:00:00-07:00

Thursday, I decided to take a half-day for my sanity, but saw an email about how Whoosh wasn't going to cut it for Firefox Input. I was CC'd about this and there was mention that Sphinx might be possible.

Sphinx is my hammer, and everything is a nail. So I said, let's do this. That translated into me spending my weekend, soothing my newborn and working on Sphinx. Luckily this was easy, since AMO and SUMO are both running Sphinx in a similar Django environment.

In order to move quickly, I copied code from the Zamboni project to Firefox Input. Even our deployment into staging and production wasn't done by our usual "Sphinx guy" in IT. Ultimately, everything landed in place.

So try it out and file bugs or let me know if searches don't go as planned.

Alphabetical sorting in Sphinx

2010-04-21T00:00:00-07:00

Sphinx 0.9.9 is great at searching full text, but treating actual strings as attributes takes some work.

Initially I employed the strategy of indexing my full text fields and storing them as attributes. E.g.:

sql_query = SELECT name, name AS name_ord FROM documents
sql_attr_str2ordinal = name_ord

This stores each attribute in lexical order. Meaning if your name's are Apple, Aardvark, Button, Choco-room they would be given the ordinal 2, 1, 3, 4 respectively.

However, this is case-insensitive. So trying this approach:

sql_query = SELECT name, UPPER(name) AS name_ord FROM documents
sql_attr_str2ordinal = name_ord

Will allow for case-insensitive alphabetical sorting in Sphinx.

Making our tests run thrice as fast

2010-03-16T00:00:00-07:00

I've written a faster version of TransactionTestCase and packaged it with test_utils. It's mysql specific since it relies on SET FOREIGN_KEY_CHECKS=0 to flush the database.

The long story...

Why speed matters

We're closing in on 300 tests for Zamboni. As of yesterday, to run our entire test suite it would have taken approximately 5 minutes. If you run tests before code-reviews, during a code-review, and before you push to master - you've spent about 15 minutes doing tests for a single feature or bug-fix. We have about 5 developers, so this cycle happens many times in a work day. In that time many sandwiches can be made and consumed.

Even shortcuts, like running a subset of tests will only go so far, and ultimately we do want to validate that all our tests pass for any code-change.

Testing Sphinx search with `TransactionTestCase`

Django recently sped up testing by running tests in a transaction. However, this means that data never gets committed to the database and therefore external tools, like the Sphinx indexer, will never see any of that data. So we resort to TransactionTestCase which will commit the data.

Unfortunately TransactionTestCase is painfully slow. The accepted practice is to only use TestCase if you want your tests to be fast. So, I decided to complain to one of our new hires and he and I decided to tinker in mysql to figure out what was slow. We discovered the following:

delete from [table] is slow
truncate [table] is slow
... unless you SET FOREIGN_KEY_CHECKS=0

So we decided we should do our own tear down. After some tinkering with cProfiler I discovered that TransactionTestCase does a (slow) database flush on setup for a test case. This wouldn't do.

Making our own `TransactionTestCase`

I decided to make our own TransactionTestCase and it would just run SET FOREIGN_KEY_CHECKS=0 and TRUNCATE on each table at tear down time. It would also not do a flush on set up.

We write our tests with the idea that they clean up after themselves. Rather than having them cleanup after the last test. This is a requirement for us since django-nose doesn't reorder tests (nor should it) and a standard django.test.TestCase assumes a clean database.

Looking at a single test test_sphinx_indexer, using django.test.TransactionTestCase took ~30 seconds. Using our new TransactionTestCase it takes ~4 seconds!

Fast tests are good

We can now run our 275 tests in ~100 seconds versus the ~300 seconds it used to take. Furthermore, skipping our sphinx tests (which are the only tests that use TransactionTestCase) only saves us ~10seconds. That's not a lot of overhead for better coverage.

This took me the better part of a day, but solving this now, means we're going to more often than not run our sphinx tests all the time rather than skip them. Our QA team will assure you that search is probably the most regression prone part of our site, so running these tests are vital to quality.

If you need to use TransactionTestCase in mysql, give ours a try.

AMO Search: Powered by Sphinx

2009-09-30T00:00:00-07:00

Last night, I gave a talk at the Addons Meetup at Threadless HQ in Chicago on the new search engine powering addons.mozilla.org. I'll recap the technical portion of the talk and give a bit more details.

First, I'd like to thank Harper and Threadless. It was a great location in the greatest city in the universe. Before and after the meetup, Harper was just an all-around great guy to hang with and the threadless headquarters was a nice hangout place for meeting people interested in addons.

Shortly after my talk, our Engineering Ops team deployed the new AMO 5.1 complete with a new Sphinx powered search engine.

So let's talk about search. Note: parts of this are a rehash of my talk, so feel free to skip around.

A bit about addons

Addons is a huge growing space. Arguably it's Mozilla's best kept secret. Sure readers of this blog probably know what Addons are, but ask people who aren't as web-savvy. Most people don't know what a browser is - and it's hard to explain it to people without getting technical.

We can just skip that step. Because Addons are small things that people can easily "get".

"It's an easy way to customize the internet when your surfing."

While perhaps not technically correct, its one way of explaining it to people. Maybe a better way is just showing people what they can do with addons.

On my flight out to Chicago, I talked to a person on the plane who didn't know what a browser was, but after showing her AMO she was really intrigued.

If everyday non-technical people can realize the potential of addons, it's only a matter of time before they start knocking down the doors to AMO.

So we better be prepared to handle them, and get them what they want.

The technical details of addons.mozilla.org

Everytime you open Firefox, it pings AMO to see if there's any updates to any of the addons that happen to be installed. Over a third of the people using Firefox have at least one addon, and Firefox is roughly 22% of the browser market. That means roughly 7% of people opening their browsers are pinging our servers for updates.

Needless to say it's a lot of traffic, and to support it we need a fair amount of hardware. AMO is clearly the largest site in the Mozilla universe in both respects.

Some stats:

1 mySQL master
4 mySQL slaves
2 memached servers
2 Sphinx indexer/search daemons
24 Web Frontend
Multiple Zeus ZXTM clusters all

Most of this is standard, we'll talk about Sphinx later, but Zeus is amazing. I didn't know what Zeus was until earlier this year when I interviewed with Mozilla's VP of Engineering Operations. All our requests get cached so much of our hits actually hit our Zeus cluster and not our web servers.

To see just how amazing they are read our mrz's ops blog.

Why search matters

If you have any kind of custom content and unique meta data a custom search solution is a must. Browsing through a site isn't going to cut it. Browsing is dead. Search is how you find things on a web site. On AMO you may see an addon that's featured somewhere, or you might want to see what's out there, but the right search query will find you the right addon in two clicks.

Improve Search

So my first job on AMO was to improve addons search. It was a vague request and born out of frustration with what we had. It wasn't a problem that certain things were indexed, or unicode didn't work, or results weren't sorted. We may have had all those problems, but as a product search needed to be replaced.

To me it meant that we needed some framework that would allow developers to quickly debug and fix any future search calamities at a moments notice.

So here were the goals I made for myself:

Do something that sucks less than what we’ve got
Do something that makes it easier to suck less in the future
Do something that’s easy to use for our operations team, web developers and most importantly, end-users
Reduce strain on our databases, developers and operations teams

Complex Data

Our data set is small (we have 5,000 addons), but there's a lot of secondary meta data about the addons that we track:

Addons work in 1 or more locales (e.g. en-US, fr, de, etc)
Addons are optionally platform specific (Linux, OS X, etc)
Addons work with one or more products (Firefox, Thunderbird, Seamonkey, Sunbird or Fennec)
Addons come in multiple flavors (extensions, themes, dictionaries and more)

We want to index all this data. Unfortunately to get at much of this data it involves either numerous queries, or numerous joins which put a strain on mysql. How much strain?

At peak we get about 10 search queries per second. If we do something smarter this won't have to cause a lot of strain.

Using Sphinx

Sphinx is an open source search indexer and daemon. It's used by Craigslist, the Pirate Bay and Mozilla Support. It was very easy to use and despite a complicated set of data and business logic, Sphinx was up to the task.

The challenges

We needed to search for addons in several languages. So indexing just addons wouldn't work, we need to make sure we have every translation of every addon indexed. For those counting, we have 5,000 addons, but 18,000 translations of addons.

All the joining and filtering that needed to be done for our old search still needs to be done, but we can do this all in one shot by using a mysql view. This view is a flat list of each translated addon as well as all meta data associated with it. This then gets fed into the sphinx indexer.

Along the way we ran into some issues which used to be dealt with outside of mysql, such as comparing versions. It was gross and quite a hack, so we turned the variety of acceptable version strings into integers.

We also learned that stemming wasn't a good idea as we assumed it would be. Stemming was great for searching through lots of text, but a great deal of addon searches were really just searches for product names, so we opted for substring searches. We'll see how that fares. There is probably room for improvement.

Much of this, however involved knowing our data, and knowing how it will be used by our users. Once we got that down, we could hammer it all out using Sphinx.

Wins

So Sphinx gains us a bit architecturally. We have a complicated query, but it only gets run once every 5 minutes versus the 180,000 times it was run "on demand."

Indexing happens rather quickly, just over a minute.

The API was a breeze to work with, and was easy to drop into our own codebase.

Because of our relatively small data set, and quick indexing, we're able to scale this simply by cloning and load balancing. Meaning, we just need to scale for traffic, but addon growth (which is slower than traffic growth) we can safely not worry about for a while.

Our ops team can monitor the sphinx clusters and just deploy additional nodes as needed.

Building a platform

What we've done is built a foundation for search. Not all the problems are gone, but a lot of the problems that our QA team finds are able to be resolved quickly. We have a nice pile of unit tests as well that help us keep our results in check when we start tweaking dials.

We even have the groundwork for some nifty advanced search syntax, that hopefully we can inject into future releases of AMO.

Enjoy. And if you find anything, let me know.

V is for Version Hell

2009-08-07T00:00:00-07:00

Versioning is quite difficult to deal with. Versions are nearly-numbers, but you can't quite sort them using standard numerical algorithms.

While the following is true:

1.1 < 1.2

The following is also true:

1.2 < 1.18 < 1.20

The "." is not a decimal point but a separator.

Mozilla uses a modestly complicated versioning system that involves stars, plusses, and sometimes "x".

I found a very convoluted way to translate these versions into large integers. The versions for applications in the AMO database have four parts at most, they are potentially alpha or beta and potentially a pre-release. In some cases we have multiple versions represented with .*, .x or + at the end.

The Toolkit docs let us translate "+" to mean "pre-release of the next version". E.g. 1.0+ is 1.1pre0. Since my primary purpose of all this is for sorting, .* and .+ may as well just be a very large "version part." Since all the version parts I deal with are a maximum of 2-digits, I turned .* and .+ into .99.

For example:

3.5+ => '03'+'05'+'99' => 030599

We also need to deal with versions that may be alpha, beta or not. If everything else is equal:

3.5a < 3.5a5 < 3.5b < 3.5b2 < 3.5 < 3.5+

We assign a single integer to represent a version's "non-alphaness":

a => 0
b => 1
non alpha/beta => 2

We assume that 3.5a = 3.5a1. Therefore:

'3.5a => 3.5.0a1 => '03'+'05'+'00'+'0'+'01' => 030500001

Similarly if it's a pre-release we assign a 0 or 1 to represent "non-pre-releaseness":

'3.5a pre2 => 3.5.0a1pre2
=> '03'+'05'+'00'+'0'+'01'+'0'+'02
=> 030500001002

So what does this get us? Integers which we can use for comparison, sorting, etc. It's a one time calculation for each version and we can do some nice SQL statements in AMO like:

mysql> SELECT version,version_int FROM appversions WHERE application_id = 1 ORDER BY version_int LIMIT 15;
+---------+--------------+
| version | version_int  |
+---------+--------------+
| 0.3     |  30000200100 |
| 0.6     |  60000200100 |
| 0.7     |  70000200100 |
| 0.7+    |  80000200000 |
| 0.8     |  80000200100 |
| 0.8+    |  90000200000 |
| 0.9     |  90000200100 |
| 0.9.0+  |  90100200000 |
| 0.9.1+  |  90200200000 |
| 0.9.2+  |  90300200000 |
| 0.9.3   |  90300200100 |
| 0.9.3+  |  90400200000 |
| 0.9.x   |  99900200100 |
| 0.9+    | 100000200000 |
| 0.10    | 100000200100 |
+---------+--------------+
15 rows in set (0.00 sec)

I can now index these integers using Sphinx and do some very easy searches for addons based on version number.