Dave Dash

Data Anonymous

2011-03-02T00:00:00-08:00

I wrote a simple database scrubber script. It takes a yaml file that describes what scrubbing needs doing and then outputs sql that you can send to mysql. It's dreadfully simple and I'd like to see if others can make use of it.

At Mozilla we have a lot of contributors and would like them to have access to realistic data since many of our bugs are based on certain states within the data.

Making our tests run thrice as fast

2010-03-16T00:00:00-07:00

I've written a faster version of TransactionTestCase and packaged it with test_utils. It's mysql specific since it relies on SET FOREIGN_KEY_CHECKS=0 to flush the database.

The long story...

Why speed matters

We're closing in on 300 tests for Zamboni. As of yesterday, to run our entire test suite it would have taken approximately 5 minutes. If you run tests before code-reviews, during a code-review, and before you push to master - you've spent about 15 minutes doing tests for a single feature or bug-fix. We have about 5 developers, so this cycle happens many times in a work day. In that time many sandwiches can be made and consumed.

Even shortcuts, like running a subset of tests will only go so far, and ultimately we do want to validate that all our tests pass for any code-change.

Testing Sphinx search with `TransactionTestCase`

Django recently sped up testing by running tests in a transaction. However, this means that data never gets committed to the database and therefore external tools, like the Sphinx indexer, will never see any of that data. So we resort to TransactionTestCase which will commit the data.

Unfortunately TransactionTestCase is painfully slow. The accepted practice is to only use TestCase if you want your tests to be fast. So, I decided to complain to one of our new hires and he and I decided to tinker in mysql to figure out what was slow. We discovered the following:

delete from [table] is slow
truncate [table] is slow
... unless you SET FOREIGN_KEY_CHECKS=0

So we decided we should do our own tear down. After some tinkering with cProfiler I discovered that TransactionTestCase does a (slow) database flush on setup for a test case. This wouldn't do.

Making our own `TransactionTestCase`

I decided to make our own TransactionTestCase and it would just run SET FOREIGN_KEY_CHECKS=0 and TRUNCATE on each table at tear down time. It would also not do a flush on set up.

We write our tests with the idea that they clean up after themselves. Rather than having them cleanup after the last test. This is a requirement for us since django-nose doesn't reorder tests (nor should it) and a standard django.test.TestCase assumes a clean database.

Looking at a single test test_sphinx_indexer, using django.test.TransactionTestCase took ~30 seconds. Using our new TransactionTestCase it takes ~4 seconds!

Fast tests are good

We can now run our 275 tests in ~100 seconds versus the ~300 seconds it used to take. Furthermore, skipping our sphinx tests (which are the only tests that use TransactionTestCase) only saves us ~10seconds. That's not a lot of overhead for better coverage.

This took me the better part of a day, but solving this now, means we're going to more often than not run our sphinx tests all the time rather than skip them. Our QA team will assure you that search is probably the most regression prone part of our site, so running these tests are vital to quality.

If you need to use TransactionTestCase in mysql, give ours a try.

Retrieving elements in a specific order in Django and mySQL

2010-02-11T00:00:00-08:00

If you have a list of ordered ids and you want to turn them into an ordered result set you can use FIELD() in mysql:

SELECT * FROM addons
ORDER BY FIELD(id, 3, 5, 9, 1);

This is a handy trick if you use an external search engine which gives you an ordered list of ids and you want to pull out entire row sets.

We do this in addons.mozilla.org using the Django ORM like so:

The code in action.

Snow Leopard for Macports and Mysql users

2009-09-02T00:00:00-07:00

I use mysql and macports on OSX and both were broken when I upgraded to Snow Leopard.

Mysql was a quick fix:

ln -s /usr/local/mysql-5.1.35-osx10.5-x86 /usr/local/mysql

(you're installed version might be different). It turns out a symlink was removed during the Snow Leopard upgrade.

As for MacPorts, I had to install Xcode from the Snow Leopard CD, install the Snow Leopard version of MacPorts and then follow this migration guide.

tunneling and mysql

2008-10-22T00:00:00-07:00

I have a mysql server for reviewsby.us that requires tunneling, and after establishing a tunnel like so:

deveshistan:~ dash* ssh user@host -L3307:database_server:3306

I thought I could just do this:

mysql -u mysql_user -P3307 -h localhost

But I was wrong, localhost should be 127.0.0.1, otherwise mysql will do things in socket mode:

mysql -u mysql_user -P3307 -h 127.0.0.1

Unicode or How to deal with f'd up text in Django

2008-02-11T00:00:00-08:00

So I've been going out of my mind trying to figure out why something like:

Pham's would look like Phamâ€™s

After searching as much as I could into django and unicode, I found this article which worked.

As near as my feeble USA-centric "cafes not cafés" understood it, my data was encoded incorrectly, and PHP's mysql ignores encoding and expects UTF-8 whereas django tries to understand the encoding.

Coming soon to reviewsby.us

2006-10-02T00:00:00-07:00

In August I took a break from reviewsby.us only to be plagued by spam. In September, I relinquished portions of the project planning to my wife. We haven't released anything publicly, yet, but there's a lot in development.

I updated the development framework to symfony 1.0 alpha and took care of a whole slew of bugs.
Katie and I came up with a wireframe that details some of the upcoming changes.
I upgraded the user logic to take advantage of sfGuardUser, a user management plugin for symfony.

I'm in progress of writing location specific searches. I'm slow to implement. It seems that this month is far busier than I'd like, and I can rarely get in a block of enough time to just crank this out. The problem with geographic-specific searches is mySQL supports those types of queries, but it's not as easy as I'd like. Zend Search Lucene with some support from PHP, however, may yield some promising results. As always, I'll share my findings in a forthcoming tutorial.

Anyway, no visible updates on the actual site, since I didn't want to put alpha software on the live site. I'm sure by next month symfony will be ready.