I’ve written a faster version of TransactionTestCase and packaged it with test_utils. It’s mysql specific since it relies on
SET FOREIGN_KEY_CHECKS=0 to flush the database.
The long story…
Why speed matters
We’re closing in on 300 tests for Zamboni. As of yesterday, to run our entire test suite it would have taken approximately 5 minutes. If you run tests before code-reviews, during a code-review, and before you push to master - you’ve spent about 15 minutes doing tests for a single feature or bug-fix. We have about 5 developers, so this cycle happens many times in a work day. In that time many sandwiches can be made and consumed.
Even shortcuts, like running a subset of tests will only go so far, and ultimately we do want to validate that all our tests pass for any code-change.
Testing Sphinx search with
Django recently sped up testing by running tests in a transaction. However, this means that data never gets committed to the database and therefore external tools, like the Sphinx indexer, will never see any of that data. So we resort to
TransactionTestCase which will commit the data.
TransactionTestCase is painfully slow. The accepted practice is to only use
TestCase if you want your tests to be fast. So, I decided to complain to one of our new hires and he and I decided to tinker in mysql to figure out what was slow. We discovered the following:
delete from [table] is slow
truncate [table] is slow
- … unless you
So we decided we should do our own tear down. After some tinkering with
cProfiler I discovered that
TransactionTestCase does a (slow) database
flush on setup for a test case. This wouldn’t do.
Making our own
I decided to make our own
TransactionTestCase and it would just run
SET FOREIGN_KEY_CHECKS=0 and
TRUNCATE on each table at tear down time. It would also not do a
flush on set up.
We write our tests with the idea that they clean up after themselves. Rather than having them cleanup after the last test. This is a requirement for us since
django-nose doesn’t reorder tests (nor should it) and a standard
django.test.TestCase assumes a clean database.
Looking at a single test
django.test.TransactionTestCase took ~30 seconds. Using our new
TransactionTestCase it takes ~4 seconds!
Fast tests are good
We can now run our 275 tests in ~100 seconds versus the ~300 seconds it used to take. Furthermore, skipping our sphinx tests (which are the only tests that use
TransactionTestCase) only saves us ~10seconds. That’s not a lot of overhead for better coverage.
This took me the better part of a day, but solving this now, means we’re going to more often than not run our sphinx tests all the time rather than skip them. Our QA team will assure you that search is probably the most regression prone part of our site, so running these tests are vital to quality.
If you need to use
TransactionTestCase in mysql, give ours a try.