Dave Dash

Predictable Unpredicatability: Weeks 5-7

2010-07-18T00:00:00-07:00

Predictable unpredictability: Week 5-7

I decided to cover three weeks at a time as things are becoming a bit more stable. As always, things have gotten considerably easier. Most things aren't worrying us. Most days have been considerably easier. There's still tough days where I feel like I can't console him, but we're usually able to figure things out.

Right now the real fright is the Whooping cough. Our pediatrician, who is normally calm and relaxed about everything, was very freaked out about the epidemic.

Katie's doing a fantastic job, despite having considerable breastfeeding woes.

We also have an issue with green stools, but Katie's been altering her diet so we should see some new results.

Here's some new things we've noticed halfway past the 4th trimester:

I put in a half-day in the office.
Katie can still cook like a champ!
Siddhartha smiles a lot, especially if you play games with him.
His hands seem more coordinated.
Often he wakes up without crying.
He's slept 7+ hours on a number of occasions and we can count on at least 4 hours.
He's over 10 lbs and 22 inches.
He flails his arms less when he's nursing.
We've been managing all on our own for over two weeks.
Katie drove to somewhere with nobody in the back seat to soothe the baby.
Katie's taken a bath and shower with him a few times.
He sleeps in his swing for quite some time.
We've gone out with him on a few occasions with no major troubles.

The Python textcluster Package

2010-07-08T00:00:00-07:00

Earlier I wrote about finding the most common Firefox issues. I had wanted to automate that process and continually find these issues. Unfortunately I never had time to do this.

When they announced Firefox Input, I thought about doing this again... just with Firefox Input data but then I went on paternity leave and time kind of crept away. But I mentioned the idea this week and it piqued some interest.

So I found myself with a bit of time to work on it. The first stage was releasing a python library called textcluster.

textcluster takes the work I did earlier and makes it a bit more general purpose. The idea is I can do something like this:

docs = (
        'Every good boy does fine.',
        'Every good girl does well.',
        'Cats eat rats.',
        "Rats don't sleep.",
        )

c = Corpus()
for doc in docs:
    c.add(doc)

print c.cluster()

Which results in:

[
    (
        "Rats don't sleep.",
        {'Cats eat rats.': 0.21353467285253394}
    ),
    (
        'Every good girl does well.',
        {'Every good boy does fine.': 0.32030200927880093}
    )
]

The number is the "similarity" between the strings relative to the entire document corpus.

My next trick is to see if I can run this memory-intensive calculation over a data-set of 25,000 opinions submitted. If I can we can get some interesting data about what people think of the new Firefox beta.

Firefox Input, powered by Sphinx

2010-07-06T00:00:00-07:00

Thursday, I decided to take a half-day for my sanity, but saw an email about how Whoosh wasn't going to cut it for Firefox Input. I was CC'd about this and there was mention that Sphinx might be possible.

Sphinx is my hammer, and everything is a nail. So I said, let's do this. That translated into me spending my weekend, soothing my newborn and working on Sphinx. Luckily this was easy, since AMO and SUMO are both running Sphinx in a similar Django environment.

In order to move quickly, I copied code from the Zamboni project to Firefox Input. Even our deployment into staging and production wasn't done by our usual "Sphinx guy" in IT. Ultimately, everything landed in place.

So try it out and file bugs or let me know if searches don't go as planned.

On Buying a Thermometer

2010-07-05T00:00:00-07:00

After talking to our doulas about thermometers, I decided it'd be best to get an in-ear thermometer. I didn't at first because I heard they were expensive - but this is a device that I'll be using regularly for the foreseeable future. Our doula has had her thermometer for 8 years.

The other thing I like about the in-ear thermometer is it takes just a few seconds. I'm not really into holding a thermometer in my kids bum for a few minutes. Holding it in their armpit is less dangerous, but still slow. The in-ear thermometer works in a few seconds.

It's apparently not accurate at a young age, but I'll be using this for years to come.

I found the Braun Thermoscan at Target. Hopefully I can resist measuring Sibi's temperature every day.

Siddhartha: Weeks 3 and 4

2010-06-27T00:00:00-07:00

A week ago, I would have said this is still hell. However, since Wednesday, I've started to feel a lot more at ease.

I think a lot of what has changed is Katie and my attitudes. Katie's been a superstar at mothering and nursing Siddhartha. While I work from home, she does her best to do things without my help (although I don't mind getting her some water or food while she's busy nursing). After I finish work, I'm excited to spend time with Siddhartha. All the great feelings of being a dad that people expect you to have right away... I probably just got all those feelings in the last few days.

So here's progress in the last two weeks:

Sibi is over 8 lbs. He's clearly eating well, thanks to Katie.
Nipple shield is no longer required for feeding.
Sibi can pay more attention to us. When he's not crying, sleeping or eating he's a lot more fun.
Sibi can track objects and you with his eyes.
Sibi soothes quickly with Katie, even if it means just putting him on the breast.
Easier for me to deal when he fusses - I'm recognizing that its temporary.
We've been able to take him out a little bit each day for walks or trips to the store.
Lately he gives us a fair amount of sleep.
Our low days have been less bad than previous low days..
I got to go into the office for a bit.

We've no longer got Katie's mom to help us, and in some ways it's like the training wheels have come off. The adjustment was painful, like ripping a bandaid off, but we're managing. My mom has been in town and has helped somewhat in terms of food and giving Sibi infant massages, but we've been trying hard to do baby stuff on our own. It's paying off. We're a lot more confident.

Your objects, and all their friends

2010-06-25T00:00:00-07:00

This is complicated.

In my ever-evolving quest to get data out of the AMO database for tests, I found myself not just extracting a single object, but a list of complicated requirements in order to fully replicate behavior in production in a testing environment.

For AMO we can use fixture magic to dump a single add-on and all of it's database dependencies so that it will insert safely into a test-database. But we need more than just valid data. We need some supporting data. For an add-on to be browsable and searchable it needs to have a valid version and the version needs to have a valid file.

In our app we can check for these things by using this:

my_addon.current_version.files.all()[0]

Of course we need to check that my_addon.current_version exists and that files.all() has at least one object. This ends up being a lot of work if you just know the id of the add-on object.

So what I want is something simple, like:

./manage.py custom_dump addon 3615

And it should get me everything I need to test add-on 3615, including a Version object and any files associated with the version.

Turns out this just works. It works if you define the following settings:

## Fixture Magic
CUSTOM_DUMPS = {
    'addon': {  # ./manage.py custom_dump addon id
        'primary': 'addons.addon',  # This is our reference model.
        'dependents': [  # These are items we wish to dump.
            # Magic turns this into current_version.files.all()[0].
            'current_version.files.all.0',
        ],
        'order': ('app1.model1', 'app2.model2',),  # stuff gets sorted
        'excludes': {
            'app1.model1': ('fields', 'to', 'hide',),
        },
    }
}

Using this we're able to find out that addon means an addons.addon object and that you want the addon object with an id of 3615. From there we'll try looking for dependent objects. Using some black magic we can turn: current_version.files.all.0 into addon.objects.get(pk=3615).current_version.files.all()[0]. This gives us a file.

If we mimic our dump_object command we can get the file into the database and everything that the file needs to be valid. This in turn gives us enough data (usually) to begin testing a single addon.

So have fun with this, if you're database is remotely complicated, this can save you some time replicating it during testing.

Also note, that you can re-order models and exclude certain fields. This can make your fixtures very easy to load.

The first two weeks: what nobody tells you

2010-06-13T00:00:00-07:00

This is hell.

Apparently we are through the worst of it. Or we're just used to the sleep deprivation. I expected lack of sleep and I expected crying, but not so intense. Every parent we've run into assures us that this goes away and that it gets better. Some even say that this is just the "forth trimester" and once they pass three months the clouds will make way for the sunlight.

Time couldn't move quickly enough.

I do know that things have been getting better. Here's some improvements in the last two weeks:

Sibi is at (or possibly above) his birth weight after having lost a great deal of weight (more than the acceptable 10%).
Sibi can be exclusively breast-fed. At the hospital they supplemented with formula until Katie's milk came in.
Sibi can feed directly on the breast, rather than via bottle. This is thanks to the lactation consultants at Good Samaritan.
We no longer need to use a syringe during the feeding process. Previously feeding required both Katie, myself, a syringe and a flailing baby.
Sibi is calm when he goes to the breast. Before he would flail and fight it.
We can now watch movies while he nurses. Before we hadn't quite mastered the skill of feeding him.
Katie is now able to get a passable amount of sleep. Her OB told her she was sleep deprived a few days after Siddhartha was born.
I am able to get a reasonable amount of sleep.
We can usually tolerate his crying better and in some cases console him.

Lucky for us, Katie's mom has been helping us quite a bit while we adjust. We're still dealing with a lot of mystery and unknowns and its scary and pushes us quite a bit. His new thing is crying inconsolably at night until he's nursing again. Hoping it's a growth spurt and I'm hoping that we see some quality of life improvements in the next few weeks.

I'm nervous about returning to work. I'm hoping that my mom who arrives on Friday when Katie's mom leaves can find ways to console him as well.

On the Labor and Delivery of our Son

2010-06-06T00:00:00-07:00

Here's my subjective account of Katie's labor. I don't have exact times and details, but our doulas have it on record somewhere. Most of the links are to my twitter updates.

Friday at the OB

Many of my coworkers wished me that my paternity leave would start by the end of the weekend. I did too...

Friday May 29th we found out that Katie needed to be induced. The fluid levels in her amniotic sac were very low. Dr. Woo strongly suggested that we immediately go home, get our things and go to the hospital. I was excited, but not thrilled that there would be an induction.

We headed home, and I alerted our doulas en route. We gave ourselves 20 minutes to wrap up everything and we were on our way.

Hurry up and wait

Lucille Packard Children's Hospital's Labor and Delivery ward was booked solid, so we were put in a closet sized room and told to wait for a while. It was quite cozy with our two doulas. Katie and I hadn't eaten for a while so I decided to get us some food from Thaiphoon.

When I got back I learned that Katie would be under observation while she had a cervidil induction. Unfortunately, we were stuck in the tiny room and I was sleeping on the floor. Our nurse moved us into a much larger room and I was able to sleep comfortably once I ignored my nasty headache.

The "pit"

The next morning we made a little progress with the cervidil. However, we needed more to get Katie into active labor, so we were asked to try pitocin.

Katie was well-read on pitocin and to her, pitocin made the following scenario likely:

pitocin -> epidural -> c-section

Katie and I decided that pitocin would be the best option, despite the possible side-effects. We just asked that they introduce it slowly and increment it slowly. This eventually caused Katie's water to break.

Unfortunately, we had the nurse who thought she knew what was "best". At some point she had discretely kicked up the pit-level a bit high. This lead to Katie getting painful, rapid contractions too early.

Transition

These rapid contractions transitioned Katie into a more intense stage of labor, but they started around 2cm -- far too early.

Each contraction brought enough physical pain (10 on the scale from 1-10) even the doulas and myself couldn't ease the pain. This wasn't normal.

We opted for an epidural. The doulas had to leave for that, and Katie, who was shaking, had to be restrained in order for her to be hooked up. Once she was on it, all was well.

OMG WTF BBQ

Until it wasn't. The fetal heart rate went down quite a bit. Several nurses and our anesthesiologist arrived, brought Katie on all fours and got the fetal heart rate back up. Except for the fact that Katie could still feel her contractions in the left side of her behind, everything was fine.

The doulas came by and checked on us before deciding to catch some shut-eye in the waiting area.

Before we went to sleep, we decided we should have the anesthesiologist adjust the epidural line. He adjusted it and all was well.

Everything happens so fast

Until everything repeated itself. All of a sudden the OB, anesthesiologist, nurses were all in our room getting Katie on all fours again. The OB wasn't comfortable with the baby's heart rate so she had everyone move to the operating room. I called our doulas, letting them know "stuff is happening." I was told to hold back by a certain nurse and then she said she'd get me into scrubs.

A nurse told me to get in scrubs and the doulas were to pack up our room. When I got into scrubs, the nurse said I couldn't go into the room, since they decided to operate (meaning C-Section). I waited for a few minutes and the nurse told me to come with her -- my son was delivered:

My son was born at 9:59PM on May 29th 2010. He had a healthy 9/10 APGAR score, and looked wonderful. For the first time in my life, I cried.

Recovery

Amazingly, after all of this, Katie was fine. She recovered and we spent a few frantic nights in the hospital getting used to the fact that we were new parents. With the help of good friends and family we've made it over a week.

Introducing...

2010-05-29T00:00:00-07:00

Siddhartha Bonn Dash

7 lbs 2 oz, 19 and 3/4 in

9:59 pm 29 May 2010

No, the Baby Has Not Arrived -- an FAQ

2010-05-27T00:00:00-07:00

People tend to ask me the same questions, so I figure I can answer some basics here.

When is the baby due?

Two days ago (25 May). Due-dates, however, are 40 weeks after LMP and statistically there's less than 5% chance of a delivery on that exact day. There is, however, a 90% likelyhood that it will happen within two weeks of that date. [1]

Is it a boy or a girl?

Boy.

What's his name?

You'll find out during the birth announcement.

Should I call you to find out if the baby is born?

No, that's a terrible idea. Katie and I both hate the phone even when there isn't a child on the way. We hate it more when we're trying to relax before the baby is born. If you absolutely need to get in touch with us, try text or email.

Is there anything we can do?

Before the birth:: Play with our cat, he's needy and needs more attention than anyone can give him.
After the birth:: Email me if you are serious, there's plenty.

How is Katie feeling?

Tired - but she blogs in more detail.

Are you nervous?

No. I am not terribly excited about the sleep disruption, but I'll manage.

Excited?

Yes. I'm excited. Think of every cliché thing about new dads -- that's what I have.

Back to My Pre-California Weight

2010-05-23T00:00:00-07:00

About a month back I hit a milestone, I went down to my pre-California weight.

Here's a chart of my weight since August 2007, a few months before I moved to California.

Here are the events:

Move from Minneapolis to SF Bay
Visit Minneapolis
Visit Minneapolis
Leaving Yahoo!
Move to San José
Try iChange

The Basics

There's nothing shocking. The more I ate and less I exercised, the more I gained weight. Travel was an easy way to gain weight because I never checked in regularly. Moving to a condo with an awesome gym made regular exercise a breeze.

I tried out iChange which offered one-on-one monitoring. That got me eating healthier food, so it put most of my calories to good use:

Sugar and saturated fats were all but eliminated from my diet and slowly reintroduced when I could moderate them.
Counting calories helped reset my internal calorie tracker. I still don't do it enough, so I try to simply eat better.
Water tastes better the more I drink it. I drink a lot of soda still, but there are days when I can completely avoid it.
Exercising a lot is nice. I get to eat more. Exercising early works best for me.
Eating more whole grains felt better. The food was flavorful.
Balanced meals were tricky, though eating more fruit and vegetables made me less hungry for higher calorie items.
I had to start by doing everything: eating better, exercising, etc. Doing one thing at a time was a failure. There were no shortcuts at the beginning.

Now I stay pretty much on track with losing weight. The days I gain a bit of weight, I understand why, and I am usually able to resolve it quickly. Therefore weight loss is no longer stressful.

So with about 10lbs less of weight from the beginning of this year, I can sling on a Moby-wrap with my son (when he decides to be born) and it'll feel like nothing.

So your Wordpress has been hacked

2010-05-19T00:00:00-07:00

Last week, someone informed me that my blog had been hacked:

I'm not quite sure what the vector was. Wordpress wasn't very secure and I didn't take too many measures to harden it. A coworker of mine (on our security team) decided it might be fun to have a look at the infected Wordpress Installation.

Here's how the hack works

Your blog appears normal to you and your visitors.
Some rogue PHP code detects if Google is crawling your site and modifies the text and links so it looks like your website is a Viagra pharmacy.
The links go to other infected blogs and thus builds up page rank for this ring of blogs. So the upside is that your blog may be a top result... for VIAGRA.

Prevention

Here are some tips for prevention, but you can find a lot more by googling for Wordpress hacks. My solutions are more technical:

Don't use Wordpress - I recently switched to Jekyll since it was conceptually easier to understand, and it's coder-friendly.
Remove all users other than your own.
Change your password.
Check your code into git so you can see what files have changed.
Prevent Wordpress from writing to your webroot.

Restoration

Here's what you'll need to do to de-spam yourself:

Verify that you are still spammed by using Google Webmaster Tools|Labs|Fetch as Googlebot.
Back up your blog and database.
Move your Wordpress installation to a new directory.
Install Wordpress from scratch.
Remove all users except for yourself.
Change your password.
Copy your theme to your new installation.
Install only the plugins you need.

By step 4, you should be able to verify, using Fetch as Googlebot, that your website is no longer an online pharmacy.

Good luck.

Alphabetical sorting in Sphinx

2010-04-21T00:00:00-07:00

Sphinx 0.9.9 is great at searching full text, but treating actual strings as attributes takes some work.

Initially I employed the strategy of indexing my full text fields and storing them as attributes. E.g.:

sql_query = SELECT name, name AS name_ord FROM documents
sql_attr_str2ordinal = name_ord

This stores each attribute in lexical order. Meaning if your name's are Apple, Aardvark, Button, Choco-room they would be given the ordinal 2, 1, 3, 4 respectively.

However, this is case-insensitive. So trying this approach:

sql_query = SELECT name, UPPER(name) AS name_ord FROM documents
sql_attr_str2ordinal = name_ord

Will allow for case-insensitive alphabetical sorting in Sphinx.

Test Driven Confidence

2010-04-20T00:00:00-07:00

If you're already testing your web applications, you can skip this post.

One of the bugs I am working for AMO on involves porting a small, but moderately complicated checkbox from our PHP site and rewriting it for Django.

I decided to look at the existing implementation and found it to not work correctly at all. This was frustrating, especially since I verified that my own code worked, and that QA verified that it worked as well.

This is frustrating on many levels. Chances are some minor assumption I made changed, and thus broke this functionality. Discovering regressions is never fun, and fixing them is can be long and tedious if you can't automatically verify that everything is working correctly.

Lucky for me, coming up with tests is easy, you just do what you would do to verify the code satisfies the requirements and then code it. Sometimes the tests can take longer than writing the actual code, but ultimately you can ship with confidence. You can be confident that your feature won't break in the future without immediate notice, and you can be confident that your new code won't break anything else.

Finding the most common Firefox issues

2010-03-18T00:00:00-07:00

Cheng Wang of the Mozilla Support team, a few months back, decided to present on some design ideas for Firefox Support. One of the issues he noted was that there are a lot of repeated issues and that it would be useful to group them. Grouping them lets you see how often something occurs, and secondly let's you see how urgent it might be.

Luckily grouping and clustering text is something computers can do. So I wrote this utility that does just that.

I ran this script over a sampling of data from the last week:

Firefox won't start after update. (65 related issues)
- 5.6: Firefox updated, Gmail not delivering mails
- 5.6: How to change My Profile when Firefox won't load?
- 7.5: Once I close firefox, cannot start firefox again except system restart
- 5.6: When intalling updates Firefox uninstalls itself
- 16.8: firefox won't start after update 3.6
- 11.2: Upgraded to Firefox 3.6 and now it won't start
- 14.9: Firefox won't start with most extensions
How do I add a bookmark to more than one folder? (64 related issues)
- 8.9: How do I get my bookmarks on the bookmarks toolbar to show up as an icon only with no text?
- 7.5: Bookmarks lost after upgrade and cannot save new bookmarks
- 7.5: why do i have to add the .com now to addy's?
- 8.7: When I open sidebar to edit bookmarks, I only see the folder for Bookmarks Toolbar. I do not see a folder just called Bookmarks nor do I see my list of bookmarks, that separately appear under bookmarks menu at top of screen
- 7.5: All my impoted bookmarks go to the same webpage
How do I remove the \ask toolbar\"?" (50 related issues)
- 14.9: How do I remove an unwanted toolbar?
- 5.6: how to remove temporary video files from computer
- 7.5: I have no Toolbars or searchbar and i cant bring them back
- 7.5: nowhere says how to REMOVE a toolbar - only how to add or modify one
not able to open youtube videos (45 related issues)
- 5.6: Cannot open bookmark/history sidebar
- 5.6: After working well for years Firefox will now not open
- 6.7: opening bookmarks do not open in new tab
- 5.6: I can't watch videos on youtube with firefox, but on internet explorer i can
I cannot download Firefox 3.6. I've tried erasing the download file. I cannot get beyond logging out of Firefox. (44 related issues)
- 8.4: when downloading files firefox download manager will freeze and i will have to start over the file download
- 5.6: Firefox will not let me download anything! Can someone help?
- 6.3: cannot download epixHD.com: not compatible with firefox 3.6
- 5.0: Several tabs are coming up when i try to downloads things
- 5.0: Firefox wont open since I downloaded the 3.6 update.

The number on the right of the related issue is a score of how strongly it relates to the main issue.

The full sample is 352 clusters from an original 3000+ issues. That's a lot less stuff to go through. We can tune this to have either less clusters, and more related issues in a cluster, or we can make more clusters of issues and that might result in more accuracy.

Despite the inaccuracy of clustering we can make some general observations:

Firefox not starting is a big issue.
Bookmarks are either confusing or broken.
People don't like toolbars
Opening things is hard
Downloading things or Firefox is hard

Hopefully we can fine tune these reports and have them run regularly... maybe automatically posting to Tumblr?

A few weeks in Chrome

2010-03-17T00:00:00-07:00

A number of weeks ago I got annoyed with Firefox and decided to use Chrome for a while. This reminded me of the olden days where I used Netscape for a while, and then IE6 came out, and then Phoenix came out all the while I'd keep switching to the newest shiniest thing (note: I'm not sure about the timeline of all the browsers either).

My browser of choice since Firefox was released has been Firefox. For some time - nothing shiny in browser-land was coming out. Little UI things in Safari kept me away (and the lack of extensions), but Chrome finally showed promise. WebKit, out of process plugins, process separated tabs and now extensions. This was great.

I immediately felt like I was going to really love Chrome, and be that guy at the office (I work at Mozilla) who insists on using Chrome (just like I was that guy at Yahoo! who used Google for everything). I also wanted to answer the question as to why so many people really like Firefox in spite of Chrome's amazing speed -- even many Googlers will admit to preferring Firefox.

Overall I'm happy with Chrome, but I'm switching back to Firefox for now. Here's some things I observed:

"/" in Firefox let's you search. Which to me seems more natural than Ctrl-F. I am pleased that Chrome supports a lot of Firefox's shortcuts, like Cmd-1..9 for switching tabs, or Cmd-Shift-T for reopening a closed tab.
No titlebar... I kind of miss it.
XML is way easier to work with in Firefox. It's collapsable and always looks pretty.
Certain sites don't work well in Chrome, like Rypple or the AmericanExpress web site. Rypple surprisingly enough is built using the Google Web Toolkit. I really wish their was a "FirefoxTab" that would open certain sites in Firefox instead.
There's a number of Jetpacks and Extensions that only exist for Firefox or they are severely lacking.
- The Jetpacks for Mozilla's Bugzilla instance are awesome.
- The Delicious and AdBlock extensions on Chrome aren't nearly as good as the ones for Firefox.
Firebug is much better than the Chrome developer tools. For example, you can adjust css values instantly.
Extensions die... and don't come back without restart and appear to never have been installed unless you remember them crashing.
AwesomeBar (the location bar in Firefox) queries your history much better than the OmniBar (the location bar in Chrome)
- At first I thought this was because of a sparse history, but after several weeks I still have a hard time finding sites I've been to.
- Chrome will show you a handful of results, and then let you know there are more results, but that takes you to a new screen which is a jarring UI.
Chrome can be slow. The extensions can take a while, and even switching between tabs can be slow. At this point startup time can be a moot point.

Overall this was a healthy exercise, since I really like to be up on new browsers, and Chrome really seems like it's can be a good browser for many people. I'll probably try it again after the next major Chrome update.

Making our tests run thrice as fast

2010-03-16T00:00:00-07:00

I've written a faster version of TransactionTestCase and packaged it with test_utils. It's mysql specific since it relies on SET FOREIGN_KEY_CHECKS=0 to flush the database.

The long story...

Why speed matters

We're closing in on 300 tests for Zamboni. As of yesterday, to run our entire test suite it would have taken approximately 5 minutes. If you run tests before code-reviews, during a code-review, and before you push to master - you've spent about 15 minutes doing tests for a single feature or bug-fix. We have about 5 developers, so this cycle happens many times in a work day. In that time many sandwiches can be made and consumed.

Even shortcuts, like running a subset of tests will only go so far, and ultimately we do want to validate that all our tests pass for any code-change.

Testing Sphinx search with `TransactionTestCase`

Django recently sped up testing by running tests in a transaction. However, this means that data never gets committed to the database and therefore external tools, like the Sphinx indexer, will never see any of that data. So we resort to TransactionTestCase which will commit the data.

Unfortunately TransactionTestCase is painfully slow. The accepted practice is to only use TestCase if you want your tests to be fast. So, I decided to complain to one of our new hires and he and I decided to tinker in mysql to figure out what was slow. We discovered the following:

delete from [table] is slow
truncate [table] is slow
... unless you SET FOREIGN_KEY_CHECKS=0

So we decided we should do our own tear down. After some tinkering with cProfiler I discovered that TransactionTestCase does a (slow) database flush on setup for a test case. This wouldn't do.

Making our own `TransactionTestCase`

I decided to make our own TransactionTestCase and it would just run SET FOREIGN_KEY_CHECKS=0 and TRUNCATE on each table at tear down time. It would also not do a flush on set up.

We write our tests with the idea that they clean up after themselves. Rather than having them cleanup after the last test. This is a requirement for us since django-nose doesn't reorder tests (nor should it) and a standard django.test.TestCase assumes a clean database.

Looking at a single test test_sphinx_indexer, using django.test.TransactionTestCase took ~30 seconds. Using our new TransactionTestCase it takes ~4 seconds!

Fast tests are good

We can now run our 275 tests in ~100 seconds versus the ~300 seconds it used to take. Furthermore, skipping our sphinx tests (which are the only tests that use TransactionTestCase) only saves us ~10seconds. That's not a lot of overhead for better coverage.

This took me the better part of a day, but solving this now, means we're going to more often than not run our sphinx tests all the time rather than skip them. Our QA team will assure you that search is probably the most regression prone part of our site, so running these tests are vital to quality.

If you need to use TransactionTestCase in mysql, give ours a try.

λ^2: safely doing class based views in Django

2010-03-09T00:00:00-08:00

When I started rewriting the API for addons.mozilla.org, my views were mostly the same: get some data and render it as either JSON or XML. I also wanted all my API methods to take an api_version parameter, so I decided class based views would be best. This way my classes could just inherit from a base class.

To do this I had to implement a __call__ method. This works fine, except I wanted to store things into the class -- after all the whole point of my use of classes was to keep the code a bit more compact, and cleaner. So, why pass the api_version around everywhere? Unfortunately thread-safety comes to play, and you need a separate instance of your class for each request.

λ

Django's urlpatterns expects a callable object. So you can't give it an instance of AddonDetailView(). But you could give it a callable that creates an instance of AddonDetailView() and passes it *args and **kwargs. Luckily python has lambda functions. You can note how we solved that in our urlpatterns.

λ λ

But wrapping all your urls with lambda is tedious and remembering to pass *args and **kwargs is error prone.

So let's make a lambda function that returns... a lambda function that turns an instance of our class into a callable.

We can now return to coding and not think about thread safety.

λλλ

django-fixture-magic: Testing issues with real data.

2010-03-05T00:00:00-08:00

I just released Fixture Magic.

When dealing with legacy data, you'll run into all kinds of edge cases. Perhaps, an object might not display correctly unless it has the right parameters, or if it has null parameters it might not display at all. So when testing Django, it's nice to actually use non-dummy data.

Luckily Django has a way of pulling real data out of your database using django.core.serializers:

from addons.models import Addon
a = Addon.objects.get(id=3615)
from django.core.serializers import serialize
jsonize = lambda a: serialize("json", a, indent=4)
jsonize([a])

This solution runs well in a Django shell and can be lots of fun for the whole family... until things get complicated.

Serializing alone isn't enough.

Serializing a fixture with foreign keys means you'll have an un-loadable fixture unless you serialize the dependent fixtures. Even for one or two foreign keys, this can be a pain. For addons.mozilla.org, we have a spidery-web of dependencies: Files need a Version which needs an Addon which need Translations.

Thus begat the dump_object management command. Give it an app, model name and a pk and it will give you not only a serialized JSON of that object, but all the objects that it requires.

Example:

./manage.py dump_object files.file 64874 64876 > my_new_fixture.json

This looks for the File model in the files app and pulls out of the database Files instances with pks of 64874 and 64876. It then recursively searches for any required objects.

Too much serial

If you create a lot of fixtures, you'll eventually have overlapping serialized objects. In addons.mozilla.org we have Addons, Versions (which depend on Addons) and AddonCategorys (which depend on Addons and Categorys). If we wanted to get serialize a specific Addon, it's dependent Versions and AddonCategorys it makes sense to start with dump_objecting the related Version and then dump_objecting the AddonCategory. Both dump_object commands will fetch the Addon in question, resulting in duplicated data.

To combat this we can use merge_fixtures to dedupe our fixtures:

./manage.py dump_object versions.version 64874 > 1.json
./manage.py dump_object categories.addoncategory > 2.json
./manage.py merge_json 1.json 2.json > happy_fixture.json

This should make creating test data slightly less painful. So give it a try.

Advice for people seeking a mortgage

2010-02-28T00:00:00-08:00

So Katie and I decided to buy our first home (a condo) in California last year. From our first house (in Minneapolis) the lessons we learned was to have your 20% saved up ahead of time before taking the plunge. We're fans of traditional 30-year loans (or 15 year loans if you can swing it) as they are fairly predictable and 20% will buy your way out of mortgage insurance.

20% seems like a lot, and it is, but that seemed to be the least of our problems. Here's some more advice:

Maintain a separate account for funding your laon. Have a separate account for funding your loan. Temporarily have your paychecks go there, have your gift money go there, stock sales, etc. This way you can build up your 20% without having to worry about miscellaneous income. E.g. I had some small amounts of money enter my account due to expenses that I filed with my employer. I had to paper-trail those.
Get your funding in advance. If you can have the aforementioned account fully funded with your 20% a few months before you seek a loan, you could save a lot of trouble. Our mortgage officer seemed to only care about 2 months of history. However, this is hard since in a down economy having a large chunk of change just sitting there isn't always a great idea.
Link your accounts. Link all your accounts to the source account for your mortgage. If you have stocks, savings or other checking accounts make sure that you can quickly transfer money into your source account. Closing costs, interest rates and other seemingly minor details can force you to quickly sell something (like stock) and fund an account. I didn't have my accounts linked, and I think I managed on funding from an unlinked source at the last minute. It was very tense.
Know your transfer times. I'd give everything about a week to clear. It usually takes a few days, but every now and then it takes much longer for money to clear out of one account and into another. Keep a calendar open.
Have everything ready to go a few weeks early. The tenseness of waiting for banks and fiduciary firms to get their acts together will give you endless migraines. Also, if everything is ready in advance, you may get your closing date moved up.
Paper trail everything. If you don't need the money to make closing, don't put it in your source account. If you do need the money, show a paper trail.
Have your loan officer make a schedule for you. Our loan officer was horrible since we had no idea what needed to be done when. So we would frantically get things together for her, and realize it was all for naught. Once you have a desired closing date, tell them to give you deadlines for having things.

While I wish I had this advice when I bought our place, we did manage to get all our ducks in a row and close without delay.

Retrieving elements in a specific order in Django and mySQL

2010-02-11T00:00:00-08:00

If you have a list of ordered ids and you want to turn them into an ordered result set you can use FIELD() in mysql:

SELECT * FROM addons
ORDER BY FIELD(id, 3, 5, 9, 1);

This is a handy trick if you use an external search engine which gives you an ordered list of ids and you want to pull out entire row sets.

We do this in addons.mozilla.org using the Django ORM like so:

The code in action.

Palm Pré: A retraction, I really like it now

2010-01-11T00:00:00-08:00

So before I went on trip to Minnesota last month, I decided maybe I would give the Palm Pré another shot. After all, my parents have no internet access, so having the Pré... if I could overcome my issues, might be a welcome distraction.

Before I packed it, I updated to WebOS 1.3.x (a few days later I updated to 1.3.5) and I was blown away. The horsepower was increased by utilizing the GPU. The following problems were fixed:

* The device was no longer hot all the time
* Shutdown and startup were long, but not nearly as long as before.
* Render times were quicker
* All the elements usually rendered quickly in an app
* Network was fairly steady
* Phone calls also seemed fairly drop-free.

All these improvements helped me get over

* The tiny keyboard... not so bad in practice.
* No soft keyboard - I missed it, but I could deal without it.

Overall the device was great, it was fast enough to use, and most of the errors were annoying, but things I could deal with. Cut and paste could be improved, and I wish the USB connector was the same as the one for HTC devices (I can't keep micro or mini USB types straight).

So I love the device, and Murphy's Law dictates if work gives you a phone you don't like you get to keep it... until you start liking it again. So I sent the phone back into rotation for other people at Mozilla to try. Have at it.

Google Chrome Extensions Puzzle

2010-01-06T00:00:00-08:00

I went to Add-on-Con some weeks back to represent my employer, the Mozilla Corporation.

One of the goodies you get as a registrant was a jigsaw puzzle from the Google Chrome Extensions team.

Perfect, my wife and I love solving jigsaw puzzles. We finally finished a few days ago. Anybody who has started at all will realize the puzzle is of a QR-code. The QR-code is an extension that will eventually lead you to a prize. It was a bit of a mini-puzzle not nearly as difficult as finding the QR code.

Although finding a QR code scanner was a bit difficult, I had to borrow a HTC Magic from Justin Scott and installed a decent barcode scanner.

Django: Model Inheritance or Related Tables wrt AMO

2009-12-15T00:00:00-08:00

When I attended DjangoCon this year, I lamented that our flagship web property was difficult to test, and not fun to develop. I figured DjangoCon was a way to placate me, and Django might mean something for some of the smaller projects at Mozilla. However, Wil Clouser, our lead web developer, announced development changes for addons.mozilla.org (AMO) that says we'll be moving to Django.

Wil was open to Django and knew that's what we in the dev team wanted. Jeff spawned our foray into a new AMO with Zamboni. I've been working on some grunt-work tasks inside and outside of Django.

One of those tasks is building a transparent layer in Django to keep users logged in from our PHP-based site. That kind of problem almost immediately forces you to ask one of the most fundamental questions you ask when using any framework:

How much do I change my app, in order to accommodate the framework?

More specifically:

Should I use the django.contrib.auth User module, and to what extent?

The more we looked into what features of Django we might want to use, django.contrib.auth was heavily tied into other things we wanted, so it made sense for us to use it. The next question is whether we try the inheritance approach or do we treat our legacy users table as a sort of User Profile and utilize the User module using the related table approach?

Using model-inheritance seems real nice, because we can pretend that our legacy user is the same thing as a djaango.contrib.auth User - but this isn't true:

Looking at our users table more closely:

mysql> explain users;
+-------------------------+---------------------+------+-----+---------------------+----------------+
| Field                   | Type                | Null | Key | Default             | Extra          |
+-------------------------+---------------------+------+-----+---------------------+----------------+
| id                      | int(11) unsigned    | NO   | PRI | NULL                | auto_increment |
| email                   | varchar(255)        | YES  | UNI | NULL                |                |
| password                | varchar(255)        | NO   |     |                     |                |
| firstname               | varchar(255)        | NO   |     |                     |                |
| lastname                | varchar(255)        | NO   |     |                     |                |
| nickname                | varchar(255)        | YES  | MUL | NULL                |                |
| bio                     | int(11) unsigned    | YES  | MUL | NULL                |                |
| emailhidden             | tinyint(1) unsigned | NO   |     | 0                   |                |
| sandboxshown            | tinyint(1) unsigned | NO   |     | 0                   |                |
| homepage                | varchar(255)        | YES  |     | NULL                |                |
| display_collections     | tinyint(1) unsigned | NO   |     | 0                   |                |
| display_collections_fav | tinyint(1) unsigned | NO   |     | 0                   |                |
| confirmationcode        | varchar(255)        | NO   |     |                     |                |
| resetcode               | varchar(255)        | NO   |     |                     |                |
| resetcode_expires       | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notifycompat            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| notifyevents            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| deleted                 | tinyint(1)          | YES  |     | 0                   |                |
| created                 | datetime            | NO   | MUL | 0000-00-00 00:00:00 |                |
| modified                | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notes                   | text                | YES  |     | NULL                |                |
| location                | varchar(255)        | NO   |     |                     |                |
| occupation              | varchar(255)        | NO   |     |                     |                |
| picture_type            | varchar(25)         | NO   |     |                     |                |
| averagerating           | varchar(255)        | YES  |     | NULL                |                |
+-------------------------+---------------------+------+-----+---------------------+----------------+

You can very easily argue that this is a profile table, which happens to have credential information thrown in.

I can see overtime, I'll just struggle to keep our legacy User to act like a Django User, whereas a UserProfile is fairly standard.

Had I been writing this app from scratch, I would have chosen the UserProfile route. This is extra data which takes up a lot of space, and changes far more often than user credentials. Changing 4M+ rows sucks, by making users our UserProfile table, any changes to that table, don't tie up the table used for sign-ins.

I'm curious what other people who port their apps to Django have done.

Palm Pre: Always hot

2009-11-19T00:00:00-08:00

So I borrowed a Palm Pré that we had at Mozilla to see what it was like. I was at first very excited, I remember before the Pre was released there was a lot of talk about how awesome-fantastic it was going to be. The stories of awesomeness sort of died, and I had thought nothing of it.

Immediately upon using the Pre I figured out why. In short, it's a crappy phone. It makes a very good attempt to do a lot, but it does them with such piss-poor performance, that nothing good is noticed.

I am disappointed. It's not even in the same class as an iPhone - maybe a future generation of Palm devices will be, but not this one. I was hoping WebOS would be a good alternative to the iPhone. It looks like Google will be doing that, though their phones haven't impressed me much either. I am hoping that maybe this phone is just a dud.

Here's what I didn't like: * The Palm was always hot. * The first run experience is painfully slow. * The first run was an indicator of things to come, startup and shutdown are ridiculously slow. * Every application is slow to render. * Not all elements of an app render. * The keys are too small. Some people aren't migrating from a Treo and aren't used to mini keys. * No soft keyboard. * The palm website doesn't let you use plus-style addressing * Media Mode was not self explanatory - and forced the phone to not work. * Network would constantly drop out. Couldn't use a lot of the data features. * Phone calls didn't work so great. * Did I mention it was ass slow, even the dialing program was slow. * The battery dies quickly * I could only cut/paste when composing, but I couldn't cut a string of text from an email. * Felt too much like an old palm

Despite the sadness there were a few good things:

When it did fetch email, and other notices, it displayed them nicely
The unification of Facebook and Gmail was pretty cool - it also made me want to trim some of those friends from highschool off my facebook - I ain't ever gonna call em.
The Icons were pretty.
The card interface was interesting.
The travel charger could be modified to work in non US chargers fairly easily.

All in all, I'm glad that I had a chance to try out this device. It showed me, that user interfaces above all need to be very fast and responsive. Furthermore, everything you try to do should be done exceptionally well. I'm hopeful that software updates can alleviate some of the problem, but I think the root of the problem is slow hardware.

AMO Search: Powered by Sphinx

2009-09-30T00:00:00-07:00

Last night, I gave a talk at the Addons Meetup at Threadless HQ in Chicago on the new search engine powering addons.mozilla.org. I'll recap the technical portion of the talk and give a bit more details.

First, I'd like to thank Harper and Threadless. It was a great location in the greatest city in the universe. Before and after the meetup, Harper was just an all-around great guy to hang with and the threadless headquarters was a nice hangout place for meeting people interested in addons.

Shortly after my talk, our Engineering Ops team deployed the new AMO 5.1 complete with a new Sphinx powered search engine.

So let's talk about search. Note: parts of this are a rehash of my talk, so feel free to skip around.

A bit about addons

Addons is a huge growing space. Arguably it's Mozilla's best kept secret. Sure readers of this blog probably know what Addons are, but ask people who aren't as web-savvy. Most people don't know what a browser is - and it's hard to explain it to people without getting technical.

We can just skip that step. Because Addons are small things that people can easily "get".

"It's an easy way to customize the internet when your surfing."

While perhaps not technically correct, its one way of explaining it to people. Maybe a better way is just showing people what they can do with addons.

On my flight out to Chicago, I talked to a person on the plane who didn't know what a browser was, but after showing her AMO she was really intrigued.

If everyday non-technical people can realize the potential of addons, it's only a matter of time before they start knocking down the doors to AMO.

So we better be prepared to handle them, and get them what they want.

The technical details of addons.mozilla.org

Everytime you open Firefox, it pings AMO to see if there's any updates to any of the addons that happen to be installed. Over a third of the people using Firefox have at least one addon, and Firefox is roughly 22% of the browser market. That means roughly 7% of people opening their browsers are pinging our servers for updates.

Needless to say it's a lot of traffic, and to support it we need a fair amount of hardware. AMO is clearly the largest site in the Mozilla universe in both respects.

Some stats:

1 mySQL master
4 mySQL slaves
2 memached servers
2 Sphinx indexer/search daemons
24 Web Frontend
Multiple Zeus ZXTM clusters all

Most of this is standard, we'll talk about Sphinx later, but Zeus is amazing. I didn't know what Zeus was until earlier this year when I interviewed with Mozilla's VP of Engineering Operations. All our requests get cached so much of our hits actually hit our Zeus cluster and not our web servers.

To see just how amazing they are read our mrz's ops blog.

Why search matters

If you have any kind of custom content and unique meta data a custom search solution is a must. Browsing through a site isn't going to cut it. Browsing is dead. Search is how you find things on a web site. On AMO you may see an addon that's featured somewhere, or you might want to see what's out there, but the right search query will find you the right addon in two clicks.

Improve Search

So my first job on AMO was to improve addons search. It was a vague request and born out of frustration with what we had. It wasn't a problem that certain things were indexed, or unicode didn't work, or results weren't sorted. We may have had all those problems, but as a product search needed to be replaced.

To me it meant that we needed some framework that would allow developers to quickly debug and fix any future search calamities at a moments notice.

So here were the goals I made for myself:

Do something that sucks less than what we’ve got
Do something that makes it easier to suck less in the future
Do something that’s easy to use for our operations team, web developers and most importantly, end-users
Reduce strain on our databases, developers and operations teams

Complex Data

Our data set is small (we have 5,000 addons), but there's a lot of secondary meta data about the addons that we track:

Addons work in 1 or more locales (e.g. en-US, fr, de, etc)
Addons are optionally platform specific (Linux, OS X, etc)
Addons work with one or more products (Firefox, Thunderbird, Seamonkey, Sunbird or Fennec)
Addons come in multiple flavors (extensions, themes, dictionaries and more)

We want to index all this data. Unfortunately to get at much of this data it involves either numerous queries, or numerous joins which put a strain on mysql. How much strain?

At peak we get about 10 search queries per second. If we do something smarter this won't have to cause a lot of strain.

Using Sphinx

Sphinx is an open source search indexer and daemon. It's used by Craigslist, the Pirate Bay and Mozilla Support. It was very easy to use and despite a complicated set of data and business logic, Sphinx was up to the task.

The challenges

We needed to search for addons in several languages. So indexing just addons wouldn't work, we need to make sure we have every translation of every addon indexed. For those counting, we have 5,000 addons, but 18,000 translations of addons.

All the joining and filtering that needed to be done for our old search still needs to be done, but we can do this all in one shot by using a mysql view. This view is a flat list of each translated addon as well as all meta data associated with it. This then gets fed into the sphinx indexer.

Along the way we ran into some issues which used to be dealt with outside of mysql, such as comparing versions. It was gross and quite a hack, so we turned the variety of acceptable version strings into integers.

We also learned that stemming wasn't a good idea as we assumed it would be. Stemming was great for searching through lots of text, but a great deal of addon searches were really just searches for product names, so we opted for substring searches. We'll see how that fares. There is probably room for improvement.

Much of this, however involved knowing our data, and knowing how it will be used by our users. Once we got that down, we could hammer it all out using Sphinx.

Wins

So Sphinx gains us a bit architecturally. We have a complicated query, but it only gets run once every 5 minutes versus the 180,000 times it was run "on demand."

Indexing happens rather quickly, just over a minute.

The API was a breeze to work with, and was easy to drop into our own codebase.

Because of our relatively small data set, and quick indexing, we're able to scale this simply by cloning and load balancing. Meaning, we just need to scale for traffic, but addon growth (which is slower than traffic growth) we can safely not worry about for a while.

Our ops team can monitor the sphinx clusters and just deploy additional nodes as needed.

Building a platform

What we've done is built a foundation for search. Not all the problems are gone, but a lot of the problems that our QA team finds are able to be resolved quickly. We have a nice pile of unit tests as well that help us keep our results in check when we start tweaking dials.

We even have the groundwork for some nifty advanced search syntax, that hopefully we can inject into future releases of AMO.

Enjoy. And if you find anything, let me know.

mySQL and the grand regexp retardedness with lettercasing

2009-09-19T00:00:00-07:00

I wanted to find a list of Firefox addons that had smushed text in their title. E.g. FireBug or StumbleUpon. The normal porter stemming algorithm that Sphinx uses does not turn "StumbleUpon" into "stumbl upon" as it would with "Stumble Upon". I was hoping for, and unfortunately could not find a method to do a regular expression search/replace using mysql. If I could, I could have Sphinx read "StumbleUpon" as "Stumble Upon" and all would be well (although in theory this would backfire).

So my Plan B was to get a list of common smushed named addons (I'd say camelCase, but camelCase is different from SmushedText). Naturally I used my exceptional skill at regular expressions to concoct this query:

mysql> SELECT name FROM translated_addons WHERE name REGEXP '[a-z][A-Z][a-z]' = 1 LIMIT 10;
+------------------------+
| name                   |
+------------------------+
| Orbit Grey             | 
| Phoenity               | 
| Pinball                | 
| Qute                   | 
| FirefoxModern          | 
| Adblock                | 
| Add Bookmark Here      | 
| All-in-One Gestures    | 
| Bookmarks Synchronizer | 
| Browser Uptime         | 
+------------------------+
10 rows in set (41.28 sec)

Wait... none of these match. I scratched my head for a bit and then thought, oh wait, mysql is case insenstivie maybe it's turning [a-z][A-Z][a-z] into [a-z][a-z][a-z] ― stupid, but consistent with mysql. Then I pulled my other regexp card out of my sleve, character classes:

mysql> SELECT name FROM translated_addons WHERE name REGEXP '[[:lower:]][[:upper:]][[:lower:]]' = 1 LIMIT 10;
+------------------------+
| name                   |
+------------------------+
| Orbit Grey             |
| Phoenity               |
| Pinball                |
| Qute                   |
| FirefoxModern          |
| Adblock                |
| Add Bookmark Here      |
| All-in-One Gestures    |
| Bookmarks Synchronizer |
| Browser Uptime         |
+------------------------+
10 rows in set (12.96 sec)

No difference. Time to pull out the mysql documentation:

REGEXP is not case sensitive, except when used with binary strings.

ORLY?

Case-insenstive regular expressions when looking for [[:upper:]] or [[:lower:]]? Fine... I'll add some syntax to make you work right:

mysql> SELECT DISTINCT name FROM translated_addons WHERE name REGEXP BINARY '[[:lower:]][[:upper:]][[:lower:]]' = 1 LIMIT 10;
+---------------------------+
| name                      |
+---------------------------+
| FirefoxModern             |
| ChatZilla                 |
| ChromEdit                 |
| CuteMenus                 |
| DownloadWith              |
| easyGestures              |
| JavaScript Console Status |
| LinkVisitor               |
| OpenBook                  |
| QuickNote                 |
+---------------------------+
10 rows in set (9.68 sec)

That's more like it!

Unfortunately there's about 2609 addons matching this query and since I can't automatically fix these in mysql, I'll need to do some work:

1.  Create a new table for additional indexable data.
2.  Upon creation of any new addons with names that have SmushedText - store the "un smushed text".
3.  Index this "extras" field in Sphinx.

Bug: 517699

Getting started with pipe viewer

2009-09-16T00:00:00-07:00

Despite working on slimming the addons.mozilla.org database through dieting and exercise - I still have to occasionally do long running database tasks. So I finally tried out pipe viewer. As someone who's impatient this has been awesome. Here's some quick examples:

[root@ml-db10 sun]# pv -cN source < addons_remora.2009.09.15.sql.gz | gunzip|pv -cN gunzip > addons_remora.2009.09.15.sql
   gunzip: 10.1GB 0:06:48 [25.5MB/s] [   <=>                                  ]
   source: 3.47GB 0:06:48 [8.72MB/s] [======================>] 100%

Here we are calling pipe viewer with an argument that says to title this progress meter as source, and feeding it the gzip'd file. Pipe viewer will output two things the progress, and the actual file. We pipe that file into gunzip to unzip it, and back into another instance of pipe viewer (again with a title, of gunzip) and the standard output gets redirected to our destination file.

Now a simpler example is checking the progress of loading a large sql file into mysql:

[root@ml-db10 sun]# pv -cN sql < addons_remora.2009.09.15.sql | mysql -uroot addons_remora -p$PWD
      sql: 2.55GB 0:18:19 [5.68MB/s] [=====>                  ] 25% ETA 0:54:30

We could have probably combined all this, however:

[root@ml-db10 sun]# pv -cN source < addons_remora.2009.09.15.sql.gz | gunzip|pv -cN gunzip | mysql -u root addons_remora -p$PWD

Armed with this knowledge you can determine whether to grab a soda, a sandwich or a 2-hour lunch.

DjangoCon wrapup

2009-09-15T00:00:00-07:00

I went to DjangoCon this past week for work. Django is one of my favorite frameworks. I dropped PHP and the symfony framework to learn python and Django and I haven't looked back. I think for Mozilla's webdev team it would be the framework of choice. We have 100s of sites in many frameworks, but not a lot of resuability. Django apps are built to built to be reusable. If you build correctly you don't have to refactor, it's already done.

Here's a collection of notes I collected through the conference.

Day one

Keynote - Avi Bryant

Frameworks lock us into RDBMS = bad

This keynote mentioned the limits of modern frameworks and modern web development. Essentially frameworks are great for getting started, but as a site grows, the framework gets replaced little by little. Sometimes it can get in the way - such as with limitation of database choices.

UR doing it wrong - James Bennet

James outlined a few key problems that many Django developers run into:

learning python as you go
- doesn't work unless you know some programming upfront
- do the python tutorial
- read python in a nutshell or dive into python
Things you should know:
- subclasses
- super()
- slides went too fast... hopefully they'll be posted

All in all RTFM for python and Django :)

Learn about other py packages... like twisted. If Twisted Matrix was implemented in Ruby it would be advertised as the second coming of Christ.

Bennet's Django App review smoketests:

installable via pip, easy_install or setup.py
- read distutils-guide
- stay away from setuptools
have a README
INSTALL file list deps
Write DOCUMENTATION
- use sphinx.pocoo.org
- store it in your package and upload package docs
LICENSE (most Django apps use BSD)
Write unit tests
django-lint - to look over code (like pep8.py)

pro-django is a decent book, but not written by Bennet.

Testing - Eric Holscher

Django 1.1 encourages you to test by auto-creating tests.py.
Support for:
- Unittests
- Doctest
- Tests done in a db transacation
Test Driven Documentation (TDD + DDD)
Doctest
- easy
- can't use PDB
- Hides certain failures
Unittests via Django TestCase
- XUnit
- setup/Teardown
- adds db fixtures
- assertions
- mail testing/inbox testing
- url testing
TestCase
- Browserless Request/Response testing
- Similar to sfBrowser in symfony
Google Summer of Code (for Django 1.2)
- Coverage reports!
I need to learn PDB

Deploying Django -

Run mod_wsgi in daemon mode.

Day 2

Keynote - Ian Bicking

GNU Manifest:

I consider that the golden rule requires that if I like a program I must share it with other people who like it. Software sellers want to divide the users and conquer them, making each user agree not to share with others. I refuse to break solidarity with other users in this way. I cannot in good conscience sign a nondisclosure agreement or a software license agreement. ...

So that I can continue to use computers without dishonor, I have decided to put together a sufficient body of free software so that I will be able to get along without any software that is not free.

GNU manifesto was the idea of sharing software amongst friends
GNU has purpose - BSD, etc is just a rule - free to share
Free is not just the absense of copyright
Free is not a reaction to existing rules, but a golden rule
Not just a fight against MS
Need to find morality (the why) within the practical (the law, or what you can do)
Open sourcing closed source code isn't building open source
This might apply to Mozilla... as webkit has taken off more than Gecko.
Open source is person to person not company to company - despite sponsorship.

Using Django in Non-standard ways - Eric Florenzano

Django loosely coupled
Replace templating with Jinja 2
Copy Django methods into djangoext to easily customize Django behavior
Not using django.contrib.auth
- reasons: writing a fb app - no auth needed
- no shoehorning needed - saves time - less overhead
skip the orm?
- legacy dbs
- non standard or db (or non-relational database)
- no database
wsgi middleware has some cool shit
- repose.bitblt: autoscales images
- repose.squeeze: will concat js/css on the fly based on statistical analysis
non standard Django based apps
- YARDBird - IRCBot framework
- djng micro framework
- Jngo- singlefile cms
using admin in a nonstandard way is hard/impossible coupled with ORM and auth

Real-time web and other Buzzwords - Chris Wanstrath

more than just getting your rss feeds faster
push vs. pull
1 persisting connection vs polling
comet/flash-xml/or html5 web socket
orbitted - open source python comet server
zeddicus - does the business logic
orbitted has its own js libs - its a simple port/socket thing for your server code to deal with - not request/response.
all connections are persisting browser/orbitted orbitted/zeddicus
You can even use orbitted to connect straight to IRC and write a client in JS
Jetty also is good for comet

Also: * see webhooks * see pubsubhubub

Pluggable, Reusable Django Apps: A Use Case and Proposed Solution - Shawn Rider and Nowell Strite

PBS moved from perl to django - build a lot of reusable apps
convincing your superiors
- need a good story -
- existing base of python helped
- With Django easy to do things right without doing things slow
- be really good...
built a lot of apps to be very reusable, and pluggable based on requirements PBS had

Day 3

Keynote - Ted Leung - Sun

Django jobs are a growing market
Preferred by startups
Bespin/wave - cool
APIs are big... still
Physically impossible to create purely server-side interactions that are usable enough - rely on rest/comet/ajax/etc to bridge gap

Scaling Django Mike Malone

MM from Pownce (now sixapart)
Slides started out as "Building Scalable Web Applications"
Django didn't get in the way too much when it came to scaling
Django had tons of caching support
Cached objects by hand (memcached) and object ID lists
Use memache for sessions too
use signals to signal cache invalidation
race conditions...
Queue shit... gearman, rabbit mq, etc.
Memecached incr/decr operators are awesome
See gh/mmalone/django-caching
See gh:.../django-multidb
to combat slavelag use a memcache key to alternate between master or slave

Gearman - working later - Chris Heisel

Gearman - a work later alt to rabbit mq
Makes the most sense for something like cesium, with a bazillion worker ~~bees~~ foxes feeding off a single queue

Also at the con, I talked to someone about rebuilding large apps... and they took a PHP app and used URL rewriting to and a lot of PHP/Python glue code to build a seamless transitory app. The rule is, all new functionality was done up in python while the old app was in maintenance mode.

More talks here!

Resolving Django dumpdata errors

2009-09-13T00:00:00-07:00

Recently I recieved this wonderful piece of news when I ran ./manage.py dumpdata for the first time:

Error: Unable to serialize database: User matching query does not exist.

I knew this might not work out since I was dealing with a legacy database, but the resolution is quite simple. First I had to narrow it down to which app was causing this. Naturally I assumed it was one of the two apps I had, either common or restaurant. So I ran: ./manage.py dumpdata common and ./manage.py dumpdata restaurant. The latter had no problem whatsoever.

This made sense, since my common application was the only one that made any reference to a User. By looking in my models.py for that application, I narrowed it down to my Profile object. Sure enough, commenting it out meant I could get my data.

It ended up being a foreign key mismatch between the profile and user tables. Since this is legacy data, this mismatch made sense. A simple SELECT id,userid FROM profile WHERE userid NOT IN (SELECT id FROM auth_user) gave me a list of bad profiles. Removing them allowed me to create my Django fixtures.