Dave Dash

Naming things and a recursion

2012-01-05T00:00:00-08:00

Most Mozilla webdev projects have an awful project structure, and it's partially my fault. I'm attemtping to fix that, but I cringe every time someone creates a new playdoh (Mozilla's Django template) based project.

The typical python project

Your typical python project looks like this:

/my_project
    someotherstuff/
    docs/
    theactualthingicareabout/
    setup.py
    LICENSE

MOZtrosity

We didn't have a good guide when we first started writing Django projects, so we opted for something like this:

theactualthingicareabout/
    apps/
        foo/
        bar/
    __init__.py
    urls.py
    settings.py
    LICENSE

In otherwords, the Django Project, which is a python module, is immediately checked out. If you check this out to an invalid directory, e.g. you do something like:

git clone github.com/davedash/myawesomeproject.git will.not.work\!

Bad things will happen.

So?

To some people this seems like an easy thing to work-a-round, but when it takes three of my excellent coworkers a week to diagnose an issue, where this ended up being the root cause... well it becomes a higher priority issue.

So here's what happened this week, when we tried to deploy the new careers site to a VM hardware.

Our ops team sensibly checked out the project like so:

git clone https://github.com/mozilla/lumbergh.git careers

They did everything right. Sure, they were creative and chose careers over the default lumbergh, but they knew the shortcomings of our system and picked a name that would resolve as a valid python package.

Unfortunately we'd hit some recursion error anytime we tried to hit a URL. So we knew there was an issue with the URL resolver, but we couldn't figure it out.

Here's what the project layout looked like:

careers/ # I could be called anything, but they chose careers
    __init__.py
    apps/
        careers/  # I'm going to cause problems,
                  # but neither devs nor ops will suspect a thing! mwahaha
            __init__.py
            models.py
            urls.py
            views.py
    settings.py
    urls.py

other files.

We configure our apps/ directory to be part of our PYTHON_PATH so we can do things like from careers import views... you can probably see where this is going.

Here's the main urls.py 1:

...
urlpatterns = patterns('',
    (r'', include('careers.urls')),
)
...

The main urls.py includes careers.urls which if you look at the above project layout, resolves to two different python packages:

careers/urls.py
careers/apps/careers/urls.py

Python chose the first, and therefore urls.py kept calling upon itself.

So what did we learn?

Do better.

First of all, we need a better project layout. This will continue to cause problems for even the brightest developers.

Secondly, if you don't do this at least name apps carefully. Django's app model can be a bit much for non third party apps. Sometimes there's one app which spans the entire project, and it's tempting to call it the same name as the project (e.g. careers), but sometimes a lamer more generic name like common is better.

But really, the second point is moot if we just clean up.

18 months

2011-11-30T00:00:00-08:00

Yesterday Siddhartha turned 18 months. I remember last year running into an 18 month old at the airport and being amazed at how well she said "bébé" (she was French) when she saw Siddhartha. So I had an expectation that Siddhartha would be able to speak at least a word or two by now.

He does.

He's shy until he gets to know you, but once he warms up he'll repeat most words you tell him, or if he knows the sign he'll sign them.

He cried after slipping and falling the other day, and we did the "cell phone test" to see if he was okay. As he sniffled, he would read all the letters that I was drawing on the screen. Amazing!

He talks, he walks, he climbs, he tries to help put away dishes, and his signs are way more accurate.

This is him a year ago:

This is him today:

Choosing a New Web Stack

2011-10-09T00:00:00-07:00

I like building web sites, a lot. Usually every few years I need to re- evaluate the stack I use for a side-project. Joshua's Stack Parts site is handy for this.

At Mozilla Webdev we stick to reeds + mysql + elasticsearch + celery + rabbitmq + memcache + git + virtualenv + python + django + jinja2 + modwsgi + commander + puppet + apache + less + jquery as our go-to stack. It's tried and true and it's been working and been evolving for two years.

So I'm at re-evaluation time. The first element of the stack I needed to decide upon was the web framework. I initially thought I'd use Django, and maybe alternate a few supporting libraries just to color my experience. But Flask caught my attention.

Flask is from Pocoo who have given me great things like:

lodgeit
Werkzeug
Jinja2
Sphinx

It was a microframework, which meant that it didn't contain as many things as Django, but at the same time, I didn't use that much of Django.

Flask was a nice way to stay mostly in my comfort-zone, and in some ways, focus me on just writing an app, and not working in a framework. Since it's python, if I start to miss Django, I can probably rewrite my code without too much effort.

Overall I'm excited, and I just got past, "Hello World."

I'm not sure what my stack will look like, I'm imagining it will evolve into:

postgresql + memcache + git + flask + jinja2 + gunicorn + fabric + puppet + nginx + less + backbonejs + jquery

This will give me a chance to learn more about things I'm interested in, and utilize what I think might be better options along the stack.

Interviewing: How You Think

2011-08-28T00:00:00-07:00

My friend and Web Dev colleague, James Socol, wrote a piece recently about some hiring things. I think it's a good idea, and it would certainly make my job easier. It reminded me that I'm putting off a few posts about hiring.

I want to touch a bit on the "build/implement/write an X" style technical questions. For example:

Implement a linked list
Write a url parser
Make an image resizer in Javascript

Often we get candidates who are exceptionally smart on paper, have great Github accounts, but fall flat on the in person implementation.

For some people this is hard. So let's get over the nerves. I'm not going to tell you it's silly to be nervous or that I'm as nervous as you (I'm not), or anything that you already know. I will tell you this. If you notice yourself not thinking as sharply as you normally do, count backwards from 10. Then carry on. Repeat if necessary. Usually our interviews are 45 minutes long. 10 seconds of you counting backwards isn't going to kill us for time.

Hopefully this will calm you. If this doesn't... well it was worth a shot.

So let's get down to brass tacks. I want to know how you implement things. I want to know how you think. I don't care as much about the answer, but the correct answer is comforting.

If I tell you "implement an image resizer in Javascript" and you start diving into JavaScript code and get flustered, I can't really help you. If you can do that without getting flustered and produce something that works, I'll be impressed, but somewhat alarmed.

What I really want is you to step back from the problem. I want you to show me what the resizer will look like and how it will behave. You don't need to go into too many details, but enough details so I know you understand the problem.

If the problem is easily testable (like writing a url parser), it wouldn't hurt to have some tests to start.

Then I want you to write the code backwards from high level to low level. Write the HTML, write the initialization code, and finally write the event handlers.

This is not only easier to follow in an interview, it's basic problems solving, because you end up breaking the problem into smaller chunks.

DjangoCon Testing Tutorial

2011-08-10T00:00:00-07:00

If you want to learn all you can about testing anything in your Django App, see my tutorial at DjangoCon. It's on September 5th, it'll be 3 hours long and so far with seven sign ups it will be very hands-on.

Here's what I think I will cover, but I may change this depending on what the audience wants:

Testing issues
- ask people to fill out etherpad with issues they've run into
- ask someone to rank them in order of complexity
List an outline of topics
- post them on etherpad
- have people + them if they are interested
Testing overview
- We started in late 2009 early 2010
- Our largest project has 2500 tests
- Our next largest has 1100
- We have pretty good coverage
How testing works in Django
- I'm not 100% sure on this
- Test runner setups up a new database
- Test runner finds and runs tests
- Tests run class setup
- Test runs each test in a test case
  - Load fixtures
  - Tests run setup
  - Tests runs the test
  - Tests runs teardown
- Tests run class Teardown
- You get an F if you're bad and a . if your not.
- Now that you know it, you can hack it.
How we've hacked testing
- 2500 tests is a lot
- We no longer recreate the database when you run the test suite
- In each test case we just load the fixtures once.
- We rearrange the tests so things with the same fixture set run together
Testing tools that we use at Mozilla
- nose/django_nose
- nose plugins
  - nicedots
  - progressive
- coverage
  - git + whatchangedpy
Testing everything, no excuses
- 100% Coverage isn't important
- 80% is nice
- Good coverage on tricky things is important
- Some coverage on everything is important
- External
- If you start depending on APIs, Search or different tools you need to be able to test for them.
- Writing these test cases will take less time than this tutorial
- It will save you so much headache in the future.
- The same headaches you save yourself by writing "normal" tests
- Mock easy things
  - use a decorator on any test/view that might use redis
  - if redis isn't setup, use the mock client
  - mock client doesn't support everything,
    - just what I need to get my tests running -
    - feel free to extend it if you use it
  - Testing Redis
- Setup/Teardown for complicated tools
  - Good for search and APIs
  - Raise SkipTest (nose) if the developer doesn't want to run these tests
  - Non realtime tools
    - Testing Sphinx search
    - SetupClass
      - load fixtures
      - run indexer
      - run server
    - Sphinx server now available for all tests in your test case
    - Teardown
      - stop server
  - Real time tools
    - Nicer, data can be added in post_save signals or elsewhere in your app
    - Testing LDAP
      - Setup
        
        Remove LDAP files
        
        Load an ldif
        
        Start slapd
      - Your code can now touch LDAP
    - Testing ElasticSearch
      - We leave ES running all the time.
      - Setup
        
        Checks for ES support or SkipTest
        
        Deletes index
        
        Creates index
      - You can now read/write to ES
      - Teardown
        
        Delete's index
- Fixtures
  - Fixture Magic
  - Model Maker
- pitfalls
  - dates
  - using PDB

Groupon is poor value

2011-07-09T00:00:00-07:00

I bought a $30 coupon to Hobee's for $15. I spent $67 with friends. However, since we ordered 2 $9 mimosas at half off, we couldn't use the Groupon. Rather than Hobee's accepting the fact that I spent well over $30, they charged me the full price for the Mimosas, making my net savings at Hobee's $6. That's less than 10% of my total bill.

This isn't the first time this has happened to me. Tandori Oven of San José and even my beloved the Mmoon have weaseled out of Groupon's or similar by negating anything that remotely seems like a deal at their stores.

I feel like a bunch of restaurant owners bought into the "Groupon" marketing concept and saw their (I imagine) already slim margins, get even slimmer - so they are making Groupons unusable.

Here's the math, as I've understood it:

You buy a coupon: $10 for $20 of an item.
Groupon gets $5, the store gets $5.
The store is effectively discounting something by up to 75%.

What these stores fail to understand:

I'm discounting my meal by a non-trivial amount.
In return I buy more than I need to.
I also tip their employees a bit more.
I will stop going to their store if they attempt to nickel and dime me.

So I'm going to hold off on Groupons unless I go somewhere where I always pay the same price and they have no "specials" or "deals" that I might get side tracked with. The process is a horrible user-experience, for moderate savings.

Siddhartha: The One Year Old

2011-05-29T00:00:00-07:00

This first year of parenting started roughly and going into year two still has its challenges but the ride has become enjoyable.

He's heavier, taller, easier to deal with, can take food from multiple sources, mobile. Overall, he's a lot of fun. I enjoy my time with him whenever I can, and we even get to do "guy stuff" like grocery shopping and running errands.

He brings us and others a lot of joy and I'm sure he will continue to do so.

Better querying for ElasticSearch

2011-05-17T00:00:00-07:00

I wrote about how to write filter queries using pyes. Unfortunately after using ElasticSearch in the Add-ons Builder, I realized that our code would become unwieldy and hard to read if we kept using straight up pyes.

I prefer to write APIs so that are natural and conform to how I think, not one that simply mirrors another system.

So rather than this:

    filters = [TermFilter("platform", "all"),
               TermFilter("product", "firefox"),
               TermFilter("version", "4.0")]
    filter = ANDFilter(filters)
    q = FilteredQuery(MatchAllQuery(), filter).search()
    q.facet.add_term_facet('type')
    results = es.search(q)

I made something simpler:

    from elasticutils import S
    results = (S(platform='all', product='firefox', version='4.0')
               .facet('type').get_results)

Here were the design thoughts:

I wanted something easy to remember, S for search.
I wanted smart defaults, by default S() matches all documents, unless you give it a query term.
I didn't want to write python that looked like Java, or JSON or even a dict.
I wanted to write something that felt like the Django-ORM
Ultimately I want code that I enjoy writing.

So here it is, I expect it to power Firefox Add-ons, the Add-ons Builder and Firefox Input shortly.

This is all part of ElasticUtils. Let me know if you are using it, and pull requests are welcome!

DeliciousTube

2011-04-27T00:00:00-07:00

What do I think about 2 of the YouTube founders buying Delicious?

Everyone who used Delicious (including those of us who worked on Delicious) and was aware that Yahoo! purchased Delicious thought at some point, "Well if I ran Delicious, I'd do X." Where X is more than getting a hamster to run in a wheel. Chad Hurley and Steve Chen actually put their money where there mouths are.

I'm glad Delicious is getting a new lease on life. The problems it solves still exist on the web, and I'm hopeful that it will flourish.

Also, I'd like to publicly apologize for not implementing the following things while I worked on Delicious:

Keeping archive copies of the web sites.
Finding which links are dead or need redirecting.

In hindsight, all the things I did work on were not nearly as important. I built the product I was told to, not the product I wanted. Oops.

Filter Queries using pyes

2011-03-25T00:00:00-07:00

I've been having a tough time navigating the Elastic Search docs, but some sleuthing in the test suite for pyes has proved helpful.

If I have documents that I'd like filtered by let's say product, version and platform, I can construct a query like so:

    filters = [TermFilter("platform", "all"),
               TermFilter("product", "firefox"),
               TermFilter("version", "4.0")]
    filter = ANDFilter(filters)
    q = FilteredQuery(MatchAllQuery(), filter)
    results = es.search(q)

There is perhaps a more succinct way of doing this, but this serves my purposes.

Let's say you need facets as well:

    filters = [TermFilter("platform", "all"),
               TermFilter("product", "firefox"),
               TermFilter("version", "4.0")]
    filter = ANDFilter(filters)
    q = FilteredQuery(MatchAllQuery(), filter).search()
    q.facet.add_term_facet('type')
    results = es.search(q)

How we slug at Mozilla

2011-03-24T00:00:00-07:00

One problem we find with slug generators, is they do an awful job with unicode. For a string like this: Bän...g (bang) you get something like bng---g--bang- or at best bang-bang. But it's 2011, urls can have unicode... here's what we really want: bäng-bang.

In some cases transliteration might be acceptable. But if we look at Django's approach it fails at Russian. Here's a comparison with ours for the Russian phrase "Быстрее и лучше!" ("Faster and better!"):

>>> from django.template.defaultfilters import slugify as djslugify
>>> from slugify import slugify
>>> str = u'Быстрее и лучше!'
>>> print djslugify(str)

>>> print slugify(str)
быстрее-и-лучше

So as you can see, the built-in Django slugify could be disastrous. So take a look at ours. If you have some more test cases, please fork it.

Data Anonymous

2011-03-02T00:00:00-08:00

I wrote a simple database scrubber script. It takes a yaml file that describes what scrubbing needs doing and then outputs sql that you can send to mysql. It's dreadfully simple and I'd like to see if others can make use of it.

At Mozilla we have a lot of contributors and would like them to have access to realistic data since many of our bugs are based on certain states within the data.

Siddhartha: More time out than in

2011-03-01T00:00:00-08:00

It's hard to believe that Sidd's spent more time out of the womb than in. We've also lived in our home longer with the baby than without. Time is clearly flying.

So here's what's new:

He crawls really well.
He pulls up on everything especially people.
He babbles a lot - a few mornings he'll repeat the word "Hi" to me.
Teething seems to be worse with each tooth.
He has two bottom teeth, two top teeth and likes to use them (ouch!).
He still has dairy sensitivity, but eggs are okay.
He's great at solids... sometimes too great
- We're just sticking to once maybe twice a day.
- He really likes broccoli, peas and blueberries, but we expect this to continually change.
Sometimes he doesn't poop for a while.
His poop has become more adult like (read smelly, but solid).
His pee smells.
Sleep
- Naps are transitioning from 2/3 a day to 1/2 and it's tough - like teething. I've been able to get him to nap with not too much fuss (normally Katie gets him to nap).
- Sleep has good days and bad days. In the end, I'm glad we're still co-sleeping. He lunges for daddy when mommy is holding him and vice versa... sometimes over and over again.

These last three months have breezed on by. I think the next three months will be just as fantastic.

Bulk load ElasticSearch using pyes

2011-02-25T00:00:00-08:00

When indexing a lot of data, you can save time by bulk loading data.

With pyes you can do the following:

from pyes import ES


es = ES()
es.index(data, 'my-index', 'my-type', 1)
es.index(data, 'my-index', 'my-type', 2)
es.index(data, 'my-index', 'my-type', 3)
es.index(data, 'my-index', 'my-type', 4)

This will make 4 independent network calls.

from pyes import ES


es = ES()
es.index(data, 'my-index', 'my-type', 1, bulk=True)
es.index(data, 'my-index', 'my-type', 2, bulk=True)
es.index(data, 'my-index', 'my-type', 3, bulk=True)
es.index(data, 'my-index', 'my-type', 4, bulk=True)
es.refresh()

Will do this in one call. This is handy for those "reindex all the items we can" weekends.

Installing ElasticSearch plugins

2011-02-24T00:00:00-08:00

I'm slowly trying to familiarize myself with ElasticSearch and the pyes python interface. ElasticSearch uses a lot of plugins, and while the plugin system is easy to use, it's not obvious where to find the plugins.

They are here.

If you want to install the attachments plugin, you can do:

bin/plugin install mapper-attachments

And voilà it's installed.

What I should have done...

2011-02-22T00:00:00-08:00

From EDD:

For California workers covered by State Disability Insurance, Paid Family Leave (PFL) insurance provides up to six weeks of benefits for individuals who must take time off to care for a seriously ill child, spouse, parent, or registered domestic partner, or to bond with a new child.

I learned about this as I did my taxes this year. Had I done my 2009 taxes by myself I may have learned of this, and taken advantage of this. If/when we have a second child I plan on taking advantage of this.

Interviews with Fibonacci

2011-01-28T00:00:00-08:00

I have the privilege of interviewing many of the people who wish to be web developers at Mozilla. I unfortunately witness programmers not being able to program. My interviews are 45 minutes long. I tend to ask people about their history, experiences and about what they know. Then I give them some time to write some code.

My Question

I use questions relating to the Fibonacci sequence. E.g.:

Print n numbers of the Fibonacci sequence
Give me a list (if the person knows python) of n numbers of the Fibonacci sequence.
In rare cases, Print the nth numbers of Fibonacci.

In some cases I dumb it down... a lot:

Give me n integers. In other words f(5) = [1 1 1 1 1]. I just want to know if you can write a for-loop.

Here's the sequence:

1 1 2 3 5 8 13 21 34 ..  k[n-2] k[n-1] k[n]

The first two numbers are 0 and 1. Each successive number is the addition of the two proceeding numbers. In other words:

k[n] = k[n-2] +k[n-1]

I don't care too much about efficiency. I often don't care if it works, if they're trying.

This is a low bar

A good number of people solve this with no issues. They calculate the sequence and print out the numbers, or store them in a list. I run the code, we optimize it slightly, we move on.

Unfortunately, I expect everyone I interview to easily solve this. Quite honestly, if they can't solve this they should not have even made it past the phone screen. It's harsh and the only exception is pressure.

Pressure

Interview pressures is a real thing:

Interviewees at Mozilla are generally excited by the prospect.
People want to appear intelligent to their peers.
They had too much coffee at their hotel. There's a lot to think about, it's not a natural environment.

I try to solve this by using a comfortable room with couches. I also refrain from yelling at the candidates like a Tiger Mom. Being nervous, however, is difficult.

This problem gives a lot of people coders-block.

My advice to anyone doing technical interviews:

Practice.

Have your friends, family, fiance, whomever ask you programming questions. You can compile a huge list, and just have them pick one at random. Make the process seem natural.

Excitement

The best people who solve these problems are the ones who think problems are exciting. They enjoy challenge. They are also find the interview exciting, they are entering in with a good attitude.

It's not about intelligence. Intelligence is rarely a factor. It's about effort, and effort is easy to exert if you find something exciting.

If they are excited to ace a simple interview question, I can usually rely on them to do their jobs, and in many cases turn to them to come up with solutions to problems I might be running into.

The excited interviewer asks about what the solutions should entail, and the rules:

Are we starting at zero or one?
Do you want a list, or a generator?
Do you want to print this out?

They eagerly churn out code. They check it with some basic tests. They let me know it works with confidence. Occasionally they stumble and sometimes come out with the wrong answer, but with some assistance they could usually figure it out. I can use their failures as a segue to talk about testing and quality assurance.

How do you think?

I don't ask this problem, because I'm dying to know the 12th number of the fibonacci sequence. I already know that's 144. I want to know how people solve problems. How do they take a set of requirements and implement them.

Our jobs as software developers are to take requirements or problems, find a suitable solution and write code to solve it.

This is my approach. I know the first number is 1 and the next number is 1, therefore I can store those and get the third number by adding them. I shift the variables around in a for loop and I can get this:

cur = next = 1
for 1..n:
    print cur
    last = cur
    cur = next
    next += last

That's just pseudo-code. I'm not a math and/or computer science genius. I can, however, take requirements and imagine how I would solve them if I had to do it by hand, and then transcribe that process.

Likewise if someone tells me, we need to know the average rating that people gave Firefox in the area of start-up time each day, I can figure out that problem and then implement it using software.

So now I have to preface my interviews with, "If you read my post about the Fibonacci sequence, let me know."

Testing Redis in Django

2010-12-07T00:00:00-08:00

For the Firefox Add-ons we've been using redis here and there mostly for cache, but lately for a few things we'd love to persist.

Unfortunately relying on redis does mean we need to be able to test it. Since redis touches some of our core components of the site, we can't just raise a SkipTest like we would for Sphinx search related tests. I also don't want to rely on our developers to have redis installed in order to run the test-suite.

So I built a simple Mock Redis client. It's part of our redisutils.py that handles connections to redis. If a test's setUp method calls mock_redis you'll get this phony object that can do a few minimal redis-like operations.

It works great for our specific cases, but feel free to fork it and make it better.

Note: This MockRedis is specifically designed to work with django.

Siddhartha: The Sixth Month

2010-11-29T00:00:00-08:00

Siddhartha: The sixth month

Six months ago, we were playing the waiting game. Katie woke up in a hospital bed and was undergoing pregnancy induction, and I woke up in an uncomfortable nook -- not realizing that long restful nights of sleep were gone. Today, I woke up on a couch with a six month old sleeping on my belly.

This month was much like last month, he just got better at everything. His rolling is now a nuisance on the changing table. His crawling is still army like and now he gets into stuff or terrorizes the cat. He bounces up on all fours. He can also sit up without support which means we can put him in a high chair or in a shopping cart. And that tooth that's been threatening to emerge since 2.5 months? It's broken ground.

We did give him an avocado for Thanksigiving and we tried once more on Saturday, but he (and it may be sheer coincidence) got some redness on his face, excessive spit-up and sneezing, so we're going to try a new vegetable later this week.

Here is my son vs a robot:

Pythonic string formatting in Javascript

2010-11-19T00:00:00-08:00

We do a lot of string manipulation on the Firefox Addons site. A lot of it has to do with localization so one thing that comes up is being able to format strings. Here's a little snippet to give yourself python like string formatting:

    /* Python(ish) string formatting:
     * >>> format('{0}', ['zzz'])
     * "zzz"
     * >>> format('{x}', {x: 1})
     * "1"
     */
    function format(s, args) {
        var re = /\{([^}]+)\}/g;
        return s.replace(re, function(_, match){ return args[match]; });
    }

My Refrigerator's Sabbath Mode

2010-11-10T00:00:00-08:00

From my refrigerator's manual:

The Sabbath mode feature turns off all lights, disables the water and ice dispenser, disables all alarms and tones, disables the automatic defrost feature and disables the control panel.

Using git to borrow from the future

2010-11-09T00:00:00-08:00

One of the great features of git is the ability to re-order commits, break commits into parts, and merge commits together.

Assuming that my master branch is a pristine copy of the site and an ancestor of mybranch we can re-order commits by running:

git rebase -i master

This will take all the commits in your current branch (mybranch) that are built upon master and allow you to reorder or edit them individually, remove them or squash them.

For example you might get:

pick 123abcd New feature supreme
pick 123abce Whitepsace fixes
pick 2222222 Rename functions.
pick 123abcf rebase me

The last commit listed is the latest commit and is where mybranch's HEAD points to.

You can edit this like so:

pick 123abce Whitepsace fixes
pick 2222222 Rename functions.
pick 123abcd New feature supreme
f 123abcf rebase me

This will re-order history so the first three items happen and the "rebase me" commit just gets rolled into the "New feature supreme". Note since this is a rebase the commit hashes will change. Let's say history is now this (reverse chronological):

323abcd New feature supreme
3222222 Rename functions.
323abce Whitepsace fixes

Great? Almost.

There's a likely hood that your apps' unit tests will not pass after the commit "Rename functions". Some functions may have been renamed somewhere later in mybranch possibly in "rebase me" which is now a part of "New feature supreme."

This is a mess, but we can run git rebase -i master again and edit the "Rename functions." commit. If you run a test-suite and things fail you can "borrow from the future". You see at this point Rename functions is something that happened in the past. mybranch's head is now New feature supreme which is the future. We can pick and choose little changes with some git-fu.

While rebasing in the Rename functions. we might notice that we forgot to rename a call, but we remembered to rename this call at some point in mybranch. We can simply do this:

git checkout -p mybranch [paths]

This will let you interactively select chunks of code from the head of mybranch and put it into your specific commit Rename functions..

You can narrow this down by specifying some paths.

Once we finish rebasing we'll have a commit history that is logically ordered and have all tests passing tests.

Siddhartha: The Fifth Month

2010-10-30T00:00:00-07:00

This month has been my favorite thus far. The first month was exciting, but nerve racking. Two was full of changes. Three was somewhat boring, but we did get sleep. Four we lost all our sleep. In this month a lot of stuff happened and we got used to the sleep regression.

This month we are also sleeping differently. He sleeps around 7 or 8PM and wakes up a few times at night. We sleep around 9 or 10PM. Katie nurses him in bed the first few times he wakes. Once it's 4am I take him out so Katie can get an hour or two of uninterrupted sleep (or more on the weekend). I'll either get him to fall back asleep on me in the couch or in his swing. Sometimes I'll need to change his diaper, or bottle feed him a bit.

But the sleep regression wasn't all for naught. He now rolls, which is amazing, and he can army crawl:

I think I'll be spending a lot of weekends hiding wires and cables and other things that babies think are delicious. Crawling (and perhaps teething) have led to a lot more spit-up too.

My biggest problem has been one of scheduling. I wake up around 5 or 6AM, get ready for work, wait for Katie to get ready, then go to work, come back around 6PM and start putting him to sleep. I don't get enough time to spend with him on weekdays, and I feel like I'm rushing to and from a job I really enjoy. I'm really hoping that he gets to sleep easier, so I won't feel rushed each day.

In any case, this next month should be exciting - our plan is to give this guy an avocado for Thanksgiving.

Faceted Search on Input

2010-10-29T00:00:00-07:00

So one trick with Sphinx search is faceted search. It's somewhat crudely implemented, by batching queries together, but does the job well. In the case of Firefox Input it can reduce quite a bit of queries (our search result pages take one batched sphinx query, and one database query now instead of 5 database queries).

Faceted search is search with filters to help narrow down a result set. I'll give you three examples. Firefox Add-ons which I wrote, Sitter City which gives you a lot of ways on narrowing down on the perfect baby sitter and ebay which lets your narrow down on auction items.

For Input we ask for the following when we do a search:

How many opinions match the term for which we are searching taking into account any preferences we have already specified (feeling, locale, operating system, date range, etc).
How many opinions show a positive sentiment, and how many show a negative sentiment?
What is the breakdown of languages for the opinion results. (I.e. how many are en-US, de, fr, etc).
How many people are on Mac, Linux or Windows.

We can batch these four queries into a single Sphinx request.

Here's our implementation.

Having done this twice, I do recognize that there is a lot of room for making the code a bit more reusable. But overall it runs fairly well.

Counting Sphinx groupBy Queries

2010-10-15T00:00:00-07:00

I quickly implemented Sphinx on Input, while revisiting it, I saw that we try to answer this type of question:

Of the results displayed:

How many are happy and how many are sad?

How many are for Windows, Linux or Mac?

How many are for English, French or Japanese

Finding these involve using faceted search. Unfortunately this is a bit awkward to do using Sphinx. For the first example, happy or sad you would have to run the query like such:

Take the query, remove any filters on happiness and do a group by on happy opinions
Restore any filters on happiness and run the query as normal.
Return both the results, and the aggregate data from step 1.

Doing the group by is easy, but you only get to know how many feelings there are and what they were. In our case: happy and sad. What we really want is how many of our original search were happy and how many were sad?

I assumed something like this would work:

sphinx.SetSelect('feeling, @count')

@count is one of those magic variables that Sphinx uses. Unfortunately this doesn't work. COUNT(*) doesn't work either. Here's what did:

sphinx.SetSelect('feeling, SUM(1) AS count')

Not the straight forward mysqlish syntax I've come to expect from Sphinx, but it works.

The Fourth Month

2010-09-29T00:00:00-07:00

This was the month where we learned about non-linear progression. For most of the second and third months we thought we had it good in terms of sleep. Usually 5-6 hours, sometimes 8 or 9. That usually meant one long stretch and a second smaller stretch of sleep.

In comes the 19 week "wonder-week"... with its characteristic "sleep regression". This is a culmination of crazy mental and physical development that is downright awesome to watch during playtime, but can lead to waking up every few hours at night. We're still in the midst of it and managing through it, but it was real rough at first. Others say in a few weeks or months sleep will return to "normal." I can't wait. On top of that we've been dealing with teething.

However, I think both Katie and I will agree, Sibi's a lot easier to manage most of the day. He entertains himself, he can play a lot longer, and he's very very chatty in the right company.

Siddhartha: 3 months old

2010-08-29T00:00:00-07:00

Finally!

Lately the week-to-week changes have been hard to notice. Obviously the occasional bad day or three where he's developing mentally or physically are noticeable, but for the most part everything is a smooth transition.

For example, he has gradually been spending more time on his tummy, and he will sometimes smile, and in one case, while on my tummy, he started having a conversation with me.

Naps and sleep in general has been all over the place (and almost impossible in the car). For a few days nursing him to sleep wasn't working. In some ways he sleeps a lot easier, we used to have to bounce and shush him while swaddled to get him to sleep. These days we've had mixed success with less elaborate measures -- the swing has been pretty good to us.

Sleep of course sets our mood, so if we can't get enough it makes everything awful. Luckily we've been able to get okay sleep on most days (averaging 4 hour stretch at night).

I like to look forward to things. In some ways it's a coping mechanism to get through tough times. In some ways, it's because I am really looking forward to an event in the future. I diligently mark on a calendar when upcoming milestones might happen, and other notable events.

During his third month, I was able to enjoy the moment and not constantly look at what's coming next. Although with the sleep issues, and occasional cranky days, it's hard to not be comforted by the future. The future months where naps are easier, sleep averages to over six hours a stretch and the baby can occupy himself for hours at a time.

I'm also looking forward to the time when we can do "kid stuff" that he'll enjoy (versus the ones we do just to get out of the house): zoos, parks, amusement parks, malls and street fairs.

Either way, next 3 months should be awesome.

Trimming Whitespace in Django Forms

2010-08-18T00:00:00-07:00

I've been using frameworks for a number of years. So I expect a lot of things to happen "for free" in Django. One is whitespace removal. In Delicious we had a lot of data in our database with leading and trailing whitespace. On the frontend we moved to symfony (actually ysymfony) and that prevented a lot of this.

So I was quite surprised that this is not the case with Django. So I decided we could solve this at the form level, and released a ridiculously simple library. After some googling, I found that I was not the first to do this.

Feel free to use this, fork it, submit pull requests, etc. I suspect in the future we'll handle other global form filtering - like stripping high order Unicode since MySQL is often not a fan.

The Perils of One Giant Fixture

2010-08-12T00:00:00-07:00

A while back, I thought it would be good to consolidate all the data used in testing the django-layer of AMO into a single data fixture. Unfortunately we have 600 tests, which were now loading and unloading large amounts of data each time the test would run. This made our tests take 20 minutes.

I decided to cut this down quite a bit, by using smaller fixture files. Each fixture file attempts to be a singular primary object (e.g. an Addon or a Collection or a User) and its associated supporting objects. It's far from perfect, but it's achieved tests that run in under 10 minutes.

The other side effect is tests will be simpler. They'll only include the addons needed to generate an effect, and if something can't be done easily with the fixtures in place, we can always alter the data during the test.

Siddhartha: Weeks 8-10

2010-08-08T00:00:00-07:00

Things have been mostly stable. Sleep has been somewhat all over the place -- 3-7 hours or so. The worst thing is dealing with the "witching hour." Siddhartha's not colicky, but definitely fussy in the early evenings, in cars and just before he does his big sleep for the night. Now that we know to expect this, it's not as alarming.

We also learned that Fisherman's Wharf might be less crowded on the weekdays, but then you have to deal with weekday rush hour. Rush hour combined with stopping to assuage the baby twice made our trip down the peninsula take 4 long hours. This was probably the most stressful four hours we've had recently.

We also learned that Music in the Park in San José is not great for families. Sibi in his moby wrap was one of the youngest, but definitely not the only child, present at the event. Unfortunately San José has a central park designed for a much smaller city, so the park was jam packed. Katie and I foolishly got ice cream sandwiches from Treatbot, but it was by no means worth it and then realized that we can't both eat treats and take care of junior at the same time.

So just past 2 months here's what we've accomplished:

I'm starting to go into work a few days a week.
The babies stool is less troublesome since Katie's avoided dairy. She's re-introduced yoghurt and some cheese and we still seem to be doing alright.
Sidd's less at risk of whooping cough thanks to his DTaP vaccine.
Sidd spins in a circle (may have been doing this at 6 or 7 weeks) on his activity mat. [1]
He can belly crawl once in a while. [1]
There is some predictability in his schedule.
Dealt well with being a tourist at Fisherman's Wharf. He even let us go in a submarine.
Although he fusses at night, he's a bit easier to handle in a Moby Wrap with me.
Katie's had less breastfeeding issues.
He's over 10.5 lbs and 23 inches.
He now sleeps in an attached co-sleeper.
He's met both his grandpa and cousin (on Katie's side).