Dave Dash

Siddhartha: 3 months old

2010-08-29T00:00:00-07:00

Finally!

Lately the week-to-week changes have been hard to notice. Obviously the occasional bad day or three where he's developing mentally or physically are noticeable, but for the most part everything is a smooth transition.

For example, he has gradually been spending more time on his tummy, and he will sometimes smile, and in one case, while on my tummy, he started having a conversation with me.

Naps and sleep in general has been all over the place (and almost impossible in the car). For a few days nursing him to sleep wasn't working. In some ways he sleeps a lot easier, we used to have to bounce and shush him while swaddled to get him to sleep. These days we've had mixed success with less elaborate measures -- the swing has been pretty good to us.

Sleep of course sets our mood, so if we can't get enough it makes everything awful. Luckily we've been able to get okay sleep on most days (averaging 4 hour stretch at night).

I like to look forward to things. In some ways it's a coping mechanism to get through tough times. In some ways, it's because I am really looking forward to an event in the future. I diligently mark on a calendar when upcoming milestones might happen, and other notable events.

During his third month, I was able to enjoy the moment and not constantly look at what's coming next. Although with the sleep issues, and occasional cranky days, it's hard to not be comforted by the future. The future months where naps are easier, sleep averages to over six hours a stretch and the baby can occupy himself for hours at a time.

I'm also looking forward to the time when we can do "kid stuff" that he'll enjoy (versus the ones we do just to get out of the house): zoos, parks, amusement parks, malls and street fairs.

Either way, next 3 months should be awesome.

Trimming Whitespace in Django Forms

2010-08-18T00:00:00-07:00

I've been using frameworks for a number of years. So I expect a lot of things to happen "for free" in Django. One is whitespace removal. In Delicious we had a lot of data in our database with leading and trailing whitespace. On the frontend we moved to symfony (actually ysymfony) and that prevented a lot of this.

So I was quite surprised that this is not the case with Django. So I decided we could solve this at the form level, and released a ridiculously simple library. After some googling, I found that I was not the first to do this.

Feel free to use this, fork it, submit pull requests, etc. I suspect in the future we'll handle other global form filtering - like stripping high order Unicode since MySQL is often not a fan.

The Perils of One Giant Fixture

2010-08-12T00:00:00-07:00

A while back, I thought it would be good to consolidate all the data used in testing the django-layer of AMO into a single data fixture. Unfortunately we have 600 tests, which were now loading and unloading large amounts of data each time the test would run. This made our tests take 20 minutes.

I decided to cut this down quite a bit, by using smaller fixture files. Each fixture file attempts to be a singular primary object (e.g. an Addon or a Collection or a User) and its associated supporting objects. It's far from perfect, but it's achieved tests that run in under 10 minutes.

The other side effect is tests will be simpler. They'll only include the addons needed to generate an effect, and if something can't be done easily with the fixtures in place, we can always alter the data during the test.

Siddhartha: Weeks 8-10

2010-08-08T00:00:00-07:00

Things have been mostly stable. Sleep has been somewhat all over the place -- 3-7 hours or so. The worst thing is dealing with the "witching hour." Siddhartha's not colicky, but definitely fussy in the early evenings, in cars and just before he does his big sleep for the night. Now that we know to expect this, it's not as alarming.

We also learned that Fisherman's Wharf might be less crowded on the weekdays, but then you have to deal with weekday rush hour. Rush hour combined with stopping to assuage the baby twice made our trip down the peninsula take 4 long hours. This was probably the most stressful four hours we've had recently.

We also learned that Music in the Park in San José is not great for families. Sibi in his moby wrap was one of the youngest, but definitely not the only child, present at the event. Unfortunately San José has a central park designed for a much smaller city, so the park was jam packed. Katie and I foolishly got ice cream sandwiches from Treatbot, but it was by no means worth it and then realized that we can't both eat treats and take care of junior at the same time.

So just past 2 months here's what we've accomplished:

I'm starting to go into work a few days a week.
The babies stool is less troublesome since Katie's avoided dairy. She's re-introduced yoghurt and some cheese and we still seem to be doing alright.
Sidd's less at risk of whooping cough thanks to his DTaP vaccine.
Sidd spins in a circle (may have been doing this at 6 or 7 weeks) on his activity mat. [1]
He can belly crawl once in a while. [1]
There is some predictability in his schedule.
Dealt well with being a tourist at Fisherman's Wharf. He even let us go in a submarine.
Although he fusses at night, he's a bit easier to handle in a Moby Wrap with me.
Katie's had less breastfeeding issues.
He's over 10.5 lbs and 23 inches.
He now sleeps in an attached co-sleeper.
He's met both his grandpa and cousin (on Katie's side).

Bottles vs. Diapers

2010-08-03T00:00:00-07:00

One of the things I've heard from other fathers is that they love bottle-feeding. I hate it.

We've introduced a bottle of pumped breast milk so that we can hire a baby sitter when Katie has doctor's appointments, or when we decide to have a date night. Recently, I've had a difficult time feeding him. He initially took to the bottle alright, although he fussed anytime I had to adjust his position or wipe him with a burp rag. This past weekend was another matter altogether. I had zero luck feeding him, but Katie, after nursing him, got him to take the bottle. Even good feedings are cumbersome and slow, and involved lots of spilled milk.

Strangely enough, I enjoy diaper changing time. If you asked me before the pregnancy, when I was so naive, what I'd be dreading, I would have said diapers. Turns out, diapers are easy at this stage. Old one comes off, new one goes on. There is some form of wiping that has to happen in many cases, but all in all it's no big deal. What makes it enjoyable is that Siddhartha is generally very happy at the changing table.

I'm sure as Siddhartha grows up bottle-time will be easier:

More fluid hand movements will mean he'll stop whacking the bottle out of his mouth.
Being able to sit him up will make help regulate the milk flow better.
Him being less temperamental will make him not fuss on the bottle or when it gets removed.

What do other infant-dad's enjoy?

Predictable Unpredicatability: Weeks 5-7

2010-07-18T00:00:00-07:00

Predictable unpredictability: Week 5-7

I decided to cover three weeks at a time as things are becoming a bit more stable. As always, things have gotten considerably easier. Most things aren't worrying us. Most days have been considerably easier. There's still tough days where I feel like I can't console him, but we're usually able to figure things out.

Right now the real fright is the Whooping cough. Our pediatrician, who is normally calm and relaxed about everything, was very freaked out about the epidemic.

Katie's doing a fantastic job, despite having considerable breastfeeding woes.

We also have an issue with green stools, but Katie's been altering her diet so we should see some new results.

Here's some new things we've noticed halfway past the 4th trimester:

I put in a half-day in the office.
Katie can still cook like a champ!
Siddhartha smiles a lot, especially if you play games with him.
His hands seem more coordinated.
Often he wakes up without crying.
He's slept 7+ hours on a number of occasions and we can count on at least 4 hours.
He's over 10 lbs and 22 inches.
He flails his arms less when he's nursing.
We've been managing all on our own for over two weeks.
Katie drove to somewhere with nobody in the back seat to soothe the baby.
Katie's taken a bath and shower with him a few times.
He sleeps in his swing for quite some time.
We've gone out with him on a few occasions with no major troubles.

The Python textcluster Package

2010-07-08T00:00:00-07:00

Earlier I wrote about finding the most common Firefox issues. I had wanted to automate that process and continually find these issues. Unfortunately I never had time to do this.

When they announced Firefox Input, I thought about doing this again... just with Firefox Input data but then I went on paternity leave and time kind of crept away. But I mentioned the idea this week and it piqued some interest.

So I found myself with a bit of time to work on it. The first stage was releasing a python library called textcluster.

textcluster takes the work I did earlier and makes it a bit more general purpose. The idea is I can do something like this:

docs = (
        'Every good boy does fine.',
        'Every good girl does well.',
        'Cats eat rats.',
        "Rats don't sleep.",
        )

c = Corpus()
for doc in docs:
    c.add(doc)

print c.cluster()

Which results in:

[
    (
        "Rats don't sleep.",
        {'Cats eat rats.': 0.21353467285253394}
    ),
    (
        'Every good girl does well.',
        {'Every good boy does fine.': 0.32030200927880093}
    )
]

The number is the "similarity" between the strings relative to the entire document corpus.

My next trick is to see if I can run this memory-intensive calculation over a data-set of 25,000 opinions submitted. If I can we can get some interesting data about what people think of the new Firefox beta.

Firefox Input, powered by Sphinx

2010-07-06T00:00:00-07:00

Thursday, I decided to take a half-day for my sanity, but saw an email about how Whoosh wasn't going to cut it for Firefox Input. I was CC'd about this and there was mention that Sphinx might be possible.

Sphinx is my hammer, and everything is a nail. So I said, let's do this. That translated into me spending my weekend, soothing my newborn and working on Sphinx. Luckily this was easy, since AMO and SUMO are both running Sphinx in a similar Django environment.

In order to move quickly, I copied code from the Zamboni project to Firefox Input. Even our deployment into staging and production wasn't done by our usual "Sphinx guy" in IT. Ultimately, everything landed in place.

So try it out and file bugs or let me know if searches don't go as planned.

On Buying a Thermometer

2010-07-05T00:00:00-07:00

After talking to our doulas about thermometers, I decided it'd be best to get an in-ear thermometer. I didn't at first because I heard they were expensive - but this is a device that I'll be using regularly for the foreseeable future. Our doula has had her thermometer for 8 years.

The other thing I like about the in-ear thermometer is it takes just a few seconds. I'm not really into holding a thermometer in my kids bum for a few minutes. Holding it in their armpit is less dangerous, but still slow. The in-ear thermometer works in a few seconds.

It's apparently not accurate at a young age, but I'll be using this for years to come.

I found the Braun Thermoscan at Target. Hopefully I can resist measuring Sibi's temperature every day.

Siddhartha: Weeks 3 and 4

2010-06-27T00:00:00-07:00

A week ago, I would have said this is still hell. However, since Wednesday, I've started to feel a lot more at ease.

I think a lot of what has changed is Katie and my attitudes. Katie's been a superstar at mothering and nursing Siddhartha. While I work from home, she does her best to do things without my help (although I don't mind getting her some water or food while she's busy nursing). After I finish work, I'm excited to spend time with Siddhartha. All the great feelings of being a dad that people expect you to have right away... I probably just got all those feelings in the last few days.

So here's progress in the last two weeks:

Sibi is over 8 lbs. He's clearly eating well, thanks to Katie.
Nipple shield is no longer required for feeding.
Sibi can pay more attention to us. When he's not crying, sleeping or eating he's a lot more fun.
Sibi can track objects and you with his eyes.
Sibi soothes quickly with Katie, even if it means just putting him on the breast.
Easier for me to deal when he fusses - I'm recognizing that its temporary.
We've been able to take him out a little bit each day for walks or trips to the store.
Lately he gives us a fair amount of sleep.
Our low days have been less bad than previous low days..
I got to go into the office for a bit.

We've no longer got Katie's mom to help us, and in some ways it's like the training wheels have come off. The adjustment was painful, like ripping a bandaid off, but we're managing. My mom has been in town and has helped somewhat in terms of food and giving Sibi infant massages, but we've been trying hard to do baby stuff on our own. It's paying off. We're a lot more confident.

Your objects, and all their friends

2010-06-25T00:00:00-07:00

This is complicated.

In my ever-evolving quest to get data out of the AMO database for tests, I found myself not just extracting a single object, but a list of complicated requirements in order to fully replicate behavior in production in a testing environment.

For AMO we can use fixture magic to dump a single add-on and all of it's database dependencies so that it will insert safely into a test-database. But we need more than just valid data. We need some supporting data. For an add-on to be browsable and searchable it needs to have a valid version and the version needs to have a valid file.

In our app we can check for these things by using this:

my_addon.current_version.files.all()[0]

Of course we need to check that my_addon.current_version exists and that files.all() has at least one object. This ends up being a lot of work if you just know the id of the add-on object.

So what I want is something simple, like:

./manage.py custom_dump addon 3615

And it should get me everything I need to test add-on 3615, including a Version object and any files associated with the version.

Turns out this just works. It works if you define the following settings:

## Fixture Magic
CUSTOM_DUMPS = {
    'addon': {  # ./manage.py custom_dump addon id
        'primary': 'addons.addon',  # This is our reference model.
        'dependents': [  # These are items we wish to dump.
            # Magic turns this into current_version.files.all()[0].
            'current_version.files.all.0',
        ],
        'order': ('app1.model1', 'app2.model2',),  # stuff gets sorted
        'excludes': {
            'app1.model1': ('fields', 'to', 'hide',),
        },
    }
}

Using this we're able to find out that addon means an addons.addon object and that you want the addon object with an id of 3615. From there we'll try looking for dependent objects. Using some black magic we can turn: current_version.files.all.0 into addon.objects.get(pk=3615).current_version.files.all()[0]. This gives us a file.

If we mimic our dump_object command we can get the file into the database and everything that the file needs to be valid. This in turn gives us enough data (usually) to begin testing a single addon.

So have fun with this, if you're database is remotely complicated, this can save you some time replicating it during testing.

Also note, that you can re-order models and exclude certain fields. This can make your fixtures very easy to load.

The first two weeks: what nobody tells you

2010-06-13T00:00:00-07:00

This is hell.

Apparently we are through the worst of it. Or we're just used to the sleep deprivation. I expected lack of sleep and I expected crying, but not so intense. Every parent we've run into assures us that this goes away and that it gets better. Some even say that this is just the "forth trimester" and once they pass three months the clouds will make way for the sunlight.

Time couldn't move quickly enough.

I do know that things have been getting better. Here's some improvements in the last two weeks:

Sibi is at (or possibly above) his birth weight after having lost a great deal of weight (more than the acceptable 10%).
Sibi can be exclusively breast-fed. At the hospital they supplemented with formula until Katie's milk came in.
Sibi can feed directly on the breast, rather than via bottle. This is thanks to the lactation consultants at Good Samaritan.
We no longer need to use a syringe during the feeding process. Previously feeding required both Katie, myself, a syringe and a flailing baby.
Sibi is calm when he goes to the breast. Before he would flail and fight it.
We can now watch movies while he nurses. Before we hadn't quite mastered the skill of feeding him.
Katie is now able to get a passable amount of sleep. Her OB told her she was sleep deprived a few days after Siddhartha was born.
I am able to get a reasonable amount of sleep.
We can usually tolerate his crying better and in some cases console him.

Lucky for us, Katie's mom has been helping us quite a bit while we adjust. We're still dealing with a lot of mystery and unknowns and its scary and pushes us quite a bit. His new thing is crying inconsolably at night until he's nursing again. Hoping it's a growth spurt and I'm hoping that we see some quality of life improvements in the next few weeks.

I'm nervous about returning to work. I'm hoping that my mom who arrives on Friday when Katie's mom leaves can find ways to console him as well.

On the Labor and Delivery of our Son

2010-06-06T00:00:00-07:00

Here's my subjective account of Katie's labor. I don't have exact times and details, but our doulas have it on record somewhere. Most of the links are to my twitter updates.

Friday at the OB

Many of my coworkers wished me that my paternity leave would start by the end of the weekend. I did too...

Friday May 29th we found out that Katie needed to be induced. The fluid levels in her amniotic sac were very low. Dr. Woo strongly suggested that we immediately go home, get our things and go to the hospital. I was excited, but not thrilled that there would be an induction.

We headed home, and I alerted our doulas en route. We gave ourselves 20 minutes to wrap up everything and we were on our way.

Hurry up and wait

Lucille Packard Children's Hospital's Labor and Delivery ward was booked solid, so we were put in a closet sized room and told to wait for a while. It was quite cozy with our two doulas. Katie and I hadn't eaten for a while so I decided to get us some food from Thaiphoon.

When I got back I learned that Katie would be under observation while she had a cervidil induction. Unfortunately, we were stuck in the tiny room and I was sleeping on the floor. Our nurse moved us into a much larger room and I was able to sleep comfortably once I ignored my nasty headache.

The "pit"

The next morning we made a little progress with the cervidil. However, we needed more to get Katie into active labor, so we were asked to try pitocin.

Katie was well-read on pitocin and to her, pitocin made the following scenario likely:

pitocin -> epidural -> c-section

Katie and I decided that pitocin would be the best option, despite the possible side-effects. We just asked that they introduce it slowly and increment it slowly. This eventually caused Katie's water to break.

Unfortunately, we had the nurse who thought she knew what was "best". At some point she had discretely kicked up the pit-level a bit high. This lead to Katie getting painful, rapid contractions too early.

Transition

These rapid contractions transitioned Katie into a more intense stage of labor, but they started around 2cm -- far too early.

Each contraction brought enough physical pain (10 on the scale from 1-10) even the doulas and myself couldn't ease the pain. This wasn't normal.

We opted for an epidural. The doulas had to leave for that, and Katie, who was shaking, had to be restrained in order for her to be hooked up. Once she was on it, all was well.

OMG WTF BBQ

Until it wasn't. The fetal heart rate went down quite a bit. Several nurses and our anesthesiologist arrived, brought Katie on all fours and got the fetal heart rate back up. Except for the fact that Katie could still feel her contractions in the left side of her behind, everything was fine.

The doulas came by and checked on us before deciding to catch some shut-eye in the waiting area.

Before we went to sleep, we decided we should have the anesthesiologist adjust the epidural line. He adjusted it and all was well.

Everything happens so fast

Until everything repeated itself. All of a sudden the OB, anesthesiologist, nurses were all in our room getting Katie on all fours again. The OB wasn't comfortable with the baby's heart rate so she had everyone move to the operating room. I called our doulas, letting them know "stuff is happening." I was told to hold back by a certain nurse and then she said she'd get me into scrubs.

A nurse told me to get in scrubs and the doulas were to pack up our room. When I got into scrubs, the nurse said I couldn't go into the room, since they decided to operate (meaning C-Section). I waited for a few minutes and the nurse told me to come with her -- my son was delivered:

My son was born at 9:59PM on May 29th 2010. He had a healthy 9/10 APGAR score, and looked wonderful. For the first time in my life, I cried.

Recovery

Amazingly, after all of this, Katie was fine. She recovered and we spent a few frantic nights in the hospital getting used to the fact that we were new parents. With the help of good friends and family we've made it over a week.

Introducing...

2010-05-29T00:00:00-07:00

Siddhartha Bonn Dash

7 lbs 2 oz, 19 and 3/4 in

9:59 pm 29 May 2010

No, the Baby Has Not Arrived -- an FAQ

2010-05-27T00:00:00-07:00

People tend to ask me the same questions, so I figure I can answer some basics here.

When is the baby due?

Two days ago (25 May). Due-dates, however, are 40 weeks after LMP and statistically there's less than 5% chance of a delivery on that exact day. There is, however, a 90% likelyhood that it will happen within two weeks of that date. [1]

Is it a boy or a girl?

Boy.

What's his name?

You'll find out during the birth announcement.

Should I call you to find out if the baby is born?

No, that's a terrible idea. Katie and I both hate the phone even when there isn't a child on the way. We hate it more when we're trying to relax before the baby is born. If you absolutely need to get in touch with us, try text or email.

Is there anything we can do?

Before the birth:: Play with our cat, he's needy and needs more attention than anyone can give him.
After the birth:: Email me if you are serious, there's plenty.

How is Katie feeling?

Tired - but she blogs in more detail.

Are you nervous?

No. I am not terribly excited about the sleep disruption, but I'll manage.

Excited?

Yes. I'm excited. Think of every cliché thing about new dads -- that's what I have.

Back to My Pre-California Weight

2010-05-23T00:00:00-07:00

About a month back I hit a milestone, I went down to my pre-California weight.

Here's a chart of my weight since August 2007, a few months before I moved to California.

Here are the events:

Move from Minneapolis to SF Bay
Visit Minneapolis
Visit Minneapolis
Leaving Yahoo!
Move to San José
Try iChange

The Basics

There's nothing shocking. The more I ate and less I exercised, the more I gained weight. Travel was an easy way to gain weight because I never checked in regularly. Moving to a condo with an awesome gym made regular exercise a breeze.

I tried out iChange which offered one-on-one monitoring. That got me eating healthier food, so it put most of my calories to good use:

Sugar and saturated fats were all but eliminated from my diet and slowly reintroduced when I could moderate them.
Counting calories helped reset my internal calorie tracker. I still don't do it enough, so I try to simply eat better.
Water tastes better the more I drink it. I drink a lot of soda still, but there are days when I can completely avoid it.
Exercising a lot is nice. I get to eat more. Exercising early works best for me.
Eating more whole grains felt better. The food was flavorful.
Balanced meals were tricky, though eating more fruit and vegetables made me less hungry for higher calorie items.
I had to start by doing everything: eating better, exercising, etc. Doing one thing at a time was a failure. There were no shortcuts at the beginning.

Now I stay pretty much on track with losing weight. The days I gain a bit of weight, I understand why, and I am usually able to resolve it quickly. Therefore weight loss is no longer stressful.

So with about 10lbs less of weight from the beginning of this year, I can sling on a Moby-wrap with my son (when he decides to be born) and it'll feel like nothing.

So your Wordpress has been hacked

2010-05-19T00:00:00-07:00

Last week, someone informed me that my blog had been hacked:

I'm not quite sure what the vector was. Wordpress wasn't very secure and I didn't take too many measures to harden it. A coworker of mine (on our security team) decided it might be fun to have a look at the infected Wordpress Installation.

Here's how the hack works

Your blog appears normal to you and your visitors.
Some rogue PHP code detects if Google is crawling your site and modifies the text and links so it looks like your website is a Viagra pharmacy.
The links go to other infected blogs and thus builds up page rank for this ring of blogs. So the upside is that your blog may be a top result... for VIAGRA.

Prevention

Here are some tips for prevention, but you can find a lot more by googling for Wordpress hacks. My solutions are more technical:

Don't use Wordpress - I recently switched to Jekyll since it was conceptually easier to understand, and it's coder-friendly.
Remove all users other than your own.
Change your password.
Check your code into git so you can see what files have changed.
Prevent Wordpress from writing to your webroot.

Restoration

Here's what you'll need to do to de-spam yourself:

Verify that you are still spammed by using Google Webmaster Tools|Labs|Fetch as Googlebot.
Back up your blog and database.
Move your Wordpress installation to a new directory.
Install Wordpress from scratch.
Remove all users except for yourself.
Change your password.
Copy your theme to your new installation.
Install only the plugins you need.

By step 4, you should be able to verify, using Fetch as Googlebot, that your website is no longer an online pharmacy.

Good luck.

Alphabetical sorting in Sphinx

2010-04-21T00:00:00-07:00

Sphinx 0.9.9 is great at searching full text, but treating actual strings as attributes takes some work.

Initially I employed the strategy of indexing my full text fields and storing them as attributes. E.g.:

sql_query = SELECT name, name AS name_ord FROM documents
sql_attr_str2ordinal = name_ord

This stores each attribute in lexical order. Meaning if your name's are Apple, Aardvark, Button, Choco-room they would be given the ordinal 2, 1, 3, 4 respectively.

However, this is case-insensitive. So trying this approach:

sql_query = SELECT name, UPPER(name) AS name_ord FROM documents
sql_attr_str2ordinal = name_ord

Will allow for case-insensitive alphabetical sorting in Sphinx.

Test Driven Confidence

2010-04-20T00:00:00-07:00

If you're already testing your web applications, you can skip this post.

One of the bugs I am working for AMO on involves porting a small, but moderately complicated checkbox from our PHP site and rewriting it for Django.

I decided to look at the existing implementation and found it to not work correctly at all. This was frustrating, especially since I verified that my own code worked, and that QA verified that it worked as well.

This is frustrating on many levels. Chances are some minor assumption I made changed, and thus broke this functionality. Discovering regressions is never fun, and fixing them is can be long and tedious if you can't automatically verify that everything is working correctly.

Lucky for me, coming up with tests is easy, you just do what you would do to verify the code satisfies the requirements and then code it. Sometimes the tests can take longer than writing the actual code, but ultimately you can ship with confidence. You can be confident that your feature won't break in the future without immediate notice, and you can be confident that your new code won't break anything else.

Finding the most common Firefox issues

2010-03-18T00:00:00-07:00

Cheng Wang of the Mozilla Support team, a few months back, decided to present on some design ideas for Firefox Support. One of the issues he noted was that there are a lot of repeated issues and that it would be useful to group them. Grouping them lets you see how often something occurs, and secondly let's you see how urgent it might be.

Luckily grouping and clustering text is something computers can do. So I wrote this utility that does just that.

I ran this script over a sampling of data from the last week:

Firefox won't start after update. (65 related issues)
- 5.6: Firefox updated, Gmail not delivering mails
- 5.6: How to change My Profile when Firefox won't load?
- 7.5: Once I close firefox, cannot start firefox again except system restart
- 5.6: When intalling updates Firefox uninstalls itself
- 16.8: firefox won't start after update 3.6
- 11.2: Upgraded to Firefox 3.6 and now it won't start
- 14.9: Firefox won't start with most extensions
How do I add a bookmark to more than one folder? (64 related issues)
- 8.9: How do I get my bookmarks on the bookmarks toolbar to show up as an icon only with no text?
- 7.5: Bookmarks lost after upgrade and cannot save new bookmarks
- 7.5: why do i have to add the .com now to addy's?
- 8.7: When I open sidebar to edit bookmarks, I only see the folder for Bookmarks Toolbar. I do not see a folder just called Bookmarks nor do I see my list of bookmarks, that separately appear under bookmarks menu at top of screen
- 7.5: All my impoted bookmarks go to the same webpage
How do I remove the \ask toolbar\"?" (50 related issues)
- 14.9: How do I remove an unwanted toolbar?
- 5.6: how to remove temporary video files from computer
- 7.5: I have no Toolbars or searchbar and i cant bring them back
- 7.5: nowhere says how to REMOVE a toolbar - only how to add or modify one
not able to open youtube videos (45 related issues)
- 5.6: Cannot open bookmark/history sidebar
- 5.6: After working well for years Firefox will now not open
- 6.7: opening bookmarks do not open in new tab
- 5.6: I can't watch videos on youtube with firefox, but on internet explorer i can
I cannot download Firefox 3.6. I've tried erasing the download file. I cannot get beyond logging out of Firefox. (44 related issues)
- 8.4: when downloading files firefox download manager will freeze and i will have to start over the file download
- 5.6: Firefox will not let me download anything! Can someone help?
- 6.3: cannot download epixHD.com: not compatible with firefox 3.6
- 5.0: Several tabs are coming up when i try to downloads things
- 5.0: Firefox wont open since I downloaded the 3.6 update.

The number on the right of the related issue is a score of how strongly it relates to the main issue.

The full sample is 352 clusters from an original 3000+ issues. That's a lot less stuff to go through. We can tune this to have either less clusters, and more related issues in a cluster, or we can make more clusters of issues and that might result in more accuracy.

Despite the inaccuracy of clustering we can make some general observations:

Firefox not starting is a big issue.
Bookmarks are either confusing or broken.
People don't like toolbars
Opening things is hard
Downloading things or Firefox is hard

Hopefully we can fine tune these reports and have them run regularly... maybe automatically posting to Tumblr?

A few weeks in Chrome

2010-03-17T00:00:00-07:00

A number of weeks ago I got annoyed with Firefox and decided to use Chrome for a while. This reminded me of the olden days where I used Netscape for a while, and then IE6 came out, and then Phoenix came out all the while I'd keep switching to the newest shiniest thing (note: I'm not sure about the timeline of all the browsers either).

My browser of choice since Firefox was released has been Firefox. For some time - nothing shiny in browser-land was coming out. Little UI things in Safari kept me away (and the lack of extensions), but Chrome finally showed promise. WebKit, out of process plugins, process separated tabs and now extensions. This was great.

I immediately felt like I was going to really love Chrome, and be that guy at the office (I work at Mozilla) who insists on using Chrome (just like I was that guy at Yahoo! who used Google for everything). I also wanted to answer the question as to why so many people really like Firefox in spite of Chrome's amazing speed -- even many Googlers will admit to preferring Firefox.

Overall I'm happy with Chrome, but I'm switching back to Firefox for now. Here's some things I observed:

"/" in Firefox let's you search. Which to me seems more natural than Ctrl-F. I am pleased that Chrome supports a lot of Firefox's shortcuts, like Cmd-1..9 for switching tabs, or Cmd-Shift-T for reopening a closed tab.
No titlebar... I kind of miss it.
XML is way easier to work with in Firefox. It's collapsable and always looks pretty.
Certain sites don't work well in Chrome, like Rypple or the AmericanExpress web site. Rypple surprisingly enough is built using the Google Web Toolkit. I really wish their was a "FirefoxTab" that would open certain sites in Firefox instead.
There's a number of Jetpacks and Extensions that only exist for Firefox or they are severely lacking.
- The Jetpacks for Mozilla's Bugzilla instance are awesome.
- The Delicious and AdBlock extensions on Chrome aren't nearly as good as the ones for Firefox.
Firebug is much better than the Chrome developer tools. For example, you can adjust css values instantly.
Extensions die... and don't come back without restart and appear to never have been installed unless you remember them crashing.
AwesomeBar (the location bar in Firefox) queries your history much better than the OmniBar (the location bar in Chrome)
- At first I thought this was because of a sparse history, but after several weeks I still have a hard time finding sites I've been to.
- Chrome will show you a handful of results, and then let you know there are more results, but that takes you to a new screen which is a jarring UI.
Chrome can be slow. The extensions can take a while, and even switching between tabs can be slow. At this point startup time can be a moot point.

Overall this was a healthy exercise, since I really like to be up on new browsers, and Chrome really seems like it's can be a good browser for many people. I'll probably try it again after the next major Chrome update.

Making our tests run thrice as fast

2010-03-16T00:00:00-07:00

I've written a faster version of TransactionTestCase and packaged it with test_utils. It's mysql specific since it relies on SET FOREIGN_KEY_CHECKS=0 to flush the database.

The long story...

Why speed matters

We're closing in on 300 tests for Zamboni. As of yesterday, to run our entire test suite it would have taken approximately 5 minutes. If you run tests before code-reviews, during a code-review, and before you push to master - you've spent about 15 minutes doing tests for a single feature or bug-fix. We have about 5 developers, so this cycle happens many times in a work day. In that time many sandwiches can be made and consumed.

Even shortcuts, like running a subset of tests will only go so far, and ultimately we do want to validate that all our tests pass for any code-change.

Testing Sphinx search with `TransactionTestCase`

Django recently sped up testing by running tests in a transaction. However, this means that data never gets committed to the database and therefore external tools, like the Sphinx indexer, will never see any of that data. So we resort to TransactionTestCase which will commit the data.

Unfortunately TransactionTestCase is painfully slow. The accepted practice is to only use TestCase if you want your tests to be fast. So, I decided to complain to one of our new hires and he and I decided to tinker in mysql to figure out what was slow. We discovered the following:

delete from [table] is slow
truncate [table] is slow
... unless you SET FOREIGN_KEY_CHECKS=0

So we decided we should do our own tear down. After some tinkering with cProfiler I discovered that TransactionTestCase does a (slow) database flush on setup for a test case. This wouldn't do.

Making our own `TransactionTestCase`

I decided to make our own TransactionTestCase and it would just run SET FOREIGN_KEY_CHECKS=0 and TRUNCATE on each table at tear down time. It would also not do a flush on set up.

We write our tests with the idea that they clean up after themselves. Rather than having them cleanup after the last test. This is a requirement for us since django-nose doesn't reorder tests (nor should it) and a standard django.test.TestCase assumes a clean database.

Looking at a single test test_sphinx_indexer, using django.test.TransactionTestCase took ~30 seconds. Using our new TransactionTestCase it takes ~4 seconds!

Fast tests are good

We can now run our 275 tests in ~100 seconds versus the ~300 seconds it used to take. Furthermore, skipping our sphinx tests (which are the only tests that use TransactionTestCase) only saves us ~10seconds. That's not a lot of overhead for better coverage.

This took me the better part of a day, but solving this now, means we're going to more often than not run our sphinx tests all the time rather than skip them. Our QA team will assure you that search is probably the most regression prone part of our site, so running these tests are vital to quality.

If you need to use TransactionTestCase in mysql, give ours a try.

λ^2: safely doing class based views in Django

2010-03-09T00:00:00-08:00

When I started rewriting the API for addons.mozilla.org, my views were mostly the same: get some data and render it as either JSON or XML. I also wanted all my API methods to take an api_version parameter, so I decided class based views would be best. This way my classes could just inherit from a base class.

To do this I had to implement a __call__ method. This works fine, except I wanted to store things into the class -- after all the whole point of my use of classes was to keep the code a bit more compact, and cleaner. So, why pass the api_version around everywhere? Unfortunately thread-safety comes to play, and you need a separate instance of your class for each request.

λ

Django's urlpatterns expects a callable object. So you can't give it an instance of AddonDetailView(). But you could give it a callable that creates an instance of AddonDetailView() and passes it *args and **kwargs. Luckily python has lambda functions. You can note how we solved that in our urlpatterns.

λ λ

But wrapping all your urls with lambda is tedious and remembering to pass *args and **kwargs is error prone.

So let's make a lambda function that returns... a lambda function that turns an instance of our class into a callable.

We can now return to coding and not think about thread safety.

λλλ

django-fixture-magic: Testing issues with real data.

2010-03-05T00:00:00-08:00

I just released Fixture Magic.

When dealing with legacy data, you'll run into all kinds of edge cases. Perhaps, an object might not display correctly unless it has the right parameters, or if it has null parameters it might not display at all. So when testing Django, it's nice to actually use non-dummy data.

Luckily Django has a way of pulling real data out of your database using django.core.serializers:

from addons.models import Addon
a = Addon.objects.get(id=3615)
from django.core.serializers import serialize
jsonize = lambda a: serialize("json", a, indent=4)
jsonize([a])

This solution runs well in a Django shell and can be lots of fun for the whole family... until things get complicated.

Serializing alone isn't enough.

Serializing a fixture with foreign keys means you'll have an un-loadable fixture unless you serialize the dependent fixtures. Even for one or two foreign keys, this can be a pain. For addons.mozilla.org, we have a spidery-web of dependencies: Files need a Version which needs an Addon which need Translations.

Thus begat the dump_object management command. Give it an app, model name and a pk and it will give you not only a serialized JSON of that object, but all the objects that it requires.

Example:

./manage.py dump_object files.file 64874 64876 > my_new_fixture.json

This looks for the File model in the files app and pulls out of the database Files instances with pks of 64874 and 64876. It then recursively searches for any required objects.

Too much serial

If you create a lot of fixtures, you'll eventually have overlapping serialized objects. In addons.mozilla.org we have Addons, Versions (which depend on Addons) and AddonCategorys (which depend on Addons and Categorys). If we wanted to get serialize a specific Addon, it's dependent Versions and AddonCategorys it makes sense to start with dump_objecting the related Version and then dump_objecting the AddonCategory. Both dump_object commands will fetch the Addon in question, resulting in duplicated data.

To combat this we can use merge_fixtures to dedupe our fixtures:

./manage.py dump_object versions.version 64874 > 1.json
./manage.py dump_object categories.addoncategory > 2.json
./manage.py merge_json 1.json 2.json > happy_fixture.json

This should make creating test data slightly less painful. So give it a try.

Advice for people seeking a mortgage

2010-02-28T00:00:00-08:00

So Katie and I decided to buy our first home (a condo) in California last year. From our first house (in Minneapolis) the lessons we learned was to have your 20% saved up ahead of time before taking the plunge. We're fans of traditional 30-year loans (or 15 year loans if you can swing it) as they are fairly predictable and 20% will buy your way out of mortgage insurance.

20% seems like a lot, and it is, but that seemed to be the least of our problems. Here's some more advice:

Maintain a separate account for funding your laon. Have a separate account for funding your loan. Temporarily have your paychecks go there, have your gift money go there, stock sales, etc. This way you can build up your 20% without having to worry about miscellaneous income. E.g. I had some small amounts of money enter my account due to expenses that I filed with my employer. I had to paper-trail those.
Get your funding in advance. If you can have the aforementioned account fully funded with your 20% a few months before you seek a loan, you could save a lot of trouble. Our mortgage officer seemed to only care about 2 months of history. However, this is hard since in a down economy having a large chunk of change just sitting there isn't always a great idea.
Link your accounts. Link all your accounts to the source account for your mortgage. If you have stocks, savings or other checking accounts make sure that you can quickly transfer money into your source account. Closing costs, interest rates and other seemingly minor details can force you to quickly sell something (like stock) and fund an account. I didn't have my accounts linked, and I think I managed on funding from an unlinked source at the last minute. It was very tense.
Know your transfer times. I'd give everything about a week to clear. It usually takes a few days, but every now and then it takes much longer for money to clear out of one account and into another. Keep a calendar open.
Have everything ready to go a few weeks early. The tenseness of waiting for banks and fiduciary firms to get their acts together will give you endless migraines. Also, if everything is ready in advance, you may get your closing date moved up.
Paper trail everything. If you don't need the money to make closing, don't put it in your source account. If you do need the money, show a paper trail.
Have your loan officer make a schedule for you. Our loan officer was horrible since we had no idea what needed to be done when. So we would frantically get things together for her, and realize it was all for naught. Once you have a desired closing date, tell them to give you deadlines for having things.

While I wish I had this advice when I bought our place, we did manage to get all our ducks in a row and close without delay.

Update: A translation in Belorussian (not sure why).

Retrieving elements in a specific order in Django and mySQL

2010-02-11T00:00:00-08:00

If you have a list of ordered ids and you want to turn them into an ordered result set you can use FIELD() in mysql:

SELECT * FROM addons
ORDER BY FIELD(id, 3, 5, 9, 1);

This is a handy trick if you use an external search engine which gives you an ordered list of ids and you want to pull out entire row sets.

We do this in addons.mozilla.org using the Django ORM like so:

The code in action.

Palm Pré: A retraction, I really like it now

2010-01-11T00:00:00-08:00

So before I went on trip to Minnesota last month, I decided maybe I would give the Palm Pré another shot. After all, my parents have no internet access, so having the Pré... if I could overcome my issues, might be a welcome distraction.

Before I packed it, I updated to WebOS 1.3.x (a few days later I updated to 1.3.5) and I was blown away. The horsepower was increased by utilizing the GPU. The following problems were fixed:

* The device was no longer hot all the time
* Shutdown and startup were long, but not nearly as long as before.
* Render times were quicker
* All the elements usually rendered quickly in an app
* Network was fairly steady
* Phone calls also seemed fairly drop-free.

All these improvements helped me get over

* The tiny keyboard... not so bad in practice.
* No soft keyboard - I missed it, but I could deal without it.

Overall the device was great, it was fast enough to use, and most of the errors were annoying, but things I could deal with. Cut and paste could be improved, and I wish the USB connector was the same as the one for HTC devices (I can't keep micro or mini USB types straight).

So I love the device, and Murphy's Law dictates if work gives you a phone you don't like you get to keep it... until you start liking it again. So I sent the phone back into rotation for other people at Mozilla to try. Have at it.

Google Chrome Extensions Puzzle

2010-01-06T00:00:00-08:00

I went to Add-on-Con some weeks back to represent my employer, the Mozilla Corporation.

One of the goodies you get as a registrant was a jigsaw puzzle from the Google Chrome Extensions team.

Perfect, my wife and I love solving jigsaw puzzles. We finally finished a few days ago. Anybody who has started at all will realize the puzzle is of a QR-code. The QR-code is an extension that will eventually lead you to a prize. It was a bit of a mini-puzzle not nearly as difficult as finding the QR code.

Although finding a QR code scanner was a bit difficult, I had to borrow a HTC Magic from Justin Scott and installed a decent barcode scanner.

Django Model Inheritance Or Related Tables Wrt Amo

2009-12-15T00:00:00-08:00

layout: post title: "Django: Model Inheritance or Related Tables wrt AMO" site: spindrop

tags: [spindrop, django, mozilla, amo, addons.mozilla.org]

When I attended DjangoCon this year, I lamented that our flagship web property was difficult to test, and not fun to develop. I figured DjangoCon was a way to placate me, and Django might mean something for some of the smaller projects at Mozilla. However, Wil Clouser, our lead web developer, announced development changes for addons.mozilla.org (AMO) that says we'll be moving to Django.

Wil was open to Django and knew that's what we in the dev team wanted. Jeff spawned our foray into a new AMO with Zamboni. I've been working on some grunt-work tasks inside and outside of Django.

One of those tasks is building a transparent layer in Django to keep users logged in from our PHP-based site. That kind of problem almost immediately forces you to ask one of the most fundamental questions you ask when using any framework:

How much do I change my app, in order to accommodate the framework?

More specifically:

Should I use the django.contrib.auth User module, and to what extent?

The more we looked into what features of Django we might want to use, django.contrib.auth was heavily tied into other things we wanted, so it made sense for us to use it. The next question is whether we try the inheritance approach or do we treat our legacy users table as a sort of User Profile and utilize the User module using the related table approach?

Using model-inheritance seems real nice, because we can pretend that our legacy user is the same thing as a djaango.contrib.auth User - but this isn't true:

Looking at our users table more closely:

mysql> explain users;
+-------------------------+---------------------+------+-----+---------------------+----------------+
| Field                   | Type                | Null | Key | Default             | Extra          |
+-------------------------+---------------------+------+-----+---------------------+----------------+
| id                      | int(11) unsigned    | NO   | PRI | NULL                | auto_increment |
| email                   | varchar(255)        | YES  | UNI | NULL                |                |
| password                | varchar(255)        | NO   |     |                     |                |
| firstname               | varchar(255)        | NO   |     |                     |                |
| lastname                | varchar(255)        | NO   |     |                     |                |
| nickname                | varchar(255)        | YES  | MUL | NULL                |                |
| bio                     | int(11) unsigned    | YES  | MUL | NULL                |                |
| emailhidden             | tinyint(1) unsigned | NO   |     | 0                   |                |
| sandboxshown            | tinyint(1) unsigned | NO   |     | 0                   |                |
| homepage                | varchar(255)        | YES  |     | NULL                |                |
| display_collections     | tinyint(1) unsigned | NO   |     | 0                   |                |
| display_collections_fav | tinyint(1) unsigned | NO   |     | 0                   |                |
| confirmationcode        | varchar(255)        | NO   |     |                     |                |
| resetcode               | varchar(255)        | NO   |     |                     |                |
| resetcode_expires       | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notifycompat            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| notifyevents            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| deleted                 | tinyint(1)          | YES  |     | 0                   |                |
| created                 | datetime            | NO   | MUL | 0000-00-00 00:00:00 |                |
| modified                | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notes                   | text                | YES  |     | NULL                |                |
| location                | varchar(255)        | NO   |     |                     |                |
| occupation              | varchar(255)        | NO   |     |                     |                |
| picture_type            | varchar(25)         | NO   |     |                     |                |
| averagerating           | varchar(255)        | YES  |     | NULL                |                |
+-------------------------+---------------------+------+-----+---------------------+----------------+

You can very easily argue that this is a profile table, which happens to have credential information thrown in.

I can see overtime, I'll just struggle to keep our legacy User to act like a Django User, whereas a UserProfile is fairly standard.

Had I been writing this app from scratch, I would have chosen the UserProfile route. This is extra data which takes up a lot of space, and changes far more often than user credentials. Changing 4M+ rows sucks, by making users our UserProfile table, any changes to that table, don't tie up the table used for sign-ins.

I'm curious what other people who port their apps to Django have done.

Palm Pre: Always hot

2009-11-19T00:00:00-08:00

So I borrowed a Palm Pré that we had at Mozilla to see what it was like. I was at first very excited, I remember before the Pre was released there was a lot of talk about how awesome-fantastic it was going to be. The stories of awesomeness sort of died, and I had thought nothing of it.

Immediately upon using the Pre I figured out why. In short, it's a crappy phone. It makes a very good attempt to do a lot, but it does them with such piss-poor performance, that nothing good is noticed.

I am disappointed. It's not even in the same class as an iPhone - maybe a future generation of Palm devices will be, but not this one. I was hoping WebOS would be a good alternative to the iPhone. It looks like Google will be doing that, though their phones haven't impressed me much either. I am hoping that maybe this phone is just a dud.

Here's what I didn't like: * The Palm was always hot. * The first run experience is painfully slow. * The first run was an indicator of things to come, startup and shutdown are ridiculously slow. * Every application is slow to render. * Not all elements of an app render. * The keys are too small. Some people aren't migrating from a Treo and aren't used to mini keys. * No soft keyboard. * The palm website doesn't let you use plus-style addressing * Media Mode was not self explanatory - and forced the phone to not work. * Network would constantly drop out. Couldn't use a lot of the data features. * Phone calls didn't work so great. * Did I mention it was ass slow, even the dialing program was slow. * The battery dies quickly * I could only cut/paste when composing, but I couldn't cut a string of text from an email. * Felt too much like an old palm

Despite the sadness there were a few good things:

When it did fetch email, and other notices, it displayed them nicely
The unification of Facebook and Gmail was pretty cool - it also made me want to trim some of those friends from highschool off my facebook - I ain't ever gonna call em.
The Icons were pretty.
The card interface was interesting.
The travel charger could be modified to work in non US chargers fairly easily.

All in all, I'm glad that I had a chance to try out this device. It showed me, that user interfaces above all need to be very fast and responsive. Furthermore, everything you try to do should be done exceptionally well. I'm hopeful that software updates can alleviate some of the problem, but I think the root of the problem is slow hardware.