Dave Dash

On leaving Delicious

2009-05-21T00:00:00-07:00

In short I am quitting my awesome job at Yahoo! working on Delicious and will be starting somewhere else next month.

Delicious has been one of my favorite web "properties" for a number of years and has influenced my work long before I started here. I've really respected my fellow engineers, managers and product team despite all the transitions we've had to bear. I am really amazed by how much everyone in our current team has changed over the year and a half.

Delicious has had it's ups and downs, both as a team, as part of Yahoo! and even for me personally. Lucky for me, I stayed motivated the whole way through and I am glad to be leaving the party while its still fun.

Optimizing via YSlow

2009-03-22T00:00:00-07:00

I'm not a performance person per se. I am a heuristics type of guy. Meaning you tell me a bunch of things to look out for, I get an understanding for those things, and if I agree, I'll fix them. I work off lists :)

YSlow is therefore a boon to me. If you aren't familiar with it, it is a plugin for Firefox (with a dependency on Firebug) that will grade the performance of a given site based on a set of rules determined by YDN. This is my list, and YDN went into great detail to explain each of these items is important.

The following is my journey through optimization. I went from a D- to a C in a single day. While that might not seem like much, I could have nabbed an A if I used a CDN, like CloudFront, or even a fake CDN and dropped Google AdSense and Analytics. Those tools are helpful for me, and worth keeping around, so I opted to take the grade penalty.

Begin: D - 61

I began with a D. It's actually not that bad. In most dimensions I was getting an A. Namely because I use a web framework (Django) that is optimized for being cached and served well.

YUI Combo-loading: Sneaking up to D - 62

My first goal was to reduce the HTTP requests.

I use the YUI CSS and javascript framework as the foundation for my CSS and javascript. One major benefit is that they offer to host this for you on a true CDN. The javascript framework is very thorough and can do quite a lot. Because it's so robust, it is split into several files. Originally I had been doing this:

That was 4 separate calls from the web browser to this server. While they are all fast requests, it'd be better to make a single request that catches all of this.

Luckily, YUI has "combo-handling". I was able to turn those four requests into the following:

A very quick fix, that brought my overall grade up by a single point.

Concatenation and Minification: Instant C- - 72

I still had a large number of HTTP requests. Luckily YUI has the YUI Compressor. It compresses javascript and CSS and even gives you tips on writing better code (or more compressible code).

This used to be scary and not something I'd want to deal with, but if I can run it from the commandline, then I can add it to my deployment script. Now on each deployment my CSS and javascript are concatenated and compressed and uploaded to my server. When my app is in production mode, it seeks out minified code, versus more verbose human-readable code.

The javascript compression was great. The output was at least compressed in half. This skyrocketed the score to 72. Less HTTP requests and Javascript minification were the individual rules in YSlow that were boosted.

Google: They want you to fail

This is of course tongue-in-cheek. Google has some wonderful tools like AdSense and Analytics that do hinder the score and ever so slightly the speed of your site.

As far as javascript is concerned, I am linking to: * YUI * My own JS * Google Analytics * Google Adsense

Google Adsense is by far the worst offender. You think you're calling just one solitary call to Google to fetch an ad, but it's actually 4 calls. I'm sure this could be engineered better. The good news is it's usually not too slow.

There's an option to self-serve ga.js the Analytics code. While this might result in a higher score, there are a few issues.

The process is more error prone, there's a chance that the file you download from Google might get corrupted and your unit tests might miss it.
You are serving from your own servers - which could either be expensive or slower.
Many sites include a call for ga.js and therefore there's a high likelihood that it's already browser-cached.
The code could get out of date.

For this reason, it's better to lose about 5 points for not serving it. This score is large because it effects a lot of YSlow's rulesets. If you'd still like to self-serve here's some discussion on that.

This will likely give you an "F" on the Expires Header ruleset as well.

Content Delivery Network (CDN)

A lot of people freak that CDN is on YSlow and feel like not all sites need/require a CDN. They are right that the don't need a CDN, but in my opinion they don't necessarily need to get an "A" either. All this is an approximation of doing the best you can do to make your frontend code as fast and efficient as possible.

With that said, CDN's are becoming more and more in reach. Amazon Cloud Files which I mentioned is a pay as you go system, which is quite affordable (I estimated < $1/month for my needs).

I configured YSlow to list some of the standard Google and Yahoo API hosts as CDNs. This didn't affect my score, and I didn't want to integrate S3/Cloudfiles at this stage, so I took the "F".

Gzip files (still at 72)

Adding Gzip to the Nginx configuration was easy, however my grade still stayed at 72, but it did up my score for that specific rule. Gzip compression is supported in most modern browsers and is usually a config change away.

Again, note that ga.js is not served compressed.

Upgrading from `urchin.js` (up to 73)

In the process of tweaking my Gzip configuration, I noticed that I was still using the legacy urchin.js instead of the more modern ga.js. Simply changing this boosted me to 73.

Move Javascript to the bottom (still at 73)

Moving the javascript to the bottom is another trick. This at least loads most of the HTML before the Javascript loads (and in many cases, waits).

Some refactoring of Django templates makes this a breeze.

Conclusions

I was able to raise the score from a low D to a mid C, at some point I could make the score a high C, but it was inevitably not worth it.

From my perspective, Google could do a few things to help webmasters who use their widgets:

Add a decent expires header to ga.js - this file is safe to cache for some degree of time. Being at the latest and greatest ga.js is nice, but not necessary.
Serve files Gzipped
Use a single host for serving public APIs. Yahoo can improve on this as well. They can potentially do combo-hosting a la Yahoo Developer Network

Ultimately YSlow is a good guideline for speeding up your page render times. It's not perfect, nor exact. There's no realistic way to determine a users experience with page load times. Slow network connectivity and bad browsers can make all your attempts in vain. However, YSlow is an easy to follow set of heuristics which every site owner should attempt to implement as well as possible.

YSlow's future incarnations will yield some more flexibility. Even in its default state it should yield better scores for people who are doing the right thing.

You can't go home again: A year in the Bay Area

2008-11-07T00:00:00-08:00

Yesterday as I left work it dawned on me that I had been in the San Francisco Bay Area for a year. It is one of the most drastic changes in my life since going to college.

I remember announcing to my family that Katie and I were moving to the Bay Area as I had accepted a job at Yahoo! working on one of my favorite web sites, Delicious.

The decision to move, the actual move itself, the settling in period, exploring the area, meeting friends new and old... it was all an adventure. The question everyone asks is how's california... it's awesome... even after a year.

We've successfully unanchored from our house and life in Minneapolis
Took the kitties on a plane ride and had them live in 4 different places in 3 days.
Took up a position working on Delicious and helped launch the new version of the site.
Had our car shipped across country
Had movers move all our stuff for us
Moved on up to a more spacious living arrangement that has rent that costs more than our mortgage.
We're a short distance a Costco, Target, restaurants and the middle of a small town
Reconnected with friends from High School, College, Minneapolis and met new friends...
3 of our friends (2 are a couple) live a block away.
We run into people we know
We got sucked into Veronica Mars, Buffy and Angel
Bike to work regularly... otherwise shuttle... rarely drive.
Learned python
Saw some pretty trees at Sequoia National Park
Went to Apple Hill
Saw Mos Def
We can't go home again...

We've been to Minnesota three times, and each time it feels less and less like home. The home I grew up in is alien, the house we own isn't our home. Stores open, stores close. Minneapolis has changed and we've changed. It still holds a special place in our heart.

enjoy...

Full Time Web Developers (willing to relocate to the Bay Area)

2008-01-22T00:00:00-08:00

[tags]jobs, yahoo, symfony[/tags]

If anybody (preferably with symfony experience) would like to work with me full time, please send me an email or leave a comment here.

I'm preferably looking for someone with symfony experience that's great with CSS and JS, especially with YUI.

-d

Textmate Snippets for YUI em calculations

2007-05-08T00:00:00-07:00

[tags]yui, yahoo, css, snippet, textmate, ems, px[/tags]

If you use YUI grid layouts you'll notice that ems are the preferred units and for good reason. But ems don't make sense to people like us who want to be super precise down to the pixel... pixels make sense.

So type in a number select it and run this ruby script as a TextMate command (that outputs as a snippet):

You'll have the proper tab stops to change the newly calculated ems from width to margin-left or margin-right or whatever it is you desire.

TextMate + YUI = YUI snippets!

2007-04-30T00:00:00-07:00

I do a lot of YUI grid layouts and I love the nestable grids:

<div class="yui-g$1"> 
    <div class="yui-u first">
        $2
    </div>
    <div class="yui-u">
        $3
    </div>
</div>

There's a tab stop after yui-g in case you want to use one of the variants (yui-gb, yui-gc, etc).

I'm working on a site that uses two equal width columns... a lot... so this comes in quite handy. So long tables.

Going international... kinda

2006-07-06T00:00:00-07:00

Some of the first non-Minnesotan restaurants to show up were Flying Dog, Bangalore and Konstam... all of them outside the US. Wasn't expecting that... but then again, I wasn't really surprised.

I finally updated our location tables to account for different countries. Currently it'll only plot what it can Geocode, and relies exclusively on Yahoo! for GeoCoding.

I've only tested this with a Canadian restaurant. Hopefully it'll work elsewhere soon. If anybody plans on adding any non-US, non-Canadian restaurants, let me know if you can figure out how to GeoCode things properly.

Also, I'm pleased as punch that the map on the homepage shows three states as having recent restaurants. Rock on!

More maps, better presentation and prices

2006-05-05T00:00:00-07:00

I added a lot more features to the reviewsby.us site. One thing I was asked about was adding a price field. I added that in, it could use some work, but it is a start. I also cleaned up some of the location formatting (see Tony Roma's at the MOA) so the phone numbers are legible.

The real exciting thing is maps. Applying the same principles from the Yahoo! Map tutorial, I added maps to all the tag pages. Want to know where to find fried food? Just look at the map. I'm taking a few shortcuts now. For example most of our restaurants are located in Minneapolis, therefore the maps seem to center in on the Twin Cities area. This of course may get messy over time when more restaurants get added outside the Minneapolis area. By then, I'll have some personalization setup to narrow down on restaurants based on where the visitor is located.

I wrote a few tutorials that we should see later this month based on what I did. Fairly simple stuff, but still useful in my opinion.

Enjoy the site, and let me know what you want to see.

Easy Yahoo! Maps and GeoRSS with symfony

2006-04-26T00:00:00-07:00

GeoRSS is an extension of RSS that incorporates geographic data (i.e. latitude/longitude coordinates). This is useful for plotting any data that might need to be placed on a map.

While building out the reviewsby.us map, I decided to use the Yahoo! Maps API versus the Google Maps API because I wanted to gain some familiarity with another API.

It was worth trying Yahoo!'s API. First of all, reviewsby.us has addresses for restaurants and Yahoo! provides a simple Geocoding REST service. This made it easy for me to convert street addresses to latitude and longitude pairs (even though this wasn't required as we'll soon see).¹ The real selling point of Yahoo! was the GeoRSS functionality. I can extend an RSS feed (which symfony generates quite easily) to add latitude or longitude points (or even the street address), direct my Yahoo! map to the feed and voila, all the locations in that feed are now on the map, and when I click on them, the RSS item is displayed. That cut down on a lot of development time.

Yahoo's GeoRSS

The GeoRSS format that Yahoo uses is fairly simple to grasp if you know what an RSS feed looks like. Here's a typical RSS feed:

<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
    <channel>
        <title>Latest restaurants</title>
        <link>http://reviewsby.us/</link>
        <description>A list of the latest restaurants posted to my reviewsby.us</description>
        <language>en</language>
        <item>
            <title>Bubba Gump Shrimp Co. Restaurant and Market</title>
            <description>A *Forest Gump* themed restaurant.  Featuring a large selection of seafood items.</description>
            <link>http://reviewsby.us/restaurant/bubba-gump-shrimp-co-restaurant-and-market</link>
            <guid>25</guid>
            <pubDate>Sun, 23 Apr 2006 08:04:00 -0700</pubDate>
        </item>
        <item>
            <title>Famous Dave's Barbeque</title>
            <link>http://reviewsby.us/restaurant/famous-daves-barbeque</link>
            <guid>24</guid>
            <pubDate>Wed, 19 Apr 2006 19:58:08 -0700</pubDate>
        </item>
    </channel>
</rss>

The <item /> in this example is a restaurant. To turn this into a GeoRSS feed, we only need to change a few things:

<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">
    <channel>
        <title>Latest restaurants' locations</title>
        <link>http://reviewsby.us/</link>
        <description>A geocoded list of the latest restaurants' locations posted to my reviewsby.us</description>
        <language>en</language>
        <item>
            <title>Bubba Gump Shrimp Co. Restaurant and Market (Mall of America (Bloomington, MN))</title>
            <link>http://reviewsby.us/restaurant/bubba-gump-shrimp-co-restaurant-and-market/location/mall-of-america</link>
            <guid>18</guid>
            <pubDate>Sun, 23 Apr 2006 08:08:19 -0700</pubDate>
            <geo:lat>44.85380173</geo:lat>
            <geo:long>-93.24040222</geo:long>
        </item>
    </channel>
</rss>

We just added xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" as an attribute to the <rss/> tag and added <geo:lat /> and <geo:long /> tags to any given item. If you'd rather use address, city, state and zip fields instead of latitude and longitude coordinates, the Yahoo! Georss page will tell you how.

symfony and Geodata

I knew full well that Yahoo! Maps did not require me to have everything in latitude longitude coordinates, but I felt that from an efficiency standpoint, it made more sense for me to convert them once using a geocoder and then Yahoo! wouldn't have to translate them later. Also, I'm trying to think ahead when more than just Minneapolis restaurants become a part of the reviewsby.us site, I now have an easy way of determining distance between a user's home and a given restaurant. Also, if Yahoo! Maps doesn't work out, I can use these coordinates in other mapping systems.

Extending the model

Models in symfony lend themselves to being easily extended. So we can easily take a model for a location:

<table name="location" phpName="Location">
    <column name="id" type="integer" primaryKey="true" autoIncrement="true"/>
    <column name="restaurant_id" type="integer" />
    <foreign-key foreignTable="restaurant">
        <reference local="restaurant_id" foreign="id"/>
    </foreign-key>
    <column name="stripped_title" type="varchar" size="255" />
    <column name="name" type="varchar" size="255" />
    <column name="address" type="varchar" size="255" />
    <column name="city" type="varchar" size="128" />
    <column name="state" type="varchar" size="16" />
    <column name="zip" type="varchar" size="9" />
    <column name="phone" type="varchar" size="16" />
    <column name="approved" type="boolean" />
    <column name="updated_at" type="TIMESTAMP" />
    <column name="created_at" type="TIMESTAMP" />
</table>

and simply add two columns inside the <table/>:

<column name="latitude" type="float" size="10" scale="8"/>
<column name="longitude" type="float" size="10" scale="8"/>

Easy! However, we don't want to have to set the latitude and longitude by hand each time we update a Location. So first we write a function that takes an address and converts it to latitude/longitude. I placed mine in a myTools.class.php in my lib folder:

class myTools
{
    public static function getLatLng($address, $city = null, $state = null, $zip = null)
    {
        $url = sfConfig::get('app_yahoo_geocode');
        $query['appid'] = sfConfig::get('app_yahoo_app_id');
        $query['street'] = $address;
        $query['city'] = $city;
        $query['state'] = $state;
        $query['zip'] = $zip;
        $query['output'] = 'php'; 
        $url .= '?' . http_build_query($query); 
        $response = @file_get_contents($url);
        if ($response) {
            $response = unserialize($response);
            return $response['ResultSet']['Result'];
        }
        return null;
    }
}

Defined in my app.yml is app_yahoo_geocode and my app_yahoo_app_id. myTools::getLatLng() queries the Yahoo! REST service and returns the coordinates that Yahoo! delivers. Note that the generated query string includes output=php. Yahoo! supports serializing output as PHP instead of XML. This can save a bundle of time over decoding XML.

So now let's look at our Location.php and override its inherited save function:

public function save($con = null)
{
    // save latitude and longitude
    $locdata = myTools::getLatLng($this->getAddress(), $this->getCity(), $this->getState(), $this->getZip());
    if ($locdata) 
    {
        $this->setLatitude($locdata['Latitude']);
        $this->setLongitude($locdata['Longitude']);
    }
    parent::save($con);
}

If you stop here, you'll at least now know how to add latitude and longitude coordinates to an object automagically.

Producing a GeoRSS feed

symfony very easily will allow you to generate an RSS feed. How do we create a GeoRSS feed? Just extend the sfFeed class. Rather than instantiating a feed like this:

$feed = sfFeed::newInstance('rss201rev2');

We do this:

$feed = sfFeed::newInstance('geoRSS');

And then create an sfGeoRssFeed.class.php and we're done. We've created a GeoRSS feed fairly easily. Comb through sfGeoRssFeed.class.php and compare it to the sfRss201rev2Feed.class.php, you'll notice it's not that different and that it's fairly easy to extend the sfFeed plugin for symfony.

Adding your feed to a Yahoo! Map.

Adding a GeoRSS feed to Yahoo! maps is simple. Before I embedded the RSS feed into my Yahoo! Map I was prepared to write an algorithm to cluster only on the points in my RSS feed, lucky for me (and you), Yahoo! Maps does this automatically. One pitfall you might reach during development is that Yahoo! Maps must be able to reach your GeoRSS feed. My development machine is my personal laptop, so this didn't work so well until I uploaded to a publically accessible staging server. The maps worked like a charm as you can see.

Conclusion

Yahoo! Maps are very powerful, and symfony is up to the task. I hope you found this tutorial useful. If you have any trouble, let me know. I hope your next meal is a good one.

Using Yahoo! Maps wasn't a requisite of using the Yahoo! Geocoder. The datasets that were returned could have been used to populate a Google Map just as easily. ↩