Dave Dash

Optimizing via YSlow

2009-03-22T00:00:00-07:00

I'm not a performance person per se. I am a heuristics type of guy. Meaning you tell me a bunch of things to look out for, I get an understanding for those things, and if I agree, I'll fix them. I work off lists :)

YSlow is therefore a boon to me. If you aren't familiar with it, it is a plugin for Firefox (with a dependency on Firebug) that will grade the performance of a given site based on a set of rules determined by YDN. This is my list, and YDN went into great detail to explain each of these items is important.

The following is my journey through optimization. I went from a D- to a C in a single day. While that might not seem like much, I could have nabbed an A if I used a CDN, like CloudFront, or even a fake CDN and dropped Google AdSense and Analytics. Those tools are helpful for me, and worth keeping around, so I opted to take the grade penalty.

Begin: D - 61

I began with a D. It's actually not that bad. In most dimensions I was getting an A. Namely because I use a web framework (Django) that is optimized for being cached and served well.

YUI Combo-loading: Sneaking up to D - 62

My first goal was to reduce the HTTP requests.

I use the YUI CSS and javascript framework as the foundation for my CSS and javascript. One major benefit is that they offer to host this for you on a true CDN. The javascript framework is very thorough and can do quite a lot. Because it's so robust, it is split into several files. Originally I had been doing this:

That was 4 separate calls from the web browser to this server. While they are all fast requests, it'd be better to make a single request that catches all of this.

Luckily, YUI has "combo-handling". I was able to turn those four requests into the following:

A very quick fix, that brought my overall grade up by a single point.

Concatenation and Minification: Instant C- - 72

I still had a large number of HTTP requests. Luckily YUI has the YUI Compressor. It compresses javascript and CSS and even gives you tips on writing better code (or more compressible code).

This used to be scary and not something I'd want to deal with, but if I can run it from the commandline, then I can add it to my deployment script. Now on each deployment my CSS and javascript are concatenated and compressed and uploaded to my server. When my app is in production mode, it seeks out minified code, versus more verbose human-readable code.

The javascript compression was great. The output was at least compressed in half. This skyrocketed the score to 72. Less HTTP requests and Javascript minification were the individual rules in YSlow that were boosted.

Google: They want you to fail

This is of course tongue-in-cheek. Google has some wonderful tools like AdSense and Analytics that do hinder the score and ever so slightly the speed of your site.

As far as javascript is concerned, I am linking to: * YUI * My own JS * Google Analytics * Google Adsense

Google Adsense is by far the worst offender. You think you're calling just one solitary call to Google to fetch an ad, but it's actually 4 calls. I'm sure this could be engineered better. The good news is it's usually not too slow.

There's an option to self-serve ga.js the Analytics code. While this might result in a higher score, there are a few issues.

The process is more error prone, there's a chance that the file you download from Google might get corrupted and your unit tests might miss it.
You are serving from your own servers - which could either be expensive or slower.
Many sites include a call for ga.js and therefore there's a high likelihood that it's already browser-cached.
The code could get out of date.

For this reason, it's better to lose about 5 points for not serving it. This score is large because it effects a lot of YSlow's rulesets. If you'd still like to self-serve here's some discussion on that.

This will likely give you an "F" on the Expires Header ruleset as well.

Content Delivery Network (CDN)

A lot of people freak that CDN is on YSlow and feel like not all sites need/require a CDN. They are right that the don't need a CDN, but in my opinion they don't necessarily need to get an "A" either. All this is an approximation of doing the best you can do to make your frontend code as fast and efficient as possible.

With that said, CDN's are becoming more and more in reach. Amazon Cloud Files which I mentioned is a pay as you go system, which is quite affordable (I estimated < $1/month for my needs).

I configured YSlow to list some of the standard Google and Yahoo API hosts as CDNs. This didn't affect my score, and I didn't want to integrate S3/Cloudfiles at this stage, so I took the "F".

Gzip files (still at 72)

Adding Gzip to the Nginx configuration was easy, however my grade still stayed at 72, but it did up my score for that specific rule. Gzip compression is supported in most modern browsers and is usually a config change away.

Again, note that ga.js is not served compressed.

Upgrading from `urchin.js` (up to 73)

In the process of tweaking my Gzip configuration, I noticed that I was still using the legacy urchin.js instead of the more modern ga.js. Simply changing this boosted me to 73.

Move Javascript to the bottom (still at 73)

Moving the javascript to the bottom is another trick. This at least loads most of the HTML before the Javascript loads (and in many cases, waits).

Some refactoring of Django templates makes this a breeze.

Conclusions

I was able to raise the score from a low D to a mid C, at some point I could make the score a high C, but it was inevitably not worth it.

From my perspective, Google could do a few things to help webmasters who use their widgets:

Add a decent expires header to ga.js - this file is safe to cache for some degree of time. Being at the latest and greatest ga.js is nice, but not necessary.
Serve files Gzipped
Use a single host for serving public APIs. Yahoo can improve on this as well. They can potentially do combo-hosting a la Yahoo Developer Network

Ultimately YSlow is a good guideline for speeding up your page render times. It's not perfect, nor exact. There's no realistic way to determine a users experience with page load times. Slow network connectivity and bad browsers can make all your attempts in vain. However, YSlow is an easy to follow set of heuristics which every site owner should attempt to implement as well as possible.

YSlow's future incarnations will yield some more flexibility. Even in its default state it should yield better scores for people who are doing the right thing.

Google Apps: In search of a worthy email system

2008-10-12T00:00:00-07:00

Email, in a lot of ways, is one of the most critical applications for me.

There was a point in my life where I made sure all my email was kept together indefinitely, but as time grew, so did this task.

For the longest time, I had my own hosted @davedash.com, @ftmax.com addresses. These were hosted diligently at pair Networks. pair introduced the qmail system which gave birth to my trend of using "minus addressing" (if your email is user@domain.com you can use user-anything@domain.com). It was my way of seeing which sites spammed me (so far I only outed one company using this trick) and just to organize my inbox.

Gmail...

In 2004, I decided to experiment with Gmail. I had a lot of email to manage and Gmail's ajax-enabled features made its webmail service far superior to any competitors and even made desktop apps cower in shame.

The ideology of search don't sort, and archive don't delete, made Gmail a perfect storage system for all my emails. I never had to lose track of email again. Just about any conversation, or discussion I've had over email - I can pull up very quickly using Gmail.

I'd say after 4 years of solid use, the experiment is over, I was sold on Gmail.

Switching hosts.

I did, however, move my domain management over to Dreamhost. This is where it still resides, and that was a problem. Minus addressing was not a part of Dreamhost. I had to painstakingly create procmail files that simulated this. Inevitably pushing anything and everything to my gmail account if it matched the syntax I chose.

I knew Gmail supported plus addressing (user+anything@gmail.com), but I never took the plunge. My habit, was and is still minus style.

Google Apps

The Dreamhost account I used has been tossed from one server to the next, unfortunately there's a lot of hand-holding I need to do in order to make sure procmail and all my other tweaks I've made to my accounts work. I finally threw in the towel today and settled on Google Apps.

Both my wife and I use Gmail (with aliased emails @katiebonn.com and @davedash.com respectively), so I enabled an app and did the painstaking migration process.

While it's not that complicated, you don't ever realize how entangled your Google account can be. We both use GTalk and Google Calendar and we both have large Gmail accounts. Mine has 70K+ addresses and is slowly being migrated to my Google Apps email address.

In the end, I think this struggle will be worth it, since I'll have my addresses managed in a clean efficient way.

This will also pave the path for me to move more of my web sites off of Dreamhost and onto Slicehost.

Wish me luck.

Easy Yahoo! Maps and GeoRSS with symfony

2006-04-26T00:00:00-07:00

GeoRSS is an extension of RSS that incorporates geographic data (i.e. latitude/longitude coordinates). This is useful for plotting any data that might need to be placed on a map.

While building out the reviewsby.us map, I decided to use the Yahoo! Maps API versus the Google Maps API because I wanted to gain some familiarity with another API.

It was worth trying Yahoo!'s API. First of all, reviewsby.us has addresses for restaurants and Yahoo! provides a simple Geocoding REST service. This made it easy for me to convert street addresses to latitude and longitude pairs (even though this wasn't required as we'll soon see).¹ The real selling point of Yahoo! was the GeoRSS functionality. I can extend an RSS feed (which symfony generates quite easily) to add latitude or longitude points (or even the street address), direct my Yahoo! map to the feed and voila, all the locations in that feed are now on the map, and when I click on them, the RSS item is displayed. That cut down on a lot of development time.

Yahoo's GeoRSS

The GeoRSS format that Yahoo uses is fairly simple to grasp if you know what an RSS feed looks like. Here's a typical RSS feed:

<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
    <channel>
        <title>Latest restaurants</title>
        <link>http://reviewsby.us/</link>
        <description>A list of the latest restaurants posted to my reviewsby.us</description>
        <language>en</language>
        <item>
            <title>Bubba Gump Shrimp Co. Restaurant and Market</title>
            <description>A *Forest Gump* themed restaurant.  Featuring a large selection of seafood items.</description>
            <link>http://reviewsby.us/restaurant/bubba-gump-shrimp-co-restaurant-and-market</link>
            <guid>25</guid>
            <pubDate>Sun, 23 Apr 2006 08:04:00 -0700</pubDate>
        </item>
        <item>
            <title>Famous Dave's Barbeque</title>
            <link>http://reviewsby.us/restaurant/famous-daves-barbeque</link>
            <guid>24</guid>
            <pubDate>Wed, 19 Apr 2006 19:58:08 -0700</pubDate>
        </item>
    </channel>
</rss>

The <item /> in this example is a restaurant. To turn this into a GeoRSS feed, we only need to change a few things:

<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">
    <channel>
        <title>Latest restaurants' locations</title>
        <link>http://reviewsby.us/</link>
        <description>A geocoded list of the latest restaurants' locations posted to my reviewsby.us</description>
        <language>en</language>
        <item>
            <title>Bubba Gump Shrimp Co. Restaurant and Market (Mall of America (Bloomington, MN))</title>
            <link>http://reviewsby.us/restaurant/bubba-gump-shrimp-co-restaurant-and-market/location/mall-of-america</link>
            <guid>18</guid>
            <pubDate>Sun, 23 Apr 2006 08:08:19 -0700</pubDate>
            <geo:lat>44.85380173</geo:lat>
            <geo:long>-93.24040222</geo:long>
        </item>
    </channel>
</rss>

We just added xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" as an attribute to the <rss/> tag and added <geo:lat /> and <geo:long /> tags to any given item. If you'd rather use address, city, state and zip fields instead of latitude and longitude coordinates, the Yahoo! Georss page will tell you how.

symfony and Geodata

I knew full well that Yahoo! Maps did not require me to have everything in latitude longitude coordinates, but I felt that from an efficiency standpoint, it made more sense for me to convert them once using a geocoder and then Yahoo! wouldn't have to translate them later. Also, I'm trying to think ahead when more than just Minneapolis restaurants become a part of the reviewsby.us site, I now have an easy way of determining distance between a user's home and a given restaurant. Also, if Yahoo! Maps doesn't work out, I can use these coordinates in other mapping systems.

Extending the model

Models in symfony lend themselves to being easily extended. So we can easily take a model for a location:

<table name="location" phpName="Location">
    <column name="id" type="integer" primaryKey="true" autoIncrement="true"/>
    <column name="restaurant_id" type="integer" />
    <foreign-key foreignTable="restaurant">
        <reference local="restaurant_id" foreign="id"/>
    </foreign-key>
    <column name="stripped_title" type="varchar" size="255" />
    <column name="name" type="varchar" size="255" />
    <column name="address" type="varchar" size="255" />
    <column name="city" type="varchar" size="128" />
    <column name="state" type="varchar" size="16" />
    <column name="zip" type="varchar" size="9" />
    <column name="phone" type="varchar" size="16" />
    <column name="approved" type="boolean" />
    <column name="updated_at" type="TIMESTAMP" />
    <column name="created_at" type="TIMESTAMP" />
</table>

and simply add two columns inside the <table/>:

<column name="latitude" type="float" size="10" scale="8"/>
<column name="longitude" type="float" size="10" scale="8"/>

Easy! However, we don't want to have to set the latitude and longitude by hand each time we update a Location. So first we write a function that takes an address and converts it to latitude/longitude. I placed mine in a myTools.class.php in my lib folder:

class myTools
{
    public static function getLatLng($address, $city = null, $state = null, $zip = null)
    {
        $url = sfConfig::get('app_yahoo_geocode');
        $query['appid'] = sfConfig::get('app_yahoo_app_id');
        $query['street'] = $address;
        $query['city'] = $city;
        $query['state'] = $state;
        $query['zip'] = $zip;
        $query['output'] = 'php'; 
        $url .= '?' . http_build_query($query); 
        $response = @file_get_contents($url);
        if ($response) {
            $response = unserialize($response);
            return $response['ResultSet']['Result'];
        }
        return null;
    }
}

Defined in my app.yml is app_yahoo_geocode and my app_yahoo_app_id. myTools::getLatLng() queries the Yahoo! REST service and returns the coordinates that Yahoo! delivers. Note that the generated query string includes output=php. Yahoo! supports serializing output as PHP instead of XML. This can save a bundle of time over decoding XML.

So now let's look at our Location.php and override its inherited save function:

public function save($con = null)
{
    // save latitude and longitude
    $locdata = myTools::getLatLng($this->getAddress(), $this->getCity(), $this->getState(), $this->getZip());
    if ($locdata) 
    {
        $this->setLatitude($locdata['Latitude']);
        $this->setLongitude($locdata['Longitude']);
    }
    parent::save($con);
}

If you stop here, you'll at least now know how to add latitude and longitude coordinates to an object automagically.

Producing a GeoRSS feed

symfony very easily will allow you to generate an RSS feed. How do we create a GeoRSS feed? Just extend the sfFeed class. Rather than instantiating a feed like this:

$feed = sfFeed::newInstance('rss201rev2');

We do this:

$feed = sfFeed::newInstance('geoRSS');

And then create an sfGeoRssFeed.class.php and we're done. We've created a GeoRSS feed fairly easily. Comb through sfGeoRssFeed.class.php and compare it to the sfRss201rev2Feed.class.php, you'll notice it's not that different and that it's fairly easy to extend the sfFeed plugin for symfony.

Adding your feed to a Yahoo! Map.

Adding a GeoRSS feed to Yahoo! maps is simple. Before I embedded the RSS feed into my Yahoo! Map I was prepared to write an algorithm to cluster only on the points in my RSS feed, lucky for me (and you), Yahoo! Maps does this automatically. One pitfall you might reach during development is that Yahoo! Maps must be able to reach your GeoRSS feed. My development machine is my personal laptop, so this didn't work so well until I uploaded to a publically accessible staging server. The maps worked like a charm as you can see.

Conclusion

Yahoo! Maps are very powerful, and symfony is up to the task. I hope you found this tutorial useful. If you have any trouble, let me know. I hope your next meal is a good one.

Using Yahoo! Maps wasn't a requisite of using the Yahoo! Geocoder. The datasets that were returned could have been used to populate a Google Map just as easily. ↩