Dave Dash

Yahoo Pipes and reducing information overload

2008-02-28T00:00:00-08:00

I've been suffering from some information overload. I subscribed to Engadget and Gizmodo because I wanted to keep up with some home based network devices like homeplug/powerline and wireless routers. But Engadget and Gizmodo are overwhelming.

Finally I decided... to write a pipe. If you're not familiar with Yahoo's Pipes it is the best thing to come out of Yahoo! (Flickr and del.icio.us are acquisitions originally). Pipes is what Yahoo! should be doing:

It's niche oriented: It appeals highly to geeks who understand pipes, yet it's very learnable, so you don't need to know how to program to use it.
It's got a great UI. The user interface is great, you drag and drop inputs and outputs - again very learnable.
It's very useful. As I'll show you, below.
It harnesses the technology that a large company that Yahoo! can provide that might otherwise be difficult: Pipes has to store a lot of data from RSS feeds, to screen scrapes, to anything that you feed it. Using Yahoo! technology it's trivial to obtain and process all that.
It's not a clone or a "me too" of other products: This is completely original, and very appealing.

So my pipes take a regexp filter (e.g. "delicious" or if your clever "delicious|Yahoo|Google") and give me items that match in a selection of RSS feeds. So it aggregates things like Engadget and Gizmodo and gives you just what you want to see:

Take a look: Gadget Filter

I also made a copy called Technology Filter which aggregates the SF Chronicle and TechCrunch.

Try them out, remember you can customize the filters... not everyone likes what I like ;)

Make Adium show your Facebook status

2007-02-03T00:00:00-08:00

By using a simple RSS feed, you can have Adium report your Facebook status.

I am a late-adopter to social networks. I participated in LiveJournal years after many of my friends started using it. I finally got into Facebook months after it was "openned up." I like this strategy because I can immediately find my friends on the network and add them and amass 100s in a few short days.

Facebook is nice for its simple and clean interface. Even the features are fairly simple. I like the status feature... for no good reason. It's a simple AJAX status update that updates your status on the site. What I don't like to do is repeat myself. I wanted Adium to repeat what Facebook said.

My first inkling was to use the Facebook API, but that seemed less appealing since it would involve authentication. I almost gave up and then discovered that in Facebook you can go to "My Profile" and under "Status" go to "See All" and there's an RSS link to your status messages. Awesome. All I needed to do is grab the first one.

I decided this looked like a task for Simple XML which can take an XML string and turn it into a PHP object.

The first step of course was to fetch the RSS feed into a string. file_get_contents made the most sense, but Facebook does a browser check even for getting RSS feeds. I'm not fond of browser sniffing, but its easy to work around if you use cURL.

Excellent... this does exactly what I want - but it's slow. Like any good script, we should use caching if it makes sense. Obviously we don't want to overload the servers at Facebook (or for that matter any place that serves up RSS feeds) so we implement PEAR's Cache_Lite package. We'll tell it to cache results for 15 minutes. The code changes are marked with //+++ at the end of each new line:

<?php

require_once('Cache/Lite.php'); //+++
    
    $url = "http://mynework.facebook.com/feeds/status.php?replace=with&your=own&feed=url";

$options = array('cacheDir' => '/tmp/', 'lifeTime' => 600); //+++

$Cache_Lite = new Cache_Lite($options); //+++

$data = null; //+++
    // attempt to load the data from cache,  //+++
    // otherwise load it anew from RSS //+++
    if (!($data = $Cache_Lite->get($id))) { //+++
    // setup curl
        $ch = curl_init();
        curl_setopt ($ch, CURLOPT_URL, $url);
        curl_setopt ($ch, CURLOPT_HEADER, 0);

//spoof Firefox
        curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.1) Gecko/20061223 Firefox/2.0.0.1");

// begin output buffering
        ob_start();
        curl_exec ($ch);
        curl_close ($ch);
        // save buffer to string
        $xmlstr = ob_get_contents();

ob_end_clean();

// convert string to xml object
        $xml = new SimpleXMLElement($xmlstr);

// status messages start with 'Dave is...' 
        // I just want everything after my name
        $data = str_replace('Dave ', null,$xml->channel->item->title); // +++

$Cache_Lite->save($data); //+++

} //+++

echo $data; //+++

That's it on the PHP side. On the Adium side you'll need to use the exec AdiumXtra. It'll allow you to set your status to something like:

/exec {/usr/local/php5/bin/php /usr/local/bin/facebook.php}

Enjoy.

Easy Yahoo! Maps and GeoRSS with symfony

2006-04-26T00:00:00-07:00

GeoRSS is an extension of RSS that incorporates geographic data (i.e. latitude/longitude coordinates). This is useful for plotting any data that might need to be placed on a map.

While building out the reviewsby.us map, I decided to use the Yahoo! Maps API versus the Google Maps API because I wanted to gain some familiarity with another API.

It was worth trying Yahoo!'s API. First of all, reviewsby.us has addresses for restaurants and Yahoo! provides a simple Geocoding REST service. This made it easy for me to convert street addresses to latitude and longitude pairs (even though this wasn't required as we'll soon see).¹ The real selling point of Yahoo! was the GeoRSS functionality. I can extend an RSS feed (which symfony generates quite easily) to add latitude or longitude points (or even the street address), direct my Yahoo! map to the feed and voila, all the locations in that feed are now on the map, and when I click on them, the RSS item is displayed. That cut down on a lot of development time.

Yahoo's GeoRSS

The GeoRSS format that Yahoo uses is fairly simple to grasp if you know what an RSS feed looks like. Here's a typical RSS feed:

<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
    <channel>
        <title>Latest restaurants</title>
        <link>http://reviewsby.us/</link>
        <description>A list of the latest restaurants posted to my reviewsby.us</description>
        <language>en</language>
        <item>
            <title>Bubba Gump Shrimp Co. Restaurant and Market</title>
            <description>A *Forest Gump* themed restaurant.  Featuring a large selection of seafood items.</description>
            <link>http://reviewsby.us/restaurant/bubba-gump-shrimp-co-restaurant-and-market</link>
            <guid>25</guid>
            <pubDate>Sun, 23 Apr 2006 08:04:00 -0700</pubDate>
        </item>
        <item>
            <title>Famous Dave's Barbeque</title>
            <link>http://reviewsby.us/restaurant/famous-daves-barbeque</link>
            <guid>24</guid>
            <pubDate>Wed, 19 Apr 2006 19:58:08 -0700</pubDate>
        </item>
    </channel>
</rss>

The <item /> in this example is a restaurant. To turn this into a GeoRSS feed, we only need to change a few things:

<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">
    <channel>
        <title>Latest restaurants' locations</title>
        <link>http://reviewsby.us/</link>
        <description>A geocoded list of the latest restaurants' locations posted to my reviewsby.us</description>
        <language>en</language>
        <item>
            <title>Bubba Gump Shrimp Co. Restaurant and Market (Mall of America (Bloomington, MN))</title>
            <link>http://reviewsby.us/restaurant/bubba-gump-shrimp-co-restaurant-and-market/location/mall-of-america</link>
            <guid>18</guid>
            <pubDate>Sun, 23 Apr 2006 08:08:19 -0700</pubDate>
            <geo:lat>44.85380173</geo:lat>
            <geo:long>-93.24040222</geo:long>
        </item>
    </channel>
</rss>

We just added xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" as an attribute to the <rss/> tag and added <geo:lat /> and <geo:long /> tags to any given item. If you'd rather use address, city, state and zip fields instead of latitude and longitude coordinates, the Yahoo! Georss page will tell you how.

symfony and Geodata

I knew full well that Yahoo! Maps did not require me to have everything in latitude longitude coordinates, but I felt that from an efficiency standpoint, it made more sense for me to convert them once using a geocoder and then Yahoo! wouldn't have to translate them later. Also, I'm trying to think ahead when more than just Minneapolis restaurants become a part of the reviewsby.us site, I now have an easy way of determining distance between a user's home and a given restaurant. Also, if Yahoo! Maps doesn't work out, I can use these coordinates in other mapping systems.

Extending the model

Models in symfony lend themselves to being easily extended. So we can easily take a model for a location:

<table name="location" phpName="Location">
    <column name="id" type="integer" primaryKey="true" autoIncrement="true"/>
    <column name="restaurant_id" type="integer" />
    <foreign-key foreignTable="restaurant">
        <reference local="restaurant_id" foreign="id"/>
    </foreign-key>
    <column name="stripped_title" type="varchar" size="255" />
    <column name="name" type="varchar" size="255" />
    <column name="address" type="varchar" size="255" />
    <column name="city" type="varchar" size="128" />
    <column name="state" type="varchar" size="16" />
    <column name="zip" type="varchar" size="9" />
    <column name="phone" type="varchar" size="16" />
    <column name="approved" type="boolean" />
    <column name="updated_at" type="TIMESTAMP" />
    <column name="created_at" type="TIMESTAMP" />
</table>

and simply add two columns inside the <table/>:

<column name="latitude" type="float" size="10" scale="8"/>
<column name="longitude" type="float" size="10" scale="8"/>

Easy! However, we don't want to have to set the latitude and longitude by hand each time we update a Location. So first we write a function that takes an address and converts it to latitude/longitude. I placed mine in a myTools.class.php in my lib folder:

class myTools
{
    public static function getLatLng($address, $city = null, $state = null, $zip = null)
    {
        $url = sfConfig::get('app_yahoo_geocode');
        $query['appid'] = sfConfig::get('app_yahoo_app_id');
        $query['street'] = $address;
        $query['city'] = $city;
        $query['state'] = $state;
        $query['zip'] = $zip;
        $query['output'] = 'php'; 
        $url .= '?' . http_build_query($query); 
        $response = @file_get_contents($url);
        if ($response) {
            $response = unserialize($response);
            return $response['ResultSet']['Result'];
        }
        return null;
    }
}

Defined in my app.yml is app_yahoo_geocode and my app_yahoo_app_id. myTools::getLatLng() queries the Yahoo! REST service and returns the coordinates that Yahoo! delivers. Note that the generated query string includes output=php. Yahoo! supports serializing output as PHP instead of XML. This can save a bundle of time over decoding XML.

So now let's look at our Location.php and override its inherited save function:

public function save($con = null)
{
    // save latitude and longitude
    $locdata = myTools::getLatLng($this->getAddress(), $this->getCity(), $this->getState(), $this->getZip());
    if ($locdata) 
    {
        $this->setLatitude($locdata['Latitude']);
        $this->setLongitude($locdata['Longitude']);
    }
    parent::save($con);
}

If you stop here, you'll at least now know how to add latitude and longitude coordinates to an object automagically.

Producing a GeoRSS feed

symfony very easily will allow you to generate an RSS feed. How do we create a GeoRSS feed? Just extend the sfFeed class. Rather than instantiating a feed like this:

$feed = sfFeed::newInstance('rss201rev2');

We do this:

$feed = sfFeed::newInstance('geoRSS');

And then create an sfGeoRssFeed.class.php and we're done. We've created a GeoRSS feed fairly easily. Comb through sfGeoRssFeed.class.php and compare it to the sfRss201rev2Feed.class.php, you'll notice it's not that different and that it's fairly easy to extend the sfFeed plugin for symfony.

Adding your feed to a Yahoo! Map.

Adding a GeoRSS feed to Yahoo! maps is simple. Before I embedded the RSS feed into my Yahoo! Map I was prepared to write an algorithm to cluster only on the points in my RSS feed, lucky for me (and you), Yahoo! Maps does this automatically. One pitfall you might reach during development is that Yahoo! Maps must be able to reach your GeoRSS feed. My development machine is my personal laptop, so this didn't work so well until I uploaded to a publically accessible staging server. The maps worked like a charm as you can see.

Conclusion

Yahoo! Maps are very powerful, and symfony is up to the task. I hope you found this tutorial useful. If you have any trouble, let me know. I hope your next meal is a good one.

Using Yahoo! Maps wasn't a requisite of using the Yahoo! Geocoder. The datasets that were returned could have been used to populate a Google Map just as easily. ↩