Dave Dash

Delicious keeps you in the know

2009-08-04T00:00:00-07:00

My last task at Delicious was to build along with the amazing Vik Singh was to build a new feed of bookmarks that was heavily influenced by Twitter. It was one of the most interesting and enjoyable pieces of code that I worked on at Delicious.

Over two months since my final check-in, the code is now in production. It is mostly as intended, but is lacking an RSS or JSON feed (which I had already built). This is somewhat disappointing since I was hoping that Delicious would remain as open as it had been in the past.

The algorithm is fairly simple we take a look at what trending topics exist at any moment in time (via Google Trends and Twitter) and we combine it with a list of popular terms. We take the whole lot of these items and query search twitter and store an in-memory data table of tweets. We also take a snapshot of new URLs to the Delicious corpus (basically anything on Delicious recent with 1 save). We cluster the Delicious URLs and then find tweets that are similar to each of these clusters.

The code for this is similar to Vik's TweetNews - but I think the Delicious data is a nicer fit.

We optimized this quite a bit and built a very fast inverted-index and tweaked the code to run in about a minute. Like TweetNews the heart of this was built using Python. Python while being a dynamic language is quite amazing for manipulating and iteratiting over sets of data.

While building this tool, it became my way to feel pulse of what's going on. I could ditch a lot of my RSS feeds and rely solely on Delicious to be on the up and up. Unfortunately I can't subscribe to a feed for this. Either delicious has made a mistake and didn't launch their feeds at the same time as their web (entirely possible, since Delicious hasn't been updated for most of 2009) or they are deliberately taking a step backwards.

This step backwards is weird from the usability issue. Delicious has always been a tool that allowed for multiple types of consumers and a tool that appealed to developers thanks to its myriad of RSS and JSON feeds. I'm glad I didn't have to be on the losing side of that decision. Delicious relies heavily on Google Trends and Twitter Search. While there is no requirement for them to share the data they are mashing up, it would be the right thing to do.

Let me know what you think of the new feeds. I wish I could share a github link or something snazzy so you could play around with it, but this post should be a good starting point for other real-time data mashups.

Update: Read Vik's account of this on the Delicious Blog.

Fixing the DiggBar with Delicious

2009-04-02T00:00:00-07:00

I've been tooling around with the DiggBar and I've been quite pleased... but I was a bit annoyed that it was hard to bookmark things in Delicious via the digg bar. I'd have to either close bar and then bookmark in delicious, or right click on the url in the DiggBar and then change the title.

I'm fundamentally a lazy programmer, so I wrote some greasemonkey to save me some time. Hooray... diggBar+Delicious.

Why code rewrites are often coupled with redesigns

2008-08-13T00:00:00-07:00

I've done some rewrites of code, and they usually are coupled with redesigns.

Redesigns and rewrites are tricky. With web sites existing users tend to prefer incremental changes with each. Changing a design element or a feature are pretty much equivalent when its done incrementally.

Usually small changes are like surgery. You make a small change, you possibly announce it, and people look at it and usually say yes, this change is better. But mostly a small change can go unnoticed.

Tiny atomic changes are nice and they improve the product.

A rewrite of code however is a discrete change. They take forever. Second system and all that. They inevitably involve a redesign as well. They tend to be received with mixed reviews.

Generally a design of a site is grafted onto code. Sometimes by templating languages, sometimes in a tightly coupled system. In either case there's usually two options:

Make a faithful port of the design.
Redesign

The latter is generally preferred since this saves a lot of time. Generally there are bugs with a design, and there is a design backlog. A faithful port of the design would mean porting every broken HTML code, every non user-friendly element, etc.

Unfortunately this leaves web developers with a scary situation: Launching a product which changes peoples functional and UI experience.

The good thing is redesigns usually mean evolutionary leaps. Meaning, iterations can happen faster, both in design and functionality. We can easily please new customers, and we can quickly bring current customers up to speed.

In my spare time, I've been rewriting code that I wrote in symfony (PHP) into django (python). It's a tiresome long process that can easily kill a side project.

... and the new Delicious has landed

2008-07-31T00:00:00-07:00

Working at Delicious with the team has been a wonderful experience. This is probably the most fun I've had working on a project. I am really glad to work on a site that I use almost daily (and regularly if not daily before I joined) with a team that really cares about their product.

So without further ado...

Delicious 2.0

Here's the official announcement.

And here's some photos I took.

Optimization heuristics

2008-02-26T00:00:00-08:00

I decided to play version control private-eye today when my coworker mentioned that we make a system call to check the time several times per request on a few specific pages.

My analysis was we didn't need to have the calls but it had me thinking...

The micro-caching heuristic

One way of optimizing code is by calculating expensive operations once and storing them in a specific location. It's a micro-cache if you will of data.

One obvious example ends up being loops:

The inefficient example computes count($arr) n times where n is the size of the array $arr. The latter computes it once.

When we write and review code we should look out for loops and repeated code. These elements can be pulled out.

With some data lookups like system calls or queries we can apply a request-wide caching. In a symfony app, for example, we might need to calculate the time stamp in several locations throughout the code. Rather than calling time() throughout the code, we can create a site wide accessible attribute that has the same value.

For example, we can subclass sfContext and add a method called getTime() which runs time() once and only once per request and stores it internally. Whenever part of the app needs the time stamp, myContext::getInstance()->getTime() (or an equivalent shortcut) can be called. Of course, I haven't tested this, and myContext might not be the best place for this, but a similar strategy will work.

-dd