Dave Dash

Versioning Django Models

2009-02-13T00:00:00-08:00

In symfony, versioning a model was not terribly difficult. I had my own specialized brute-force way of doing this.

In my experience it's been a lot easier to write python code than PHP code, so naturally I figured this would be an easy task. It was not. I suspect that it was not easy because I have a naive understanding of Django and my symfony knowledge was fairly well grounded.

I tried looking for a generic implementation of django model versioning, but failed. So I came up with a specific solution.

How I think of versions

For my app I chose to use a sparse versioning system. I'd have one interface for interacting with a model. For example, I have a Restaurant model and a RestaurantVersion model.

Restaurant would directly store immutable elements like name or rating or approved and it would encapsulate versioned elements like description by proxying to RestaurantVersion.

This means I only need to interface with one class and let the versioning happen behind the scenes.

The Django implementation of `RestaurantVersion`

RestaurantVersion held the data that I thought might suffer from corruption, and therefore require reversion. Therefore there's nothing surprising in this table:

The only thing of note is that I'm enforcing a 1:M relation between Restaurant and RestaurantVersion.

The Django implementation of `Restaurant`

Restaurant is where the magic happens. It needs to do a few things:

Do what a normal object does.
Proxy attributes to the corresponding attribute of a RestaurantVersion
Generate a new RestaurantVersion on demand.
(optional) Manage which version is "active".

I say "4" is optional because I don't do this myself. I built versioning into a few models because it was easier to do upfront than worry about it down the road.

Here's the code I use:

class Restaurant(models.Model):
    name           = models.CharField(max_length=765, blank=True)
    stripped_title = models.CharField(max_length=384, blank=True)
    approved       = models.IntegerField(null=True, blank=True)
    version        = models.ForeignKey(RestaurantVersion, related_name="the_restaurant")
    updated_at     = models.DateTimeField(auto_now=True)
    created_at     = models.DateTimeField(auto_now_add=True)
    new_version    = None
    
    def save(self, force_insert=False, force_update=False):
        if not self.stripped_title:
            self.stripped_title = slugify(self.name)
        
        super(Restaurant, self).save(force_insert, force_update)
        if self.new_version:
            self.new_version.restaurant = self
            self.new_version.save()
            self.version = self.new_version
            super(Restaurant, self).save(force_insert, force_update)        
        
    def __setattr__(self, name, value):
        if name == 'description':
            self.get_new_version().description = value
        
        elif name == 'url':
            self.get_new_version().url = value
        
        else:
            object.__setattr__(self, name, value)

def __getattr__(self, name):
        try:
            if name == 'description':
                return self.version.description

elif name == 'url':
                return self.version.url
        except RestaurantVersion.DoesNotExist:
            return ''
            
        models.Model.__getattribute__(self, name)
    
    def get_new_version(self):
        if self.new_version == None:
            try:
                rv    = self.version
                rv.id = None
            except RestaurantVersion.DoesNotExist:
                rv = RestaurantVersion()
            
            self.new_version = rv

return self.new_version
            
    class Meta:
        db_table     = u'restaurant'

It's not rocket science, but it wasn't necessarily easy either. Let's look at requirements 2 and 4 in a bit more detail.

Proxy attributes to the corresponding `RestaurantVersion` attributes

Proxying attributes is a way of masking the whole versioning infrastructure from the developer. We do this with __getattr__ and __setattr__ methods.

The __getattr__ determins if you are looking for description or url attributes of a Restaurant. If you are then it uses the current RestaurantVersion's attribute.

__setattr__ is very similar, except since we're changing a versionable attribute we're going to make a request to a method, get_new_version(), which we will detail later. It does what it says, though, it gets the "new" RestaurantVersion of the Restaurant.

Generate a new `RestaurantVersion` on demand

As you can see from the above code, I am "auto-versioning". This is done mostly via get_new_version(). We also have to do a few tricks to make sure the bidirectional relationship gets maintained on save.

get_new_version() either returns the current "new version", a brand new "new version" or a "copy" of the current version.

The "copy" is done simply by setting the id attribute of the new version to None. Django takes care of assigning it a new id on save, thus preserving the old version.

Note that when we create a brand new RestaurantVersion, we don't immediately set it's restaurant attribute. That's because we haven't saved the current restaurant yet. It's a "chicken and the egg" problem that gets solved in our save() method:

We first save the Restaurant, then we save a new version if there is one. If we save a new version we want to update the Restaurant a second time. This ensures that there's a 1:1 relationship between Restaurant and the current RestaurantVersion and it also establishes a 1:M relationship between Restaurant and all RestaurantVersions.

Conclusion and Drawbacks

While, I think that this setup works in my particular situation, I feel like it has some major flaws.

The code is quite messy and very specific to this model. My goal was to take care of this quickly, not necessarily in a reusable manner. I also was not familiar enough with the Django model to create some sort of extension.

This way of versioning does not allow for versioned attributes to be set upon instantiation of the model:

r=Restaurant(description="Great place")

won't work. You'll have to do:

r=Restaurant()
r.description="Great place"

I figured this was acceptable.

Lastly I'm not entirely happy with having to explicitly save the Restaurant model twice, but I think my bidirectional relation requires this.

All in all this works in my particular situation. I'm hoping that this can be simplified. I'm curious to hear about other versioning schemes.

PHPs strengths: array_count_values

2008-12-07T00:00:00-08:00

I always like to think of what different interpreted programming languages bring to the table.

Perl is great with string manipulation. I tend to use perl if I need to rewrite/reformat text.

Python makes it easy to write clean, organized code and has a lot of syntactical sugar like list comprehensions.

For PHP one of my coworkers suggested that maybe it's ease is what it brings to the table. I've accepted that wryly at first, but after some thought it's actually a real strength.

PHP has positioned itself to be easily adoptable. When I learned PHP/FI 2.0 it was because I had exhausted server-side includes and needed something better. PHP/FI was a great solution because it was easy to install (my ISP had it) and documented easy things that I could do. After all it was just some additional changes to my templates.

I never for once thought that I'd be paid at a company to code in it... but then the internet just got big.

The other way php positioned itself is the library of functions it has out of the box. Today I found myself looking for a python equivalent to array_count_values.

Its a function I wouldn't use every day, but there are rare instances where having a set of data aggregated by count would be very useful. That's exactly what this function does.

Even python's lists, sets and dictionaries, which can do with simple operations what PHP needs an army of array_ functions to do, didn't reveal anything useful. So I will use the following:

Optimization heuristics

2008-02-26T00:00:00-08:00

I decided to play version control private-eye today when my coworker mentioned that we make a system call to check the time several times per request on a few specific pages.

My analysis was we didn't need to have the calls but it had me thinking...

The micro-caching heuristic

One way of optimizing code is by calculating expensive operations once and storing them in a specific location. It's a micro-cache if you will of data.

One obvious example ends up being loops:

The inefficient example computes count($arr) n times where n is the size of the array $arr. The latter computes it once.

When we write and review code we should look out for loops and repeated code. These elements can be pulled out.

With some data lookups like system calls or queries we can apply a request-wide caching. In a symfony app, for example, we might need to calculate the time stamp in several locations throughout the code. Rather than calling time() throughout the code, we can create a site wide accessible attribute that has the same value.

For example, we can subclass sfContext and add a method called getTime() which runs time() once and only once per request and stores it internally. Whenever part of the app needs the time stamp, myContext::getInstance()->getTime() (or an equivalent shortcut) can be called. Of course, I haven't tested this, and myContext might not be the best place for this, but a similar strategy will work.

-dd

py vs php: stemming

2008-02-16T00:00:00-08:00

I've been porting some PHP to python during SuperHappyDevHouse and was amazed at how little code I needed to write since python makes list manipulation a breeze.

Today I was working on stemming (ala Porter Stemming algorithm). reviewsby.us uses stemming in the search engine to make queries:

Stemming turns hello everybody how are you guy's into a collection 'everybodi', 'gui', 'hello'. To produce this in php I do the following:

With some magic python:

The magic here is list mappings. Learning about them, they don't seem that great, but as soon as you start coding you stop using a lot of for loops.

I'm sure my PHP can be cleaned up and reduced as well, but its fun exploiting the magic of languages.

the magic of django: get_callable

2008-02-08T00:00:00-08:00

symfony is an amazing framework simply because of the uphill battle it has with PHP. There's a lot of behind the scenes magic that ameliorates the frustrations one might have with PHPs lack of syntactical magic.

A lot of work is done to make sure certain things are autoloaded and that certain things behave the way you expect.

I hadn't had a chance to peek under Django's hood too much myself, but it appears that despite the inherent magic of python, there were some much needed additions. My savior for today was get_callable.

I have a very particular vision of interacting with OpenID. Basically I like to decouple as much of the OpenID interactions away from the inner workings of my web site. I do, however, want a hook where I can supply some of my own logic after OpenID has been verified.

It's roughly how I implimented the sfOpenIDPlugin. Luckily this time, I was able to rely on the OpenID 2.1 libraries from OpenIDEnabled.

I did ditch their example code, because I wanted a sexy package. The package isn't quite ready for primetime, but you can have a gander (and yes, I do welcome collaborators and am open to merging my work with the myriad of work that's out there).

Getting to the point

Anyway, get_callable is what allowed me to write my "hook" into my OpenID logic.

In my settings.py I defined:

OPENID_SUCCESS = 'myproject.myapp.views.openid_handler'

The way my OpenID library works, is it does it's business and when it verifies a url belongs to someone it then delivers he request to the OPENID_SUCCESS hook. Here's how:

if settings.OPENID_SUCCESS:
    view = get_callable(settings.OPENID_SUCCESS)
    return view(request, result)

get_callable works some python magic by splitting myproject.myapp.views.openid_handler into the module: myproject.myapp.views and the attribute openid_handler. It uses this to produce a callable object. In the example above I gave it parameters of request and result as they were required by my app.

Why Django Templating is awesome and why I get smarty again

2008-02-01T00:00:00-08:00

I get Smarty thanks to django... yeah, it's weird.

Back to my original comment about templating, smarty really is trying to limit the scope of PHP in a good way. Too often I see a lot of heavy-lifting in the templates. It's so bad it makes my MVC'd brain explode.

Django templates are very limited, based on a philosophy of keeping view logic and only view logic in the templates.

This is what smart tries to do and it's a reasonable solution to the fact that PHP is a templating language with a kitchen sink. It's saying, okay... well let's treat PHP as a programming language, and keep Smarty as the template.

symfony of course says (well not really, or in any official capacity, but would you believe... frameworks talk to me), some PHP is view and some is model/controller. We'll suggest which ones are which, but really we're not getting in the way of making your life a living hell by sticking complicated business logic inline.

Templating

2008-01-28T00:00:00-08:00

[tags]smarty, symfony, php, python, django[/tags]

I used to use the Smarty templating system quite heavily in my PHP apps. Then after switching over to symfony, I broke that habit.

PHP is a templating language for better or worse. Smarty was a way of saying that "well things could be easier", but that's against the grain of what PHP fundamentally is.

One frustration of PHP is this. A "pure" PHP file with no HTML looks like this:

PHP is an HTML templating system, by default everything in PHP is just stuff waiting to be sent untouched to the browser. So in essence, PHP web apps aren't really web apps... they are very complicated templates all talking to each other.

So you'll note in the above example, even a pure PHP file must begin with <?php, and signal that we're actually doing code. In fact, to prevent any unintended whitespace being sent to the browser, a PHP only file will by convention omit the closing ?>.

So my shiny new Django framework is a breath of fresh air. There's no <?py tags, it's all just python. And template files are explicitly template files parsed by a template engine built on top of a scripting language.

I can easily see PHP evolving into something similar one day. Where all PHP files are just PHP by default. It would be more in line with the way MVC development is going.

python: relative paths

2008-01-27T00:00:00-08:00

So I started yesterday with Django, and I decided I didn't want to futz with creating another mysql database that I'd need to manage, etc. Instead I'll just use sqlite.

I wanted to keep my sqlite database within my project regardless of where I might move my project later. So I did this:

I confused a lot of people on IRC, but it's really quite easy:

__file__ is the filename of the current script, very similar to PHP's __FILE__
os.path.abspath calculates the absolute path, hence the absolute path of the current file
os.path.join does all the nasty business of joining paths together and figuring out what type of slashes are needed, etc.
'data/db.sqlite' is a string

So really all we were doing is creating a relative path, but setting it absolutely.

python: it begins

2008-01-26T00:00:00-08:00

[tags]php, python, symfony, frameworks, programming[/tags]

The problem I have with python, is I really like what I can do with it, and it makes me hate that I have to use PHP.

PHP works. I've used it for over 10 years without problem. My web development methodology was this: I want to do x, but I need to first learn y. Well one day y became PHP/FI and then PHP 3 and then PHP 4 and then PHP5. It was a natural progression.

I came from a (what was at the time) traditional C/C++/Java background and decided that PHP while amateurish was still a better solution for web development. A few years later I was proven right by companies like Yahoo! and Facebook.

I did have my issues, and I worked through them, symfony and other frameworks like it, have made PHP development so much better, but eventually you get let in on little secrets. Secrets like other scripting languages.

In the last year I dabbled with ruby, but decided I didn't want to look at rails. A few people pointed me in the direction of python and Django as an alternative to PHP alternatives. So I've spent time learning python.

Yesterday and today I gave myself a real project. I started hacking away at del.icio.us for a Winter Hack Day at Yahoo! I would write a few lines of code, and then realize that I can write it a little better a little cleaner using some neat little python constructs. It's enjoyable in the way that PHP is not.

It might just be a phase, but I sincerely doubt it. A lot of people I know who use python love it. Many have been using it for years (even while I secretly thought, "python? really?" Well, I'm starting to think they've been onto something. We'll see where I end up.

Fixing broken PATH_INFO

2008-01-23T00:00:00-08:00

[tags]php, cgi, path_info, nginx, symfony[/tags]

symfony and other applications rely on the server's PATH_INFO being set properly. According to NCSA:

The extra path information, as given by the client. In other words, scripts can be accessed by their virtual pathname, followed by extra information at the end of this path. The extra information is sent as `PATH_INFO`. This information *should be decoded by the server* if it comes from a URL before it is passed to the CGI script.

Unfortunately, I use a nonstandard server that doesn't natively support CGI, so everything sent to the FastCGI server is done so via parameters that are usually obtained from the HTTP request, but I can't figure out how to do a urldecode in my configuration.

So to workaround this I used the auto_prepend_file directive in php.ini. With OP code caching this shouldn't hurt too much:

auto_prepend_file = /var/www/pathinfofix.php

I then added the following script:

<?php 
$_SERVER['PATH_INFO'] = urldecode($_SERVER['ORIG_PATH_INFO']);

Voila, the PATH_INFO is in a format that symfony (and any other PHP script that depends on PATH_INFO) needs.

doctine and getState()

2007-07-10T00:00:00-07:00

[tags]doctrine, php, symfony, sfDoctrine, database,errors[/tags]

I tend to have models with a field called state. Doctrine offers a few ways of getting to the state field:

$obj->get('state');
$obj['state'];

$obj->getState() however conflicts with Doctrine_Record::getState() from which all objects inherit. Use one of the above alternatives.

Using sfDoctrine to match allowed email domains

2007-07-09T00:00:00-07:00

[tags]php, propel, doctrine, validators, symfony, optiopt, startup[/tags]

I'm a co-founder at an online finance web site and I'm in charge with building out the site. Our rollout strategy is to let a a handful of companies at a time, so we're limiting registration based on your company's email address.

I decided to follow the bandwagon and use PHP Doctrine. We'll define two objects, Company and CompanyDomain:

Company:
  tableName: company
  columns:
    name: {type: string(100)}
    created_at: timestamp
    updated_at: timestamp

CompanyDomain:
  tableName: company_domain
  columns:
    company_id:
      foreignClass: Company
      cascadeDelete: true
    pattern: {type: string(100)}

Where a CompanyDomain represents a domain name for a company. E.g. Motorola might have both motorola.com and mot.com.

We can validate a signup form and see if we've got an email address from a domain we recognize. I like using the validationXXX methods in an action class for specific validation. I made one called validateSignup:

public function validateSignup()
{
    if ($this->isPost())
    {
        $email = $this->getRequestParameter('company_email');
        if (!$company = CompanyTable::match($email))
        {
            $this->getRequest()->setError('company_email', 'Your work email doesn\'t appear to belong to any of the registered companies.');
            return false;
        }
        return true;
    }
}

In our CompanyTable we create a static function CompanyTable::match:

static public function match($email)
{
    $company = Doctrine_Query::create()->from('Company c, c.CompanyDomains d')->where('? LIKE CONCAT(\'%\', pattern)', $email)->execute()->getFirst();

    return $company;
}

Note that we transparently do a join in the ->from() statement. If there's a match we get a company object (which we can associate with the new user) otherwise we get a null object.

Enjoy.

Boosting terms in Zend Search Lucene

2007-05-29T00:00:00-07:00

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl[/tags]

Boosting terms — some fields are better than others

Lucene supports boosting or weighting terms. For example, if I search for members of a web site, and I type in Dash, I want people with the name Dash to take precendence over somebody who has a hobby of running the 50-yard Dash.

If we look at our generateZSLDocument() method we defined we just need to adjust a few lines:

Should be turned into:

This is pretty straight forward way to add weight (1.5 times the weight of a normal term) and you can customize it to the needs of your site.

Finding things using Zend Search Lucene in symfony

2007-05-23T00:00:00-07:00

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl[/tags]

This is part of an on going series about the Zend Search Lucene libraries and symfony. We'll pretty everything up when we're done =)

We now know how to manipulate the index via our model classes. But let's actually do something useful with our search engine... let's search!

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl[/tags]

At the time of this writing we're dealing with Propel which uses Peer classes which are meant for dealing with multiple objects¹. This is the perfect place for a ::search() method. In otherwords, UserPeer::search('dave'); should query Lucene for users matching "dave". Let's make that happen:

What we're doing is retrieving our Lucene index. Somewhere between tutorials we wrote this Peer function to handle that:

If our index is missing we'll conveniently create it on the fly. We then use the Zend Search Lucene API to retrieve the matching hits in this index and then use some Propel trickery to retrieve by an array of primary keys.

It's now simple to use ::search() functions in the same manner as you use ::doSelect().

At this point you should be able to create a basic symfony app that can utilize a Lucene index.

The examples refer to using Propel, but it's trivial to adapt this to sfDoctrine ↩

Creating, Updating, Deleting documents in a Lucene Index with symfony

2007-04-24T00:00:00-07:00

Previously we covered an all-at-once approach to indexing objects in your symfony app. But for some reason, people find the need to allow users to sign up, or change their email addresses and then all of a sudden our wonderful Lucene index is out of date.

Here lies the strength of using Zend Search Lucene in your app, you can now get the flexibility of interacting with a Lucene index, no matter how it was created and add, update and delete documents to it.

The last thing you want to do is have a cron job in charge of making sure your index is always up to date by reindexing regularly. This is an inelegant and inefficient process.

A smarter method would be to trigger an update of the index each time you update your database. Luckily the ORM layer allows us to do this using objects (in our case Propel objects).

If we look at our user example from before, we did set ourselves up to easily do this using our User::generateZSLDocument() function, which did most of the heavy lifting.

We can make a few small changes to the User class:

We have an attribute called $reindex. When it is false we don't need to worry about the index. When something significant changes, like an update to your name or email address, then we set $reindex to true. Then when we save with an overridden save method:

Now we've got the exact same data that we created during our original indexing. This handled creating and updating object, but we miss updating the index when deleting objects.

Luckily we already made a function User::removeFromIndex() to remove any related documents from the index, so our delete function can be pretty simple:

The Lucene Search Index and symfony

2007-04-23T00:00:00-07:00

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl, index[/tags]

This article is meant to followup sfZendPlugin where we learn a newer way of obtaining the Zend Framework.

In this tutorial we're going to delve into the Lucene index. Zend Search Lucene relies on building a Lucene index. This is a directory that contains files that can be indexed and queried by Lucene or other ports. In our example we'll be creating a search for user profiles.

We'll want to store in our app.yml the precise location of this index file so we can refer to it in our app¹.

Here's an example:

all:
  search:
    user_index: /tmp/myapp.user.lucene.index

Now when we need to refer to the index we can do sfConfig::get('app_search_user_index').

Index Something

Let's try a user search where we can find a user by their name or email address. It's fairly simple to accomplish, and hardly requires the use of ZSL, but by using ZSL we can easily extend it to do a full-text search of a user's profile or any other textual data.

Each "thing" stored in the index is a Lucene "document". Each document then consists of several "fields" (Zend_Search_Lucene_Field objects). In our example, each document will be an individual user and the fields will be relevant attributes of the user (username, first name, last name, email, the text of their profile).

Initially we'll want to populate our index. We may also want to regularly reindex all the users at once to optimize the search performance. Since reindexing involves multiple users it would make sense to have a static reindex method in our UserPeer class².

Very simply, we're creating a new index, getting all the users, adding a document to the index and then committing the index (to disk). You might have noticed that there's a strange function, User::generateZSLDocument(). This function contains all the magic. In order to not repeat ourselves we keep the internals of making a document for the Lucene index in the User class itself. Let's look at it:

We're really just dumping the relevant search terms into this document. The beauty of keeping this code internalized in the User class is we can reuse it later if we need to index a single User at a time.

A couple things to note. Zend_Search_Lucene_Field::Keyword allows us to store data that we can lookup later. We store the User::id in a field called uid since id is a reserved word for the index and we can't access it from Zend Search Lucene.

In a batch script or a reindex action we can now just call UserPeer::reindex() and have a working search index for our users.

Storing things in app.yml is great for indexes that don't need to be searched in multiple applications. ↩
Since we're using a Lucene index, which has an open documented structure, we aren't limited to just using Zend Search Lucene or Apache Lucene (java). We can mix and match and read and write to the same index file. For very large indexes (65,000+ documents), I rewrote a Java application to index all the documents at once as PHP would time out during such a task. ↩

MinneBar

2007-04-21T00:00:00-07:00

[tags]Minnebar, symfony, barcamp, frameworks, php, php5[/tags]

I gave a quick overview of the symfony framework for PHP5 at MinneBar. If you have questions or comments about symfony or my talk, feel free to leave them here.

sfZendPlugin

2007-04-10T00:00:00-07:00

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl, plugins[/tags]

I originally intended to rewrite my Zend Search Lucene tutorial, but Peter Van Garderen covered the bulk of what's changed and I was too busy developing search functionality for lyro.com (not to mention finding inconsistencies with the Zend Search Lucene port and Lucene) to finish the tutorial. So I broke it up into smaller pieces.

I packaged Zend Framework into a symfony plugin. symfony is easily extended using plugins.

You can obtain this from subversion with the following command (from your /plugins directory):

svn export http://svn.symfony-project.com/plugins/sfZendPlugin

symfony has a Zend Framework Bridge which let's us autoload the framework by adding the following to settings.yml:

.settings:
  zend_lib_dir:   %SF_ROOT_DIR%/plugins/sfZendPlugin/lib
  autoloading_functions:
    - [sfZendFrameworkBridge, autoload]

First we define sf_zend_lib_dir to be in our plugin's lib directory. Then we autoload the bridge framework.

After setting this up, all the Zend classes will be available and auto-loaded from elsewhere in your code.

Caching REST with sfFunctionCache

2007-04-07T00:00:00-07:00

[tags]geocoding, caching, REST, symfony, Cache_Lite, php, cache, sfFunctionCache[/tags]

For reviewsby.us we do a lot of geocoding. To facilitate we use Yahoo! Geocoding API. This helps us normalize data, obtain latitude and longitude, interpret location specific searches.

These REST queries happen a lot and will continue to happen, but this data that Yahoo! provides is fairly static. We're basically querying a database of sorts. So it makes sense that we should cache this data.

We'll demonstrate how to cache these queries using symfony's sfFunctionCache class.

I wrote a wrapper (I'll release it as a plugin if requested) for the Geocoding API, the bulk of the work (the REST call) occurs in a function called doQueryGIS:

The call to this function is always wrapped with queryGIS:

This wrapper creates a sfFunctionCache objet and calls the function and caches it for subsequent queries.

What this means is once Yahoo! teaches reviewsby.us that India is located at (25.42°, 77.830002°) and that the precision is 'country' we remember it in the future.

These features will be incorporated into future versions of reviewsby.us.

PHP double versus single quotes

2007-03-03T00:00:00-08:00

[tags]best practices, php[/tags]

If you're familiar with PHP, the difference between double and single quotes is obvious. Double quotes evaluate variables and control characters (e.g. \n or \r), whereas single quotes do not.

I've been indoctrinated with the "use single quotes whenever possible" methodology, but I never really put it to the test. Is it really worth it for me to go back and look at old code that uses double quotes and change them? Like all best practices, the answer is "maybe."

So I wrote a simple test harness (there's a lot of room for error with these, but I tried my best).

I tried different permutations of which order to run f1(), f2() and f3() they seemed to give similar results no matter which order they were run.

My results were:

Time 1: 2.898087978363
Time 2: 3.5480048656464
Time 3: 2.8503499031067

My interpretation is that quotes versus double quotes isn't as big of a deal as concatenation versus putting variables within double quotes. My guess is that PHP 5.2.0 (which is what I used for the tests) is smarter with strings than older versions.

So really if you look at the test harness there isn't any discernible differences until you hit 2 million iterations and even then nobody gets fired over 0.6 seconds of performance. Chances are it doesn't matter too much, but over time you can write enough code in the right spots or shared in the right open source projects and that few seconds will snowball. After all, I'm not so much concerned about how optimized a script is that I write, but rather how optimized is a script that everyone ends up using. But rather than do mental calculations about whether or not to optimize something, let's just assume that everything should be as optimal as we can stand to write it.

Update: 6 March 2006, I updated the test harness to reflect my intended tests. Update: 7 March 2006, I updated the results to be more clear.

Parsing a list of Key:Value pairs

2007-02-24T00:00:00-08:00

[tags]best practices,php,openID[/tags] I'm working on implementing openID for reviewsby.us and for use in symfony apps. One thing I was having trouble with was parsing key value pairs, which is one of the requirements to reading responses. It's a fairly easy task, but PHP offers so many ways to do this.

openID calls for the following Key-Value format:

Lines of:

some_key:some value

There MUST NOT be a space before or after the colon.

Newline characters MUST be Unix-style, just ASCII character 10 ("\n").

Newlines MUST BE at end of each line as well as between lines.

MIME type is unspecified, but text/plain is RECOMMENDED.

Character encoding MUST BE UTF-8.

So here is my attempt at parsing something like this as efficiently and error free as possible:

I wrote this function as I was writing the post... and it came out way faster than my previous implementations using strtok or a combination of explode and trim. Which is good to hear since I do use preg_ functions quite a bit in PHP and they definitely have their place.

I'm curious if people have found a faster way of parsing through a string of Key-Value pairs. I'll run it in a test harness and stand corrected if someone comes through with something faster ;).

Coming soon to reviewsby.us

2006-10-02T00:00:00-07:00

In August I took a break from reviewsby.us only to be plagued by spam. In September, I relinquished portions of the project planning to my wife. We haven't released anything publicly, yet, but there's a lot in development.

I updated the development framework to symfony 1.0 alpha and took care of a whole slew of bugs.
Katie and I came up with a wireframe that details some of the upcoming changes.
I upgraded the user logic to take advantage of sfGuardUser, a user management plugin for symfony.

I'm in progress of writing location specific searches. I'm slow to implement. It seems that this month is far busier than I'd like, and I can rarely get in a block of enough time to just crank this out. The problem with geographic-specific searches is mySQL supports those types of queries, but it's not as easy as I'd like. Zend Search Lucene with some support from PHP, however, may yield some promising results. As always, I'll share my findings in a forthcoming tutorial.

Anyway, no visible updates on the actual site, since I didn't want to put alpha software on the live site. I'm sure by next month symfony will be ready.

Using Zend Search Lucene in a symfony app

2006-08-25T00:00:00-07:00

[tags]zend, search, lucene, zend search lucene, zsl, symfony,php[/tags]

If you're like me you've probably followed the Askeet tutorial on Search in order to create a decent search engine for your web app. It's fairly straight forward, but they hinted that when Zend Search Lucene (ZSL) is released, that might be the way to go. Well we are in luck, ZSL is available, so let's just dive right in.

If you aren't using symfony have a look at this article from the Zend Developer Zone. It covers just enough to get you started. If you are using symfony, just follow along and we'll get you where you need to go.

Obtaining Zend Search Lucene

First download the Zend Framework (ZF). The Zend Framework is supposed to be fairly "easy" in terms of installation. So let's put that to the test. Open your ZF archive. Copy Zend.php and Zend/Search to your symfony project's library folder:

cp Zend.php $SF_PROJECT/lib              
mkdir $SF_PROJECT/lib/Zend
cp -r Zend/Search $SF_PROJECT/lib/Zend
cp Zend/Exception.php $SF_PROJECT/lib/Zend                 
chmod -R a+r $SF_PROJECT/lib/Zend*

Index Something

We'll deviate slightly from food themed tutorials and do something generic. Let's try a user search where we can find a user by their name or email address. It's fairly simple to accomplish, and hardly requires the use of ZSL, but by using ZSL we can easily extend it to do a full-text search of a user's profile or any other textual data.

Each "thing" stored in the index is a "document" in ZSL, specifically a Zend_Search_Lucene_Document. Each document then consists of several "fields" (Zend_Search_Lucene_Field objects). In our example, our document will be an individual user and the fields will be relevant attributes of the user (username, first name, last name, email, the text of their profile).

We're going to write a general re-indexing tool. Something that will index all users.

In our userActions class let's add the following action:

public function executeReindex()
    {
        require_once 'Zend/Search/Lucene.php';
        $index = new Zend_Search_Lucene(sfConfig::get('app_search_user_index_file'),true);
        
        $users = UserPeer::doSelect(new Criteria());
        foreach ($users AS $user)
        {
            $doc = new Zend_Search_Lucene_Document();
            $doc->addField(Zend_Search_Lucene_Field::Keyword('id', $user->getId()));
            $doc->addField(Zend_Search_Lucene_Field::Keyword('username', $user->getUsername()));
            $doc->addField(Zend_Search_Lucene_Field::Keyword('email', $user->getEmail()));
            $doc->addField(Zend_Search_Lucene_Field::Text('firstname', $user->getFirstname()));
            $doc->addField(Zend_Search_Lucene_Field::Text('lastname', $user->getLastname()));
            $doc->addField(Zend_Search_Lucene_Field::Unstored('contents', "{$user->getEmail()} {$user->getFirstname()} {$user->getLastname()} {$user->getUsername()}"));
            $index->addDocument($doc);
        }
        
        $index->commit();
    }

The code should be fairly easy to follow. First of all we're requiring the necessary libraries for Lucene. The next line we are creating the index:

app_search_user_index_file is a symfony configuration that you define in your app.yml. It defines which file you want to use for your index. /tmp/lucene.user.index works for our purposes. The second parameter tells Lucene we are creating a new index.

We then loop through all the users and for each user create a document. For all the search relevant attributes that a user might have we add a field into the document. Note the last field:

By default search is made for the "contents" field. So in this example we want people to be able to type in someone's name, email, username without having to specify what field we're searching for.

Find those users

Finding the user's is equally as straight-forward. We make a new action called search:

If we have a query, open the ZSL index (note that we only have one parameter here). Run the find method to find our query and store it to the $hits array. Note that our query was cleaned with strtolower, since ZSL is case sensitive.

The template takes care of the rest:

Fairly simple... but it could use some cleaning up (enjoy).

What about new users?

Regularly reindexing might be nice in terms of having an optimized search index, but its lousy if you want to be able to search the network immediately when new people join on. So why not automatically re-index each user every time they are created or everytime one of their indexed components is summoned?

This should be fairly simple by adding to the User class:

We have an attribute called $reindex. When it is false we don't need to worry about indexes. When something significant changes, like an update to your name or email address, then we set $reindex to true. Then when we save:

We're calling a new function called generateZSLDocument. It might look familiar:

Now, whenever a user is updated, so is our index. Additionally we can modify our reindex action:

That's a lot easier to deal with.

...and beyond

Hope this article helps some of you jumpstart your symfony apps. Really cool, easy to implement search is here. We no longer have to stick with shoddy solutions like HT://Dig or spend time rolling our own full text search, as the symfony team diligently showed us we could. But there is a lot more ground to cover. Including optimization techniques and best practices.

Let me know what you think, and if you use this in any of your apps.

Dave Dash

Versioning Django Models

How I think of versions

The Django implementation of RestaurantVersion

The Django implementation of Restaurant

Proxy attributes to the corresponding RestaurantVersion attributes

Generate a new RestaurantVersion on demand

Conclusion and Drawbacks

PHPs strengths: array_count_values

Optimization heuristics

The micro-caching heuristic

py vs php: stemming

the magic of django: get_callable

Getting to the point

Why Django Templating is awesome and why I get smarty again

Templating

python: relative paths

python: it begins

Fixing broken PATH_INFO

doctine and getState()

Using sfDoctrine to match allowed email domains

Boosting terms in Zend Search Lucene

Boosting terms — some fields are better than others

Finding things using Zend Search Lucene in symfony

Creating, Updating, Deleting documents in a Lucene Index with symfony

The Lucene Search Index and symfony

Index Something

MinneBar

sfZendPlugin

Caching REST with sfFunctionCache

PHP double versus single quotes

Parsing a list of Key:Value pairs

Coming soon to reviewsby.us

Using Zend Search Lucene in a symfony app

Obtaining Zend Search Lucene

Index Something

Find those users

What about new users?

...and beyond

The Django implementation of `RestaurantVersion`

The Django implementation of `Restaurant`

Proxy attributes to the corresponding `RestaurantVersion` attributes

Generate a new `RestaurantVersion` on demand