Dave Dash

symfonyCamp

2007-09-11T00:00:00-07:00

[tags]symfony, symfonyCamp, sensio, dop, zend search lucene, zsl[/tags]

Well Katie and I are back from symfonyCamp and it was great. I opted to socialize on the business day, except to hear Fabien Potencier's brief overview of what's to come in symfony 1.1/2.0.

I personally know only a handful of adept symfony developers, so going to camp it was nice to see 50 people or so who knew symfony to varying degrees. I am a fan of the small successes, such as Fabian Lange's historical reenactment site which paid the way for two developers to attend the conference.

One of the most interesting talks to hear was Fabien's overview of symfony 2.0. It was a 12-step process from going to vanilla PHP to building a strong framework in about 200 lines. A leaner more robust symfony sounds very appealing, it's also what appeals to me about being a developer. Every developer builds off of existing technologies and is able to create something great. I frequently have to dig into the symfony core code to see how things "really" work and everything is simple, easy to follow building blocks. Effectively giving us a nice framework. symfony 2.0 seems to be a leaner more flexible framework.

Later I spoke on Zend Search Lucene as well as Ajax. Both are somewhat difficult to speak on for 45 minutes. Zend Search Lucene really only takes 15 minutes to explain, and maybe just as long to implement. Sure it can be tweaked quite a bit, but it's straightforward - that's the point.

Ajax on the other hand is hard to explain in terms of symfony. Sure there's a helper layer, but the Javascript layer is very independent of the PHP layer. Anything can support Ajax. So I tried to cover not just the standard helpers, but show a few demos and how easy it is with the helper system. Unfortunately I don't really code this way, I try to use UJS and the jQuery plugins with new work, but that would move the talk to a more advanced topic.

The next day we ended up cleaning up the symfony project. We split into teams to take care of some house-work. Some people cleaned up the wiki. Some cleaned up tickets, some wrote new modules for the site. Our team worked on plugins and it went rather well. We closed a number of tickets, created a good deal of patches (which I have yet to apply), but overall the plugins are all a bit better.

Overall everything was great. Dutch Open Projects was great, especially Stefan for arranging so much, Guido for making sure everyone was comfortable and Floris for the great food.

You can also read Fabien's overview of the camp.

Symfony Camp: Ajax and Zend, what would you like to know?

2007-08-16T00:00:00-07:00

[tags]symfonyCamp, symfony, netherlands, ajax, zend search lucene, zsl, jquery[/tags]

I've been asked to speak at SymfonyCamp (symfony['camp']) next month (you should all go if you can) and I thought I'd present as well as I could on Ajax and the Zend Framework Bridge (including Zend Search Lucene).

If you're attending the camp and/or would like to hear about these topics please let me know any specific questions you might have about "symfony and Ajax" and "symfony and Zend" and I'll try to address them in my presentations.

If you are unable to go fear not, I'll try to post my notes on this site.

Boosting terms in Zend Search Lucene

2007-05-29T00:00:00-07:00

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl[/tags]

Boosting terms — some fields are better than others

Lucene supports boosting or weighting terms. For example, if I search for members of a web site, and I type in Dash, I want people with the name Dash to take precendence over somebody who has a hobby of running the 50-yard Dash.

If we look at our generateZSLDocument() method we defined we just need to adjust a few lines:

Should be turned into:

This is pretty straight forward way to add weight (1.5 times the weight of a normal term) and you can customize it to the needs of your site.

Finding things using Zend Search Lucene in symfony

2007-05-23T00:00:00-07:00

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl[/tags]

This is part of an on going series about the Zend Search Lucene libraries and symfony. We'll pretty everything up when we're done =)

We now know how to manipulate the index via our model classes. But let's actually do something useful with our search engine... let's search!

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl[/tags]

At the time of this writing we're dealing with Propel which uses Peer classes which are meant for dealing with multiple objects¹. This is the perfect place for a ::search() method. In otherwords, UserPeer::search('dave'); should query Lucene for users matching "dave". Let's make that happen:

What we're doing is retrieving our Lucene index. Somewhere between tutorials we wrote this Peer function to handle that:

If our index is missing we'll conveniently create it on the fly. We then use the Zend Search Lucene API to retrieve the matching hits in this index and then use some Propel trickery to retrieve by an array of primary keys.

It's now simple to use ::search() functions in the same manner as you use ::doSelect().

At this point you should be able to create a basic symfony app that can utilize a Lucene index.

The examples refer to using Propel, but it's trivial to adapt this to sfDoctrine ↩

Creating, Updating, Deleting documents in a Lucene Index with symfony

2007-04-24T00:00:00-07:00

Previously we covered an all-at-once approach to indexing objects in your symfony app. But for some reason, people find the need to allow users to sign up, or change their email addresses and then all of a sudden our wonderful Lucene index is out of date.

Here lies the strength of using Zend Search Lucene in your app, you can now get the flexibility of interacting with a Lucene index, no matter how it was created and add, update and delete documents to it.

The last thing you want to do is have a cron job in charge of making sure your index is always up to date by reindexing regularly. This is an inelegant and inefficient process.

A smarter method would be to trigger an update of the index each time you update your database. Luckily the ORM layer allows us to do this using objects (in our case Propel objects).

If we look at our user example from before, we did set ourselves up to easily do this using our User::generateZSLDocument() function, which did most of the heavy lifting.

We can make a few small changes to the User class:

We have an attribute called $reindex. When it is false we don't need to worry about the index. When something significant changes, like an update to your name or email address, then we set $reindex to true. Then when we save with an overridden save method:

Now we've got the exact same data that we created during our original indexing. This handled creating and updating object, but we miss updating the index when deleting objects.

Luckily we already made a function User::removeFromIndex() to remove any related documents from the index, so our delete function can be pretty simple:

The Lucene Search Index and symfony

2007-04-23T00:00:00-07:00

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl, index[/tags]

This article is meant to followup sfZendPlugin where we learn a newer way of obtaining the Zend Framework.

In this tutorial we're going to delve into the Lucene index. Zend Search Lucene relies on building a Lucene index. This is a directory that contains files that can be indexed and queried by Lucene or other ports. In our example we'll be creating a search for user profiles.

We'll want to store in our app.yml the precise location of this index file so we can refer to it in our app¹.

Here's an example:

all:
  search:
    user_index: /tmp/myapp.user.lucene.index

Now when we need to refer to the index we can do sfConfig::get('app_search_user_index').

Index Something

Let's try a user search where we can find a user by their name or email address. It's fairly simple to accomplish, and hardly requires the use of ZSL, but by using ZSL we can easily extend it to do a full-text search of a user's profile or any other textual data.

Each "thing" stored in the index is a Lucene "document". Each document then consists of several "fields" (Zend_Search_Lucene_Field objects). In our example, each document will be an individual user and the fields will be relevant attributes of the user (username, first name, last name, email, the text of their profile).

Initially we'll want to populate our index. We may also want to regularly reindex all the users at once to optimize the search performance. Since reindexing involves multiple users it would make sense to have a static reindex method in our UserPeer class².

Very simply, we're creating a new index, getting all the users, adding a document to the index and then committing the index (to disk). You might have noticed that there's a strange function, User::generateZSLDocument(). This function contains all the magic. In order to not repeat ourselves we keep the internals of making a document for the Lucene index in the User class itself. Let's look at it:

We're really just dumping the relevant search terms into this document. The beauty of keeping this code internalized in the User class is we can reuse it later if we need to index a single User at a time.

A couple things to note. Zend_Search_Lucene_Field::Keyword allows us to store data that we can lookup later. We store the User::id in a field called uid since id is a reserved word for the index and we can't access it from Zend Search Lucene.

In a batch script or a reindex action we can now just call UserPeer::reindex() and have a working search index for our users.

Storing things in app.yml is great for indexes that don't need to be searched in multiple applications. ↩
Since we're using a Lucene index, which has an open documented structure, we aren't limited to just using Zend Search Lucene or Apache Lucene (java). We can mix and match and read and write to the same index file. For very large indexes (65,000+ documents), I rewrote a Java application to index all the documents at once as PHP would time out during such a task. ↩

sfZendPlugin

2007-04-10T00:00:00-07:00

[tags]Zend, Zend Search Lucene, Search, Lucene, php, symfony, zsl, plugins[/tags]

I originally intended to rewrite my Zend Search Lucene tutorial, but Peter Van Garderen covered the bulk of what's changed and I was too busy developing search functionality for lyro.com (not to mention finding inconsistencies with the Zend Search Lucene port and Lucene) to finish the tutorial. So I broke it up into smaller pieces.

I packaged Zend Framework into a symfony plugin. symfony is easily extended using plugins.

You can obtain this from subversion with the following command (from your /plugins directory):

svn export http://svn.symfony-project.com/plugins/sfZendPlugin

symfony has a Zend Framework Bridge which let's us autoload the framework by adding the following to settings.yml:

.settings:
  zend_lib_dir:   %SF_ROOT_DIR%/plugins/sfZendPlugin/lib
  autoloading_functions:
    - [sfZendFrameworkBridge, autoload]

First we define sf_zend_lib_dir to be in our plugin's lib directory. Then we autoload the bridge framework.

After setting this up, all the Zend classes will be available and auto-loaded from elsewhere in your code.

Coming soon to reviewsby.us

2006-10-02T00:00:00-07:00

In August I took a break from reviewsby.us only to be plagued by spam. In September, I relinquished portions of the project planning to my wife. We haven't released anything publicly, yet, but there's a lot in development.

I updated the development framework to symfony 1.0 alpha and took care of a whole slew of bugs.
Katie and I came up with a wireframe that details some of the upcoming changes.
I upgraded the user logic to take advantage of sfGuardUser, a user management plugin for symfony.

I'm in progress of writing location specific searches. I'm slow to implement. It seems that this month is far busier than I'd like, and I can rarely get in a block of enough time to just crank this out. The problem with geographic-specific searches is mySQL supports those types of queries, but it's not as easy as I'd like. Zend Search Lucene with some support from PHP, however, may yield some promising results. As always, I'll share my findings in a forthcoming tutorial.

Anyway, no visible updates on the actual site, since I didn't want to put alpha software on the live site. I'm sure by next month symfony will be ready.

Using Zend Search Lucene in a symfony app

2006-08-25T00:00:00-07:00

[tags]zend, search, lucene, zend search lucene, zsl, symfony,php[/tags]

If you're like me you've probably followed the Askeet tutorial on Search in order to create a decent search engine for your web app. It's fairly straight forward, but they hinted that when Zend Search Lucene (ZSL) is released, that might be the way to go. Well we are in luck, ZSL is available, so let's just dive right in.

If you aren't using symfony have a look at this article from the Zend Developer Zone. It covers just enough to get you started. If you are using symfony, just follow along and we'll get you where you need to go.

Obtaining Zend Search Lucene

First download the Zend Framework (ZF). The Zend Framework is supposed to be fairly "easy" in terms of installation. So let's put that to the test. Open your ZF archive. Copy Zend.php and Zend/Search to your symfony project's library folder:

cp Zend.php $SF_PROJECT/lib              
mkdir $SF_PROJECT/lib/Zend
cp -r Zend/Search $SF_PROJECT/lib/Zend
cp Zend/Exception.php $SF_PROJECT/lib/Zend                 
chmod -R a+r $SF_PROJECT/lib/Zend*

Index Something

We'll deviate slightly from food themed tutorials and do something generic. Let's try a user search where we can find a user by their name or email address. It's fairly simple to accomplish, and hardly requires the use of ZSL, but by using ZSL we can easily extend it to do a full-text search of a user's profile or any other textual data.

Each "thing" stored in the index is a "document" in ZSL, specifically a Zend_Search_Lucene_Document. Each document then consists of several "fields" (Zend_Search_Lucene_Field objects). In our example, our document will be an individual user and the fields will be relevant attributes of the user (username, first name, last name, email, the text of their profile).

We're going to write a general re-indexing tool. Something that will index all users.

In our userActions class let's add the following action:

public function executeReindex()
    {
        require_once 'Zend/Search/Lucene.php';
        $index = new Zend_Search_Lucene(sfConfig::get('app_search_user_index_file'),true);
        
        $users = UserPeer::doSelect(new Criteria());
        foreach ($users AS $user)
        {
            $doc = new Zend_Search_Lucene_Document();
            $doc->addField(Zend_Search_Lucene_Field::Keyword('id', $user->getId()));
            $doc->addField(Zend_Search_Lucene_Field::Keyword('username', $user->getUsername()));
            $doc->addField(Zend_Search_Lucene_Field::Keyword('email', $user->getEmail()));
            $doc->addField(Zend_Search_Lucene_Field::Text('firstname', $user->getFirstname()));
            $doc->addField(Zend_Search_Lucene_Field::Text('lastname', $user->getLastname()));
            $doc->addField(Zend_Search_Lucene_Field::Unstored('contents', "{$user->getEmail()} {$user->getFirstname()} {$user->getLastname()} {$user->getUsername()}"));
            $index->addDocument($doc);
        }
        
        $index->commit();
    }

The code should be fairly easy to follow. First of all we're requiring the necessary libraries for Lucene. The next line we are creating the index:

app_search_user_index_file is a symfony configuration that you define in your app.yml. It defines which file you want to use for your index. /tmp/lucene.user.index works for our purposes. The second parameter tells Lucene we are creating a new index.

We then loop through all the users and for each user create a document. For all the search relevant attributes that a user might have we add a field into the document. Note the last field:

By default search is made for the "contents" field. So in this example we want people to be able to type in someone's name, email, username without having to specify what field we're searching for.

Find those users

Finding the user's is equally as straight-forward. We make a new action called search:

If we have a query, open the ZSL index (note that we only have one parameter here). Run the find method to find our query and store it to the $hits array. Note that our query was cleaned with strtolower, since ZSL is case sensitive.

The template takes care of the rest:

Fairly simple... but it could use some cleaning up (enjoy).

What about new users?

Regularly reindexing might be nice in terms of having an optimized search index, but its lousy if you want to be able to search the network immediately when new people join on. So why not automatically re-index each user every time they are created or everytime one of their indexed components is summoned?

This should be fairly simple by adding to the User class:

We have an attribute called $reindex. When it is false we don't need to worry about indexes. When something significant changes, like an update to your name or email address, then we set $reindex to true. Then when we save:

We're calling a new function called generateZSLDocument. It might look familiar:

Now, whenever a user is updated, so is our index. Additionally we can modify our reindex action:

That's a lot easier to deal with.

...and beyond

Hope this article helps some of you jumpstart your symfony apps. Really cool, easy to implement search is here. We no longer have to stick with shoddy solutions like HT://Dig or spend time rolling our own full text search, as the symfony team diligently showed us we could. But there is a lot more ground to cover. Including optimization techniques and best practices.

Let me know what you think, and if you use this in any of your apps.