Dave Dash

Naming things and a recursion

2012-01-05T00:00:00-08:00

Most Mozilla webdev projects have an awful project structure, and it's partially my fault. I'm attemtping to fix that, but I cringe every time someone creates a new playdoh (Mozilla's Django template) based project.

The typical python project

Your typical python project looks like this:

/my_project
    someotherstuff/
    docs/
    theactualthingicareabout/
    setup.py
    LICENSE

MOZtrosity

We didn't have a good guide when we first started writing Django projects, so we opted for something like this:

theactualthingicareabout/
    apps/
        foo/
        bar/
    __init__.py
    urls.py
    settings.py
    LICENSE

In otherwords, the Django Project, which is a python module, is immediately checked out. If you check this out to an invalid directory, e.g. you do something like:

git clone github.com/davedash/myawesomeproject.git will.not.work\!

Bad things will happen.

So?

To some people this seems like an easy thing to work-a-round, but when it takes three of my excellent coworkers a week to diagnose an issue, where this ended up being the root cause... well it becomes a higher priority issue.

So here's what happened this week, when we tried to deploy the new careers site to a VM hardware.

Our ops team sensibly checked out the project like so:

git clone https://github.com/mozilla/lumbergh.git careers

They did everything right. Sure, they were creative and chose careers over the default lumbergh, but they knew the shortcomings of our system and picked a name that would resolve as a valid python package.

Unfortunately we'd hit some recursion error anytime we tried to hit a URL. So we knew there was an issue with the URL resolver, but we couldn't figure it out.

Here's what the project layout looked like:

careers/ # I could be called anything, but they chose careers
    __init__.py
    apps/
        careers/  # I'm going to cause problems,
                  # but neither devs nor ops will suspect a thing! mwahaha
            __init__.py
            models.py
            urls.py
            views.py
    settings.py
    urls.py

other files.

We configure our apps/ directory to be part of our PYTHON_PATH so we can do things like from careers import views... you can probably see where this is going.

Here's the main urls.py 1:

...
urlpatterns = patterns('',
    (r'', include('careers.urls')),
)
...

The main urls.py includes careers.urls which if you look at the above project layout, resolves to two different python packages:

careers/urls.py
careers/apps/careers/urls.py

Python chose the first, and therefore urls.py kept calling upon itself.

So what did we learn?

Do better.

First of all, we need a better project layout. This will continue to cause problems for even the brightest developers.

Secondly, if you don't do this at least name apps carefully. Django's app model can be a bit much for non third party apps. Sometimes there's one app which spans the entire project, and it's tempting to call it the same name as the project (e.g. careers), but sometimes a lamer more generic name like common is better.

But really, the second point is moot if we just clean up.

Better querying for ElasticSearch

2011-05-17T00:00:00-07:00

I wrote about how to write filter queries using pyes. Unfortunately after using ElasticSearch in the Add-ons Builder, I realized that our code would become unwieldy and hard to read if we kept using straight up pyes.

I prefer to write APIs so that are natural and conform to how I think, not one that simply mirrors another system.

So rather than this:

    filters = [TermFilter("platform", "all"),
               TermFilter("product", "firefox"),
               TermFilter("version", "4.0")]
    filter = ANDFilter(filters)
    q = FilteredQuery(MatchAllQuery(), filter).search()
    q.facet.add_term_facet('type')
    results = es.search(q)

I made something simpler:

    from elasticutils import S
    results = (S(platform='all', product='firefox', version='4.0')
               .facet('type').get_results)

Here were the design thoughts:

I wanted something easy to remember, S for search.
I wanted smart defaults, by default S() matches all documents, unless you give it a query term.
I didn't want to write python that looked like Java, or JSON or even a dict.
I wanted to write something that felt like the Django-ORM
Ultimately I want code that I enjoy writing.

So here it is, I expect it to power Firefox Add-ons, the Add-ons Builder and Firefox Input shortly.

This is all part of ElasticUtils. Let me know if you are using it, and pull requests are welcome!

How we slug at Mozilla

2011-03-24T00:00:00-07:00

One problem we find with slug generators, is they do an awful job with unicode. For a string like this: Bän...g (bang) you get something like bng---g--bang- or at best bang-bang. But it's 2011, urls can have unicode... here's what we really want: bäng-bang.

In some cases transliteration might be acceptable. But if we look at Django's approach it fails at Russian. Here's a comparison with ours for the Russian phrase "Быстрее и лучше!" ("Faster and better!"):

>>> from django.template.defaultfilters import slugify as djslugify
>>> from slugify import slugify
>>> str = u'Быстрее и лучше!'
>>> print djslugify(str)

>>> print slugify(str)
быстрее-и-лучше

So as you can see, the built-in Django slugify could be disastrous. So take a look at ours. If you have some more test cases, please fork it.

Bulk load ElasticSearch using pyes

2011-02-25T00:00:00-08:00

When indexing a lot of data, you can save time by bulk loading data.

With pyes you can do the following:

from pyes import ES


es = ES()
es.index(data, 'my-index', 'my-type', 1)
es.index(data, 'my-index', 'my-type', 2)
es.index(data, 'my-index', 'my-type', 3)
es.index(data, 'my-index', 'my-type', 4)

This will make 4 independent network calls.

from pyes import ES


es = ES()
es.index(data, 'my-index', 'my-type', 1, bulk=True)
es.index(data, 'my-index', 'my-type', 2, bulk=True)
es.index(data, 'my-index', 'my-type', 3, bulk=True)
es.index(data, 'my-index', 'my-type', 4, bulk=True)
es.refresh()

Will do this in one call. This is handy for those "reindex all the items we can" weekends.

Installing ElasticSearch plugins

2011-02-24T00:00:00-08:00

I'm slowly trying to familiarize myself with ElasticSearch and the pyes python interface. ElasticSearch uses a lot of plugins, and while the plugin system is easy to use, it's not obvious where to find the plugins.

They are here.

If you want to install the attachments plugin, you can do:

bin/plugin install mapper-attachments

And voilà it's installed.

Pythonic string formatting in Javascript

2010-11-19T00:00:00-08:00

We do a lot of string manipulation on the Firefox Addons site. A lot of it has to do with localization so one thing that comes up is being able to format strings. Here's a little snippet to give yourself python like string formatting:

    /* Python(ish) string formatting:
     * >>> format('{0}', ['zzz'])
     * "zzz"
     * >>> format('{x}', {x: 1})
     * "1"
     */
    function format(s, args) {
        var re = /\{([^}]+)\}/g;
        return s.replace(re, function(_, match){ return args[match]; });
    }

The Python textcluster Package

2010-07-08T00:00:00-07:00

Earlier I wrote about finding the most common Firefox issues. I had wanted to automate that process and continually find these issues. Unfortunately I never had time to do this.

When they announced Firefox Input, I thought about doing this again... just with Firefox Input data but then I went on paternity leave and time kind of crept away. But I mentioned the idea this week and it piqued some interest.

So I found myself with a bit of time to work on it. The first stage was releasing a python library called textcluster.

textcluster takes the work I did earlier and makes it a bit more general purpose. The idea is I can do something like this:

docs = (
        'Every good boy does fine.',
        'Every good girl does well.',
        'Cats eat rats.',
        "Rats don't sleep.",
        )

c = Corpus()
for doc in docs:
    c.add(doc)

print c.cluster()

Which results in:

[
    (
        "Rats don't sleep.",
        {'Cats eat rats.': 0.21353467285253394}
    ),
    (
        'Every good girl does well.',
        {'Every good boy does fine.': 0.32030200927880093}
    )
]

The number is the "similarity" between the strings relative to the entire document corpus.

My next trick is to see if I can run this memory-intensive calculation over a data-set of 25,000 opinions submitted. If I can we can get some interesting data about what people think of the new Firefox beta.

λ^2: safely doing class based views in Django

2010-03-09T00:00:00-08:00

When I started rewriting the API for addons.mozilla.org, my views were mostly the same: get some data and render it as either JSON or XML. I also wanted all my API methods to take an api_version parameter, so I decided class based views would be best. This way my classes could just inherit from a base class.

To do this I had to implement a __call__ method. This works fine, except I wanted to store things into the class -- after all the whole point of my use of classes was to keep the code a bit more compact, and cleaner. So, why pass the api_version around everywhere? Unfortunately thread-safety comes to play, and you need a separate instance of your class for each request.

λ

Django's urlpatterns expects a callable object. So you can't give it an instance of AddonDetailView(). But you could give it a callable that creates an instance of AddonDetailView() and passes it *args and **kwargs. Luckily python has lambda functions. You can note how we solved that in our urlpatterns.

λ λ

But wrapping all your urls with lambda is tedious and remembering to pass *args and **kwargs is error prone.

So let's make a lambda function that returns... a lambda function that turns an instance of our class into a callable.

We can now return to coding and not think about thread safety.

λλλ

Delicious keeps you in the know

2009-08-04T00:00:00-07:00

My last task at Delicious was to build along with the amazing Vik Singh was to build a new feed of bookmarks that was heavily influenced by Twitter. It was one of the most interesting and enjoyable pieces of code that I worked on at Delicious.

Over two months since my final check-in, the code is now in production. It is mostly as intended, but is lacking an RSS or JSON feed (which I had already built). This is somewhat disappointing since I was hoping that Delicious would remain as open as it had been in the past.

The algorithm is fairly simple we take a look at what trending topics exist at any moment in time (via Google Trends and Twitter) and we combine it with a list of popular terms. We take the whole lot of these items and query search twitter and store an in-memory data table of tweets. We also take a snapshot of new URLs to the Delicious corpus (basically anything on Delicious recent with 1 save). We cluster the Delicious URLs and then find tweets that are similar to each of these clusters.

The code for this is similar to Vik's TweetNews - but I think the Delicious data is a nicer fit.

We optimized this quite a bit and built a very fast inverted-index and tweaked the code to run in about a minute. Like TweetNews the heart of this was built using Python. Python while being a dynamic language is quite amazing for manipulating and iteratiting over sets of data.

While building this tool, it became my way to feel pulse of what's going on. I could ditch a lot of my RSS feeds and rely solely on Delicious to be on the up and up. Unfortunately I can't subscribe to a feed for this. Either delicious has made a mistake and didn't launch their feeds at the same time as their web (entirely possible, since Delicious hasn't been updated for most of 2009) or they are deliberately taking a step backwards.

This step backwards is weird from the usability issue. Delicious has always been a tool that allowed for multiple types of consumers and a tool that appealed to developers thanks to its myriad of RSS and JSON feeds. I'm glad I didn't have to be on the losing side of that decision. Delicious relies heavily on Google Trends and Twitter Search. While there is no requirement for them to share the data they are mashing up, it would be the right thing to do.

Let me know what you think of the new feeds. I wish I could share a github link or something snazzy so you could play around with it, but this post should be a good starting point for other real-time data mashups.

Update: Read Vik's account of this on the Delicious Blog.

From Delicious to Mozilla

2009-05-26T00:00:00-07:00

Today I said my good-byes to Delicious.com and Yahoo! and tonight I went to the Addons Meetup @ Mozilla to get a sneak peak at what I'll be working on in less than two weeks.

I was thrilled. I had no idea how many people to expect, but the Mozilla living room was packed - and most people were there the whole time. Real developers with really cool addons giving feedback to addons.mozilla.org directly. No matter how many blog comments, forums answered, customer care emails I responded to at Delicious - nothing beats the real insight and instant feedback you get from meeting a group of users face to face.

My brain, because of Delicious is always in data mining and analysis mode so through each presentation and each question asked, my brain was churning through things that I could build to bring some level of utility to the community.

I'm also happy to be joining an organization where everything is open sourced and available for comment. So I'm hoping to post a lot more on some of the cool tricks I do at Mozilla.

Reading urlopen and probably any file-ish things in python fast

2009-05-21T00:00:00-07:00

So I've been churning away in my last few days in Delicious-land trying to optimize some python code.

I was doing essentially this:

and it was taking almost a minute... for a ~3000 line json feed. Being the n00b that I am, I started getting my python learn on... and thought... an open url connection seems like a bad idea... let's do this:

Zoom! Took less than a second... actually putting real code in their (e.g. simplejson.loads()) bumped it to 2 seconds. Hooray.

Python Generators

2009-04-08T00:00:00-07:00

Someone had mentioned "generators" in python to me, so I decided to figure out what it was... and I figured it out. I think a simple example would help explain it:

Basically you can iterate over this "generator" which is a special function that "yield"s more than one value if you keep pinging it.

Adding this to the toolkit in my brain. I'm thinking one potential use is taking a resultset from a datastore (e.g. mysql) and converting it into an object.

A stitch in Fabric saves time

2009-03-02T00:00:00-08:00

Each day as I grow as a developer I pick up better habits. One is automating anything that I'll have to do more than once.

Therefore, deploying important projects has evolved:

Edit files live on the site.
FTP files from my local machine onto the live site and do tweaking as needed.
Develop using a cross-platform capable framework and deploy live via rsync.
One step deployment.

I looked at a few methods for performing one-step deployment. Shell scripts seemed to basic, and things like Puppet seemed to large a scale. So I looked into Capistrano and Fabric, and settled on Fabric, because it was barebones and python.

I'm at a stage with my project where I need to test it on an external server, so rather than uploading it and winging it I actually thought about a few things.

The task

I was thinking the ideal situation is a single command line that I can type that will upload my files to the server. Furthermore it should upload first to a new directory and then symlink and restart the server to do the cutover. I can easily write another script to rollback.

The way I setup my Nginx and Apache servers was they would look for the symlinks: staging, test, production to serve up the respective environments (I'm using a single host for this example).

Directory Structure

I use a very particular directory structure:

$WWW_APPS/mysite.com/ - is my root app deployment directory.
- releases/ - each code push is stored in a unique directory
  - 1/
  - 2/
  - ...
  - 100/ - the directory name is the svn revision, but is a series of precise svn exports, not a checkout.
- staging - this symbolic link links to a specific revision directory that has been designated for staging purposes. This might be the last directory in releases/, e.g. $WWW_APPS/mysite.com/releases/100/
  - config/ - any config files go here (e.g. nginx or apache)
  - mysite.com/ - the actual Django project
  - scripts/ - any utility scripts, mostly database model changes
  - site-packages/ - any related libraries that I need, this isn't the best approach I need to investigate virtualenv
  - static/ - all my static assets go here
- staging.rollback - the former staging symlink is demoted to staging.rollback
$WWW/mysite.com/
- staging - This is where the static assets get served from according to nginx. It's a symbolic link to $WWW_APPS/staging/static. This link does not change during deployment.
- uploads_staging/ - This directory contains uploads (user data). It requires manual adjustment when there are data changes (deletes, adds, updates). For the most part it can stay unchanged from one deployment to the next.

Subversion

I'm currently using subversion to maintain my code. If you are starting a project anew, I suggest using git as it is designed with branching in mind. The script below should be adaptable for git instead of svn.

That being said, when I deploy code, I use the revision number as the directory name for that deployment (e.g. $WWW_APPS/mysite.com/releases/100/ for r100) .

This forces me to commit my changes and not deploy code that is not checked in.

Other viable alternatives would be time-stamped directories. Anything that is unique between deployments will be sufficient.

The `fabfile.py`

Here's the fabfile.py I constructed. It's by no means final

def staging():
    "Pushes current code to staging, hups Apache"
    # get the build number    
    local('svn up mysite.com')
    
    config.svn_version   = svn_get_version()
    
    if not config.svn_version:
        abort()
    
    config.static_path   = '/var/www/static.mysite.com'
    config.svn_path      = 'http://svn.mysite.com/trunk'
    config.svn_export    = 'svn export -q -r %(svn_version)s'
    
    run('mkdir %(path)s', fail='abort')
    
    # svn export mysite.com to path 
    run('%(svn_export)s %(svn_path)s/mysite.com %(path)s/mysite.com', fail='abort')
    
    # svn export site-packages to site-packages
    run('%(svn_export)s %(svn_path)s/site-packages %(path)s/site-packages', fail='abort')
    
    # svn export mysite.com to path 
    run('%(svn_export)s %(svn_path)s/scripts %(path)s/scripts', fail='warn')
    
    # svn export configs
    run('%(svn_export)s %(svn_path)s/config %(path)s/config', fail='abort')
    
    # export /var/www/static.mysite.com/releases/%(svn_version) 
    run('%(svn_export)s %(svn_path)s/static %(path)s/static', fail='abort')
    
    # symlink to images from /var/www/static.mysite.com/staging/images/menuitems/* new release dir
    run("rm -r %(path)s/static/images/menuitems", fail=abort)
    run("ln -s %(static_path)s/menuitems_staging %(path)s/static/images/menuitems", fail=abort)

# rotate "staging" symlinks
    run('rm %(releases_path)s/staging.rollback', fail='warn')
    run('mv %(releases_path)s/staging  %(releases_path)s/staging.rollback', fail='warn')

# staging sym to new destination
    run('ln -s %(path)s %(releases_path)s/staging', fail='abort')
    
    # server is hup'd
    invoke(hup)

def rm_cur_rev():
    config.svn_version   = svn_get_version()
    run('rm -rf %(path)s', fail='abort')

def hup():
    sudo('/etc/init.d/apache2 restart')
    sudo('/etc/init.d/nginx restart')
    
    
def svn_get_version():
    from subprocess import Popen, PIPE
    output = Popen(["svn", "info", "mysite.com"], stdout=PIPE).communicate()[0]
    return output.partition('Revision: ')[2].partition('\n')[0]

config.fab_hosts = ['mysite.com']
config.fab_user = 'builder'
config.releases_path = '/var/www_apps/mysite.com'
config.path          = '%(releases_path)s/releases/$(svn_version)'

Final Thoughts

There's room for improvement, but this is a start. virtualenv looks like another avenue I might want to explore (especially when it comes time to upgrade Django, python, MysqlDB or PIL). This also doesn't take care of priming the server with necessary packages - again, something virtualenv could make easier for me.

However, this file was not too difficult to create. I sat down, thought about the ideal solution, implemented the steps and iterated on that until the script worked. I have pushed 19 revisions of code so far, mostly to test this process, but also to make necessary adjustments. I've probably saved countless hours of logging in and doing things manually.

How do you deploy your apps?

Versioning Django Models

2009-02-13T00:00:00-08:00

In symfony, versioning a model was not terribly difficult. I had my own specialized brute-force way of doing this.

In my experience it's been a lot easier to write python code than PHP code, so naturally I figured this would be an easy task. It was not. I suspect that it was not easy because I have a naive understanding of Django and my symfony knowledge was fairly well grounded.

I tried looking for a generic implementation of django model versioning, but failed. So I came up with a specific solution.

How I think of versions

For my app I chose to use a sparse versioning system. I'd have one interface for interacting with a model. For example, I have a Restaurant model and a RestaurantVersion model.

Restaurant would directly store immutable elements like name or rating or approved and it would encapsulate versioned elements like description by proxying to RestaurantVersion.

This means I only need to interface with one class and let the versioning happen behind the scenes.

The Django implementation of `RestaurantVersion`

RestaurantVersion held the data that I thought might suffer from corruption, and therefore require reversion. Therefore there's nothing surprising in this table:

The only thing of note is that I'm enforcing a 1:M relation between Restaurant and RestaurantVersion.

The Django implementation of `Restaurant`

Restaurant is where the magic happens. It needs to do a few things:

Do what a normal object does.
Proxy attributes to the corresponding attribute of a RestaurantVersion
Generate a new RestaurantVersion on demand.
(optional) Manage which version is "active".

I say "4" is optional because I don't do this myself. I built versioning into a few models because it was easier to do upfront than worry about it down the road.

Here's the code I use:

class Restaurant(models.Model):
    name           = models.CharField(max_length=765, blank=True)
    stripped_title = models.CharField(max_length=384, blank=True)
    approved       = models.IntegerField(null=True, blank=True)
    version        = models.ForeignKey(RestaurantVersion, related_name="the_restaurant")
    updated_at     = models.DateTimeField(auto_now=True)
    created_at     = models.DateTimeField(auto_now_add=True)
    new_version    = None
    
    def save(self, force_insert=False, force_update=False):
        if not self.stripped_title:
            self.stripped_title = slugify(self.name)
        
        super(Restaurant, self).save(force_insert, force_update)
        if self.new_version:
            self.new_version.restaurant = self
            self.new_version.save()
            self.version = self.new_version
            super(Restaurant, self).save(force_insert, force_update)        
        
    def __setattr__(self, name, value):
        if name == 'description':
            self.get_new_version().description = value
        
        elif name == 'url':
            self.get_new_version().url = value
        
        else:
            object.__setattr__(self, name, value)

def __getattr__(self, name):
        try:
            if name == 'description':
                return self.version.description

elif name == 'url':
                return self.version.url
        except RestaurantVersion.DoesNotExist:
            return ''
            
        models.Model.__getattribute__(self, name)
    
    def get_new_version(self):
        if self.new_version == None:
            try:
                rv    = self.version
                rv.id = None
            except RestaurantVersion.DoesNotExist:
                rv = RestaurantVersion()
            
            self.new_version = rv

return self.new_version
            
    class Meta:
        db_table     = u'restaurant'

It's not rocket science, but it wasn't necessarily easy either. Let's look at requirements 2 and 4 in a bit more detail.

Proxy attributes to the corresponding `RestaurantVersion` attributes

Proxying attributes is a way of masking the whole versioning infrastructure from the developer. We do this with __getattr__ and __setattr__ methods.

The __getattr__ determins if you are looking for description or url attributes of a Restaurant. If you are then it uses the current RestaurantVersion's attribute.

__setattr__ is very similar, except since we're changing a versionable attribute we're going to make a request to a method, get_new_version(), which we will detail later. It does what it says, though, it gets the "new" RestaurantVersion of the Restaurant.

Generate a new `RestaurantVersion` on demand

As you can see from the above code, I am "auto-versioning". This is done mostly via get_new_version(). We also have to do a few tricks to make sure the bidirectional relationship gets maintained on save.

get_new_version() either returns the current "new version", a brand new "new version" or a "copy" of the current version.

The "copy" is done simply by setting the id attribute of the new version to None. Django takes care of assigning it a new id on save, thus preserving the old version.

Note that when we create a brand new RestaurantVersion, we don't immediately set it's restaurant attribute. That's because we haven't saved the current restaurant yet. It's a "chicken and the egg" problem that gets solved in our save() method:

We first save the Restaurant, then we save a new version if there is one. If we save a new version we want to update the Restaurant a second time. This ensures that there's a 1:1 relationship between Restaurant and the current RestaurantVersion and it also establishes a 1:M relationship between Restaurant and all RestaurantVersions.

Conclusion and Drawbacks

While, I think that this setup works in my particular situation, I feel like it has some major flaws.

The code is quite messy and very specific to this model. My goal was to take care of this quickly, not necessarily in a reusable manner. I also was not familiar enough with the Django model to create some sort of extension.

This way of versioning does not allow for versioned attributes to be set upon instantiation of the model:

r=Restaurant(description="Great place")

won't work. You'll have to do:

r=Restaurant()
r.description="Great place"

I figured this was acceptable.

Lastly I'm not entirely happy with having to explicitly save the Restaurant model twice, but I think my bidirectional relation requires this.

All in all this works in my particular situation. I'm hoping that this can be simplified. I'm curious to hear about other versioning schemes.

PHPs strengths: array_count_values

2008-12-07T00:00:00-08:00

I always like to think of what different interpreted programming languages bring to the table.

Perl is great with string manipulation. I tend to use perl if I need to rewrite/reformat text.

Python makes it easy to write clean, organized code and has a lot of syntactical sugar like list comprehensions.

For PHP one of my coworkers suggested that maybe it's ease is what it brings to the table. I've accepted that wryly at first, but after some thought it's actually a real strength.

PHP has positioned itself to be easily adoptable. When I learned PHP/FI 2.0 it was because I had exhausted server-side includes and needed something better. PHP/FI was a great solution because it was easy to install (my ISP had it) and documented easy things that I could do. After all it was just some additional changes to my templates.

I never for once thought that I'd be paid at a company to code in it... but then the internet just got big.

The other way php positioned itself is the library of functions it has out of the box. Today I found myself looking for a python equivalent to array_count_values.

Its a function I wouldn't use every day, but there are rare instances where having a set of data aggregated by count would be very useful. That's exactly what this function does.

Even python's lists, sets and dictionaries, which can do with simple operations what PHP needs an army of array_ functions to do, didn't reveal anything useful. So I will use the following:

Python String Formatting

2008-12-06T00:00:00-08:00

Python 2.6 (and Py3K) introduce a new way to format strings. Perviously you did this:

"%s, Here is my string, %s!" % ('Salutations', 'Dave')

Now you do this:

"{0}, Here is my string, {1}!".format('Salutations','Dave')

At first glance this looks like syntactical-sugar. But now you can do some clever things (which you could have done before, but I think are more intuitive now). You can easily repeat strings:

def tag(name, value):
    return "<{0}>{1}</{0}>".format(name, value)

postgresql and python in osx

2008-08-10T00:00:00-07:00

I want to start dabbling with postgreSQL on OS X. After several SVN checkouts, binary packages, etc, I've realize the easiest path to success is just installing from fink unstable.

Running:

fink install psycopg2-py2.5

will get you everything you need, python, the pyscopg2 driver and even postgresql.

Unfortunately I had MacPython binary and a postgres binary which I couldn't easily install (and if anybody knows how to uninstall at least the former, I'm all ears). I decided to just merge the site-packages for both the fink installed python2.5 and the MacPython:

$ rm -rf /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/
$ ln -s /sw/lib/python2.5/site-packages /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/

Now if I installed more python packages via fink they will work in MacPython.

py vs php: stemming

2008-02-16T00:00:00-08:00

I've been porting some PHP to python during SuperHappyDevHouse and was amazed at how little code I needed to write since python makes list manipulation a breeze.

Today I was working on stemming (ala Porter Stemming algorithm). reviewsby.us uses stemming in the search engine to make queries:

Stemming turns hello everybody how are you guy's into a collection 'everybodi', 'gui', 'hello'. To produce this in php I do the following:

With some magic python:

The magic here is list mappings. Learning about them, they don't seem that great, but as soon as you start coding you stop using a lot of for loops.

I'm sure my PHP can be cleaned up and reduced as well, but its fun exploiting the magic of languages.

unix timestamp in python

2008-02-14T00:00:00-08:00

I spent far too much time learning that:

gives me the nice numeric representation of the unix timestamp.

the magic of django: get_callable

2008-02-08T00:00:00-08:00

symfony is an amazing framework simply because of the uphill battle it has with PHP. There's a lot of behind the scenes magic that ameliorates the frustrations one might have with PHPs lack of syntactical magic.

A lot of work is done to make sure certain things are autoloaded and that certain things behave the way you expect.

I hadn't had a chance to peek under Django's hood too much myself, but it appears that despite the inherent magic of python, there were some much needed additions. My savior for today was get_callable.

I have a very particular vision of interacting with OpenID. Basically I like to decouple as much of the OpenID interactions away from the inner workings of my web site. I do, however, want a hook where I can supply some of my own logic after OpenID has been verified.

It's roughly how I implimented the sfOpenIDPlugin. Luckily this time, I was able to rely on the OpenID 2.1 libraries from OpenIDEnabled.

I did ditch their example code, because I wanted a sexy package. The package isn't quite ready for primetime, but you can have a gander (and yes, I do welcome collaborators and am open to merging my work with the myriad of work that's out there).

Getting to the point

Anyway, get_callable is what allowed me to write my "hook" into my OpenID logic.

In my settings.py I defined:

OPENID_SUCCESS = 'myproject.myapp.views.openid_handler'

The way my OpenID library works, is it does it's business and when it verifies a url belongs to someone it then delivers he request to the OPENID_SUCCESS hook. Here's how:

if settings.OPENID_SUCCESS:
    view = get_callable(settings.OPENID_SUCCESS)
    return view(request, result)

get_callable works some python magic by splitting myproject.myapp.views.openid_handler into the module: myproject.myapp.views and the attribute openid_handler. It uses this to produce a callable object. In the example above I gave it parameters of request and result as they were required by my app.

Templating

2008-01-28T00:00:00-08:00

[tags]smarty, symfony, php, python, django[/tags]

I used to use the Smarty templating system quite heavily in my PHP apps. Then after switching over to symfony, I broke that habit.

PHP is a templating language for better or worse. Smarty was a way of saying that "well things could be easier", but that's against the grain of what PHP fundamentally is.

One frustration of PHP is this. A "pure" PHP file with no HTML looks like this:

PHP is an HTML templating system, by default everything in PHP is just stuff waiting to be sent untouched to the browser. So in essence, PHP web apps aren't really web apps... they are very complicated templates all talking to each other.

So you'll note in the above example, even a pure PHP file must begin with <?php, and signal that we're actually doing code. In fact, to prevent any unintended whitespace being sent to the browser, a PHP only file will by convention omit the closing ?>.

So my shiny new Django framework is a breath of fresh air. There's no <?py tags, it's all just python. And template files are explicitly template files parsed by a template engine built on top of a scripting language.

I can easily see PHP evolving into something similar one day. Where all PHP files are just PHP by default. It would be more in line with the way MVC development is going.

python: relative paths

2008-01-27T00:00:00-08:00

So I started yesterday with Django, and I decided I didn't want to futz with creating another mysql database that I'd need to manage, etc. Instead I'll just use sqlite.

I wanted to keep my sqlite database within my project regardless of where I might move my project later. So I did this:

I confused a lot of people on IRC, but it's really quite easy:

__file__ is the filename of the current script, very similar to PHP's __FILE__
os.path.abspath calculates the absolute path, hence the absolute path of the current file
os.path.join does all the nasty business of joining paths together and figuring out what type of slashes are needed, etc.
'data/db.sqlite' is a string

So really all we were doing is creating a relative path, but setting it absolutely.

python: it begins

2008-01-26T00:00:00-08:00

[tags]php, python, symfony, frameworks, programming[/tags]

The problem I have with python, is I really like what I can do with it, and it makes me hate that I have to use PHP.

PHP works. I've used it for over 10 years without problem. My web development methodology was this: I want to do x, but I need to first learn y. Well one day y became PHP/FI and then PHP 3 and then PHP 4 and then PHP5. It was a natural progression.

I came from a (what was at the time) traditional C/C++/Java background and decided that PHP while amateurish was still a better solution for web development. A few years later I was proven right by companies like Yahoo! and Facebook.

I did have my issues, and I worked through them, symfony and other frameworks like it, have made PHP development so much better, but eventually you get let in on little secrets. Secrets like other scripting languages.

In the last year I dabbled with ruby, but decided I didn't want to look at rails. A few people pointed me in the direction of python and Django as an alternative to PHP alternatives. So I've spent time learning python.

Yesterday and today I gave myself a real project. I started hacking away at del.icio.us for a Winter Hack Day at Yahoo! I would write a few lines of code, and then realize that I can write it a little better a little cleaner using some neat little python constructs. It's enjoyable in the way that PHP is not.

It might just be a phase, but I sincerely doubt it. A lot of people I know who use python love it. Many have been using it for years (even while I secretly thought, "python? really?" Well, I'm starting to think they've been onto something. We'll see where I end up.

python: UnicodeEncodeError: 'ascii' codec can't encode character u'\xbb' ...: ordinal not in range...

2008-01-25T00:00:00-08:00

I don't fully understand utf-8 errors, but I've been getting this error.

So here's a quick fix:

import sys
sys.setdefaultencoding('utf-8')

Python, Named arguments: Pure Genius

2007-12-21T00:00:00-08:00

I decided I want to learn python, if only to learn Django and to "get" what all the python hub-bub is about.

Python's named arguments in function calls is pure genius. Let me explain.

In PHP, and many other languages you can define a function as such:

function foo($a = 2, $b = 2)
{
    return pow($a,$b);
}

If you follow, foo() will give you 4. foo(3) is 9 and foo(99,0) is 1. In python we can do the same thing, but it'll pay to use some better variable names:

def foo(base=2, exponent=2):
    return base**exponent

Similarly foo() will give you 4. foo(3) is 9 and foo(99,0) is 1. But what if we forgot what the order was? Did base come first or was it exponent? We can do this:

foo(exponent=99, base=2)

Since base and exponent both have default values, we can even omit base and let it use the default:

foo(exponent=10)

This means rather than passing an $options array to my functions and checking whether an option was set or not, I can just specify which options I want in my function call. Or instead of remembering the order of the arguments, I can use whatever order suits me. Or instead of calling a function like bar(null, null, null, 2) I can just skip those first three arguments all together.

A side effect of this, is now there's a real use, even for simple functions, to give your variables easy to remember names.