Dave Dash

Plus Addressing

2012-04-07T00:00:00-07:00

One of my biggest pet peeves is registration forms that don't accept "plus" addresses, e.g. myname+whatever@mysite.com.

I've been using plus addressing for years, and before that, "minus" addressing. Since GMail handles my @davedash.com email addresses I am forced to use the "plus" style if I want to easily create multiple addresses on a whim.

I use this style of addressing to keep track of who might be spamming me (JC Penny for example). I might sign up with myname+facebook@mysite.com for Facebook, myname+pinterest@mysite.com for Pinterest, etc. Unfortunately this wasn't working when I signed up for Pinterest. My wife and a few of my friends were quick to tell me about this:

So when I joined Pinterest, I immediately filed a bug, and then after talking to a few people, decided to fix it. The fix was remarkably simple.

There's usually two causes for this problem. 1. People use the wrong validation for their emails. Namely one that says + is bad. Luckily most web frameworks, like Django, get this right. 2. They use the email address as some sort of parameter in a URL. For example: http://mysite.com/newsletter/unsubscribe?email=myname+whatever@mysite.com.

The wrong validation

If you're using the wrong validation, fix it. There's no excuse for anybody seeing "sorry please use a valid" email address when their email is perfectly valid. Django has a regular expression which you can use.

The URL issue

The URL issue is an issue of URL encoding. In our example (http://mysite.com/newsletter/unsubscribe?email=myname+whatever@mysite.com):

The email variable gets decoded as:

myname whatever@mysite.com

Clearly that is invalid. On a URL bar, the + is interpreted as a space. The solution is to encode your email address. For Pinterest, this is all we needed to do. JavaScript has a handy encodeURIComponent to change this. If you are using an ajax call from jQuery, you can simply pass the data an object:

{'email': 'myname whatever@mysite.com'}

This will get encoded properly.

Call to Action

Plus addressing a minimal use case, but please... make sure that people aren't hitting validation errors in your registration. Each input on a registration form is a reason to not register. Each validation error, is a reason to give up prematurely. Do your users and your product a favor and don't invalidate them unnecessarily.

Naming things and a recursion

2012-01-05T00:00:00-08:00

Most Mozilla webdev projects have an awful project structure, and it's partially my fault. I'm attemtping to fix that, but I cringe every time someone creates a new playdoh (Mozilla's Django template) based project.

The typical python project

Your typical python project looks like this:

/my_project
    someotherstuff/
    docs/
    theactualthingicareabout/
    setup.py
    LICENSE

MOZtrosity

We didn't have a good guide when we first started writing Django projects, so we opted for something like this:

theactualthingicareabout/
    apps/
        foo/
        bar/
    __init__.py
    urls.py
    settings.py
    LICENSE

In otherwords, the Django Project, which is a python module, is immediately checked out. If you check this out to an invalid directory, e.g. you do something like:

git clone github.com/davedash/myawesomeproject.git will.not.work\!

Bad things will happen.

So?

To some people this seems like an easy thing to work-a-round, but when it takes three of my excellent coworkers a week to diagnose an issue, where this ended up being the root cause... well it becomes a higher priority issue.

So here's what happened this week, when we tried to deploy the new careers site to a VM hardware.

Our ops team sensibly checked out the project like so:

git clone https://github.com/mozilla/lumbergh.git careers

They did everything right. Sure, they were creative and chose careers over the default lumbergh, but they knew the shortcomings of our system and picked a name that would resolve as a valid python package.

Unfortunately we'd hit some recursion error anytime we tried to hit a URL. So we knew there was an issue with the URL resolver, but we couldn't figure it out.

Here's what the project layout looked like:

careers/ # I could be called anything, but they chose careers
    __init__.py
    apps/
        careers/  # I'm going to cause problems,
                  # but neither devs nor ops will suspect a thing! mwahaha
            __init__.py
            models.py
            urls.py
            views.py
    settings.py
    urls.py

other files.

We configure our apps/ directory to be part of our PYTHON_PATH so we can do things like from careers import views... you can probably see where this is going.

Here's the main urls.py 1:

...
urlpatterns = patterns('',
    (r'', include('careers.urls')),
)
...

The main urls.py includes careers.urls which if you look at the above project layout, resolves to two different python packages:

careers/urls.py
careers/apps/careers/urls.py

Python chose the first, and therefore urls.py kept calling upon itself.

So what did we learn?

Do better.

First of all, we need a better project layout. This will continue to cause problems for even the brightest developers.

Secondly, if you don't do this at least name apps carefully. Django's app model can be a bit much for non third party apps. Sometimes there's one app which spans the entire project, and it's tempting to call it the same name as the project (e.g. careers), but sometimes a lamer more generic name like common is better.

But really, the second point is moot if we just clean up.

Choosing a New Web Stack

2011-10-09T00:00:00-07:00

I like building web sites, a lot. Usually every few years I need to re- evaluate the stack I use for a side-project. Joshua's Stack Parts site is handy for this.

At Mozilla Webdev we stick to reeds + mysql + elasticsearch + celery + rabbitmq + memcache + git + virtualenv + python + django + jinja2 + modwsgi + commander + puppet + apache + less + jquery as our go-to stack. It's tried and true and it's been working and been evolving for two years.

So I'm at re-evaluation time. The first element of the stack I needed to decide upon was the web framework. I initially thought I'd use Django, and maybe alternate a few supporting libraries just to color my experience. But Flask caught my attention.

Flask is from Pocoo who have given me great things like:

lodgeit
Werkzeug
Jinja2
Sphinx

It was a microframework, which meant that it didn't contain as many things as Django, but at the same time, I didn't use that much of Django.

Flask was a nice way to stay mostly in my comfort-zone, and in some ways, focus me on just writing an app, and not working in a framework. Since it's python, if I start to miss Django, I can probably rewrite my code without too much effort.

Overall I'm excited, and I just got past, "Hello World."

I'm not sure what my stack will look like, I'm imagining it will evolve into:

postgresql + memcache + git + flask + jinja2 + gunicorn + fabric + puppet + nginx + less + backbonejs + jquery

This will give me a chance to learn more about things I'm interested in, and utilize what I think might be better options along the stack.

DjangoCon Testing Tutorial

2011-08-10T00:00:00-07:00

If you want to learn all you can about testing anything in your Django App, see my tutorial at DjangoCon. It's on September 5th, it'll be 3 hours long and so far with seven sign ups it will be very hands-on.

Here's what I think I will cover, but I may change this depending on what the audience wants:

Testing issues
- ask people to fill out etherpad with issues they've run into
- ask someone to rank them in order of complexity
List an outline of topics
- post them on etherpad
- have people + them if they are interested
Testing overview
- We started in late 2009 early 2010
- Our largest project has 2500 tests
- Our next largest has 1100
- We have pretty good coverage
How testing works in Django
- I'm not 100% sure on this
- Test runner setups up a new database
- Test runner finds and runs tests
- Tests run class setup
- Test runs each test in a test case
  - Load fixtures
  - Tests run setup
  - Tests runs the test
  - Tests runs teardown
- Tests run class Teardown
- You get an F if you're bad and a . if your not.
- Now that you know it, you can hack it.
How we've hacked testing
- 2500 tests is a lot
- We no longer recreate the database when you run the test suite
- In each test case we just load the fixtures once.
- We rearrange the tests so things with the same fixture set run together
Testing tools that we use at Mozilla
- nose/django_nose
- nose plugins
  - nicedots
  - progressive
- coverage
  - git + whatchangedpy
Testing everything, no excuses
- 100% Coverage isn't important
- 80% is nice
- Good coverage on tricky things is important
- Some coverage on everything is important
- External
- If you start depending on APIs, Search or different tools you need to be able to test for them.
- Writing these test cases will take less time than this tutorial
- It will save you so much headache in the future.
- The same headaches you save yourself by writing "normal" tests
- Mock easy things
  - use a decorator on any test/view that might use redis
  - if redis isn't setup, use the mock client
  - mock client doesn't support everything,
    - just what I need to get my tests running -
    - feel free to extend it if you use it
  - Testing Redis
- Setup/Teardown for complicated tools
  - Good for search and APIs
  - Raise SkipTest (nose) if the developer doesn't want to run these tests
  - Non realtime tools
    - Testing Sphinx search
    - SetupClass
      - load fixtures
      - run indexer
      - run server
    - Sphinx server now available for all tests in your test case
    - Teardown
      - stop server
  - Real time tools
    - Nicer, data can be added in post_save signals or elsewhere in your app
    - Testing LDAP
      - Setup
        
        Remove LDAP files
        
        Load an ldif
        
        Start slapd
      - Your code can now touch LDAP
    - Testing ElasticSearch
      - We leave ES running all the time.
      - Setup
        
        Checks for ES support or SkipTest
        
        Deletes index
        
        Creates index
      - You can now read/write to ES
      - Teardown
        
        Delete's index
- Fixtures
  - Fixture Magic
  - Model Maker
- pitfalls
  - dates
  - using PDB

How we slug at Mozilla

2011-03-24T00:00:00-07:00

One problem we find with slug generators, is they do an awful job with unicode. For a string like this: Bän...g (bang) you get something like bng---g--bang- or at best bang-bang. But it's 2011, urls can have unicode... here's what we really want: bäng-bang.

In some cases transliteration might be acceptable. But if we look at Django's approach it fails at Russian. Here's a comparison with ours for the Russian phrase "Быстрее и лучше!" ("Faster and better!"):

>>> from django.template.defaultfilters import slugify as djslugify
>>> from slugify import slugify
>>> str = u'Быстрее и лучше!'
>>> print djslugify(str)

>>> print slugify(str)
быстрее-и-лучше

So as you can see, the built-in Django slugify could be disastrous. So take a look at ours. If you have some more test cases, please fork it.

Testing Redis in Django

2010-12-07T00:00:00-08:00

For the Firefox Add-ons we've been using redis here and there mostly for cache, but lately for a few things we'd love to persist.

Unfortunately relying on redis does mean we need to be able to test it. Since redis touches some of our core components of the site, we can't just raise a SkipTest like we would for Sphinx search related tests. I also don't want to rely on our developers to have redis installed in order to run the test-suite.

So I built a simple Mock Redis client. It's part of our redisutils.py that handles connections to redis. If a test's setUp method calls mock_redis you'll get this phony object that can do a few minimal redis-like operations.

It works great for our specific cases, but feel free to fork it and make it better.

Note: This MockRedis is specifically designed to work with django.

Faceted Search on Input

2010-10-29T00:00:00-07:00

So one trick with Sphinx search is faceted search. It's somewhat crudely implemented, by batching queries together, but does the job well. In the case of Firefox Input it can reduce quite a bit of queries (our search result pages take one batched sphinx query, and one database query now instead of 5 database queries).

Faceted search is search with filters to help narrow down a result set. I'll give you three examples. Firefox Add-ons which I wrote, Sitter City which gives you a lot of ways on narrowing down on the perfect baby sitter and ebay which lets your narrow down on auction items.

For Input we ask for the following when we do a search:

How many opinions match the term for which we are searching taking into account any preferences we have already specified (feeling, locale, operating system, date range, etc).
How many opinions show a positive sentiment, and how many show a negative sentiment?
What is the breakdown of languages for the opinion results. (I.e. how many are en-US, de, fr, etc).
How many people are on Mac, Linux or Windows.

We can batch these four queries into a single Sphinx request.

Here's our implementation.

Having done this twice, I do recognize that there is a lot of room for making the code a bit more reusable. But overall it runs fairly well.

Trimming Whitespace in Django Forms

2010-08-18T00:00:00-07:00

I've been using frameworks for a number of years. So I expect a lot of things to happen "for free" in Django. One is whitespace removal. In Delicious we had a lot of data in our database with leading and trailing whitespace. On the frontend we moved to symfony (actually ysymfony) and that prevented a lot of this.

So I was quite surprised that this is not the case with Django. So I decided we could solve this at the form level, and released a ridiculously simple library. After some googling, I found that I was not the first to do this.

Feel free to use this, fork it, submit pull requests, etc. I suspect in the future we'll handle other global form filtering - like stripping high order Unicode since MySQL is often not a fan.

The Perils of One Giant Fixture

2010-08-12T00:00:00-07:00

A while back, I thought it would be good to consolidate all the data used in testing the django-layer of AMO into a single data fixture. Unfortunately we have 600 tests, which were now loading and unloading large amounts of data each time the test would run. This made our tests take 20 minutes.

I decided to cut this down quite a bit, by using smaller fixture files. Each fixture file attempts to be a singular primary object (e.g. an Addon or a Collection or a User) and its associated supporting objects. It's far from perfect, but it's achieved tests that run in under 10 minutes.

The other side effect is tests will be simpler. They'll only include the addons needed to generate an effect, and if something can't be done easily with the fixtures in place, we can always alter the data during the test.

Your objects, and all their friends

2010-06-25T00:00:00-07:00

This is complicated.

In my ever-evolving quest to get data out of the AMO database for tests, I found myself not just extracting a single object, but a list of complicated requirements in order to fully replicate behavior in production in a testing environment.

For AMO we can use fixture magic to dump a single add-on and all of it's database dependencies so that it will insert safely into a test-database. But we need more than just valid data. We need some supporting data. For an add-on to be browsable and searchable it needs to have a valid version and the version needs to have a valid file.

In our app we can check for these things by using this:

my_addon.current_version.files.all()[0]

Of course we need to check that my_addon.current_version exists and that files.all() has at least one object. This ends up being a lot of work if you just know the id of the add-on object.

So what I want is something simple, like:

./manage.py custom_dump addon 3615

And it should get me everything I need to test add-on 3615, including a Version object and any files associated with the version.

Turns out this just works. It works if you define the following settings:

## Fixture Magic
CUSTOM_DUMPS = {
    'addon': {  # ./manage.py custom_dump addon id
        'primary': 'addons.addon',  # This is our reference model.
        'dependents': [  # These are items we wish to dump.
            # Magic turns this into current_version.files.all()[0].
            'current_version.files.all.0',
        ],
        'order': ('app1.model1', 'app2.model2',),  # stuff gets sorted
        'excludes': {
            'app1.model1': ('fields', 'to', 'hide',),
        },
    }
}

Using this we're able to find out that addon means an addons.addon object and that you want the addon object with an id of 3615. From there we'll try looking for dependent objects. Using some black magic we can turn: current_version.files.all.0 into addon.objects.get(pk=3615).current_version.files.all()[0]. This gives us a file.

If we mimic our dump_object command we can get the file into the database and everything that the file needs to be valid. This in turn gives us enough data (usually) to begin testing a single addon.

So have fun with this, if you're database is remotely complicated, this can save you some time replicating it during testing.

Also note, that you can re-order models and exclude certain fields. This can make your fixtures very easy to load.

Test Driven Confidence

2010-04-20T00:00:00-07:00

If you're already testing your web applications, you can skip this post.

One of the bugs I am working for AMO on involves porting a small, but moderately complicated checkbox from our PHP site and rewriting it for Django.

I decided to look at the existing implementation and found it to not work correctly at all. This was frustrating, especially since I verified that my own code worked, and that QA verified that it worked as well.

This is frustrating on many levels. Chances are some minor assumption I made changed, and thus broke this functionality. Discovering regressions is never fun, and fixing them is can be long and tedious if you can't automatically verify that everything is working correctly.

Lucky for me, coming up with tests is easy, you just do what you would do to verify the code satisfies the requirements and then code it. Sometimes the tests can take longer than writing the actual code, but ultimately you can ship with confidence. You can be confident that your feature won't break in the future without immediate notice, and you can be confident that your new code won't break anything else.

Making our tests run thrice as fast

2010-03-16T00:00:00-07:00

I've written a faster version of TransactionTestCase and packaged it with test_utils. It's mysql specific since it relies on SET FOREIGN_KEY_CHECKS=0 to flush the database.

The long story...

Why speed matters

We're closing in on 300 tests for Zamboni. As of yesterday, to run our entire test suite it would have taken approximately 5 minutes. If you run tests before code-reviews, during a code-review, and before you push to master - you've spent about 15 minutes doing tests for a single feature or bug-fix. We have about 5 developers, so this cycle happens many times in a work day. In that time many sandwiches can be made and consumed.

Even shortcuts, like running a subset of tests will only go so far, and ultimately we do want to validate that all our tests pass for any code-change.

Testing Sphinx search with `TransactionTestCase`

Django recently sped up testing by running tests in a transaction. However, this means that data never gets committed to the database and therefore external tools, like the Sphinx indexer, will never see any of that data. So we resort to TransactionTestCase which will commit the data.

Unfortunately TransactionTestCase is painfully slow. The accepted practice is to only use TestCase if you want your tests to be fast. So, I decided to complain to one of our new hires and he and I decided to tinker in mysql to figure out what was slow. We discovered the following:

delete from [table] is slow
truncate [table] is slow
... unless you SET FOREIGN_KEY_CHECKS=0

So we decided we should do our own tear down. After some tinkering with cProfiler I discovered that TransactionTestCase does a (slow) database flush on setup for a test case. This wouldn't do.

Making our own `TransactionTestCase`

I decided to make our own TransactionTestCase and it would just run SET FOREIGN_KEY_CHECKS=0 and TRUNCATE on each table at tear down time. It would also not do a flush on set up.

We write our tests with the idea that they clean up after themselves. Rather than having them cleanup after the last test. This is a requirement for us since django-nose doesn't reorder tests (nor should it) and a standard django.test.TestCase assumes a clean database.

Looking at a single test test_sphinx_indexer, using django.test.TransactionTestCase took ~30 seconds. Using our new TransactionTestCase it takes ~4 seconds!

Fast tests are good

We can now run our 275 tests in ~100 seconds versus the ~300 seconds it used to take. Furthermore, skipping our sphinx tests (which are the only tests that use TransactionTestCase) only saves us ~10seconds. That's not a lot of overhead for better coverage.

This took me the better part of a day, but solving this now, means we're going to more often than not run our sphinx tests all the time rather than skip them. Our QA team will assure you that search is probably the most regression prone part of our site, so running these tests are vital to quality.

If you need to use TransactionTestCase in mysql, give ours a try.

λ^2: safely doing class based views in Django

2010-03-09T00:00:00-08:00

When I started rewriting the API for addons.mozilla.org, my views were mostly the same: get some data and render it as either JSON or XML. I also wanted all my API methods to take an api_version parameter, so I decided class based views would be best. This way my classes could just inherit from a base class.

To do this I had to implement a __call__ method. This works fine, except I wanted to store things into the class -- after all the whole point of my use of classes was to keep the code a bit more compact, and cleaner. So, why pass the api_version around everywhere? Unfortunately thread-safety comes to play, and you need a separate instance of your class for each request.

λ

Django's urlpatterns expects a callable object. So you can't give it an instance of AddonDetailView(). But you could give it a callable that creates an instance of AddonDetailView() and passes it *args and **kwargs. Luckily python has lambda functions. You can note how we solved that in our urlpatterns.

λ λ

But wrapping all your urls with lambda is tedious and remembering to pass *args and **kwargs is error prone.

So let's make a lambda function that returns... a lambda function that turns an instance of our class into a callable.

We can now return to coding and not think about thread safety.

λλλ

django-fixture-magic: Testing issues with real data.

2010-03-05T00:00:00-08:00

I just released Fixture Magic.

When dealing with legacy data, you'll run into all kinds of edge cases. Perhaps, an object might not display correctly unless it has the right parameters, or if it has null parameters it might not display at all. So when testing Django, it's nice to actually use non-dummy data.

Luckily Django has a way of pulling real data out of your database using django.core.serializers:

from addons.models import Addon
a = Addon.objects.get(id=3615)
from django.core.serializers import serialize
jsonize = lambda a: serialize("json", a, indent=4)
jsonize([a])

This solution runs well in a Django shell and can be lots of fun for the whole family... until things get complicated.

Serializing alone isn't enough.

Serializing a fixture with foreign keys means you'll have an un-loadable fixture unless you serialize the dependent fixtures. Even for one or two foreign keys, this can be a pain. For addons.mozilla.org, we have a spidery-web of dependencies: Files need a Version which needs an Addon which need Translations.

Thus begat the dump_object management command. Give it an app, model name and a pk and it will give you not only a serialized JSON of that object, but all the objects that it requires.

Example:

./manage.py dump_object files.file 64874 64876 > my_new_fixture.json

This looks for the File model in the files app and pulls out of the database Files instances with pks of 64874 and 64876. It then recursively searches for any required objects.

Too much serial

If you create a lot of fixtures, you'll eventually have overlapping serialized objects. In addons.mozilla.org we have Addons, Versions (which depend on Addons) and AddonCategorys (which depend on Addons and Categorys). If we wanted to get serialize a specific Addon, it's dependent Versions and AddonCategorys it makes sense to start with dump_objecting the related Version and then dump_objecting the AddonCategory. Both dump_object commands will fetch the Addon in question, resulting in duplicated data.

To combat this we can use merge_fixtures to dedupe our fixtures:

./manage.py dump_object versions.version 64874 > 1.json
./manage.py dump_object categories.addoncategory > 2.json
./manage.py merge_json 1.json 2.json > happy_fixture.json

This should make creating test data slightly less painful. So give it a try.

Retrieving elements in a specific order in Django and mySQL

2010-02-11T00:00:00-08:00

If you have a list of ordered ids and you want to turn them into an ordered result set you can use FIELD() in mysql:

SELECT * FROM addons
ORDER BY FIELD(id, 3, 5, 9, 1);

This is a handy trick if you use an external search engine which gives you an ordered list of ids and you want to pull out entire row sets.

We do this in addons.mozilla.org using the Django ORM like so:

The code in action.

Django: Model Inheritance or Related Tables wrt AMO

2009-12-15T00:00:00-08:00

When I attended DjangoCon this year, I lamented that our flagship web property was difficult to test, and not fun to develop. I figured DjangoCon was a way to placate me, and Django might mean something for some of the smaller projects at Mozilla. However, Wil Clouser, our lead web developer, announced development changes for addons.mozilla.org (AMO) that says we'll be moving to Django.

Wil was open to Django and knew that's what we in the dev team wanted. Jeff spawned our foray into a new AMO with Zamboni. I've been working on some grunt-work tasks inside and outside of Django.

One of those tasks is building a transparent layer in Django to keep users logged in from our PHP-based site. That kind of problem almost immediately forces you to ask one of the most fundamental questions you ask when using any framework:

How much do I change my app, in order to accommodate the framework?

More specifically:

Should I use the django.contrib.auth User module, and to what extent?

The more we looked into what features of Django we might want to use, django.contrib.auth was heavily tied into other things we wanted, so it made sense for us to use it. The next question is whether we try the inheritance approach or do we treat our legacy users table as a sort of User Profile and utilize the User module using the related table approach?

Using model-inheritance seems real nice, because we can pretend that our legacy user is the same thing as a djaango.contrib.auth User - but this isn't true:

Looking at our users table more closely:

mysql> explain users;
+-------------------------+---------------------+------+-----+---------------------+----------------+
| Field                   | Type                | Null | Key | Default             | Extra          |
+-------------------------+---------------------+------+-----+---------------------+----------------+
| id                      | int(11) unsigned    | NO   | PRI | NULL                | auto_increment |
| email                   | varchar(255)        | YES  | UNI | NULL                |                |
| password                | varchar(255)        | NO   |     |                     |                |
| firstname               | varchar(255)        | NO   |     |                     |                |
| lastname                | varchar(255)        | NO   |     |                     |                |
| nickname                | varchar(255)        | YES  | MUL | NULL                |                |
| bio                     | int(11) unsigned    | YES  | MUL | NULL                |                |
| emailhidden             | tinyint(1) unsigned | NO   |     | 0                   |                |
| sandboxshown            | tinyint(1) unsigned | NO   |     | 0                   |                |
| homepage                | varchar(255)        | YES  |     | NULL                |                |
| display_collections     | tinyint(1) unsigned | NO   |     | 0                   |                |
| display_collections_fav | tinyint(1) unsigned | NO   |     | 0                   |                |
| confirmationcode        | varchar(255)        | NO   |     |                     |                |
| resetcode               | varchar(255)        | NO   |     |                     |                |
| resetcode_expires       | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notifycompat            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| notifyevents            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| deleted                 | tinyint(1)          | YES  |     | 0                   |                |
| created                 | datetime            | NO   | MUL | 0000-00-00 00:00:00 |                |
| modified                | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notes                   | text                | YES  |     | NULL                |                |
| location                | varchar(255)        | NO   |     |                     |                |
| occupation              | varchar(255)        | NO   |     |                     |                |
| picture_type            | varchar(25)         | NO   |     |                     |                |
| averagerating           | varchar(255)        | YES  |     | NULL                |                |
+-------------------------+---------------------+------+-----+---------------------+----------------+

You can very easily argue that this is a profile table, which happens to have credential information thrown in.

I can see overtime, I'll just struggle to keep our legacy User to act like a Django User, whereas a UserProfile is fairly standard.

Had I been writing this app from scratch, I would have chosen the UserProfile route. This is extra data which takes up a lot of space, and changes far more often than user credentials. Changing 4M+ rows sucks, by making users our UserProfile table, any changes to that table, don't tie up the table used for sign-ins.

I'm curious what other people who port their apps to Django have done.

DjangoCon wrapup

2009-09-15T00:00:00-07:00

I went to DjangoCon this past week for work. Django is one of my favorite frameworks. I dropped PHP and the symfony framework to learn python and Django and I haven't looked back. I think for Mozilla's webdev team it would be the framework of choice. We have 100s of sites in many frameworks, but not a lot of resuability. Django apps are built to built to be reusable. If you build correctly you don't have to refactor, it's already done.

Here's a collection of notes I collected through the conference.

Day one

Keynote - Avi Bryant

Frameworks lock us into RDBMS = bad

This keynote mentioned the limits of modern frameworks and modern web development. Essentially frameworks are great for getting started, but as a site grows, the framework gets replaced little by little. Sometimes it can get in the way - such as with limitation of database choices.

UR doing it wrong - James Bennet

James outlined a few key problems that many Django developers run into:

learning python as you go
- doesn't work unless you know some programming upfront
- do the python tutorial
- read python in a nutshell or dive into python
Things you should know:
- subclasses
- super()
- slides went too fast... hopefully they'll be posted

All in all RTFM for python and Django :)

Learn about other py packages... like twisted. If Twisted Matrix was implemented in Ruby it would be advertised as the second coming of Christ.

Bennet's Django App review smoketests:

installable via pip, easy_install or setup.py
- read distutils-guide
- stay away from setuptools
have a README
INSTALL file list deps
Write DOCUMENTATION
- use sphinx.pocoo.org
- store it in your package and upload package docs
LICENSE (most Django apps use BSD)
Write unit tests
django-lint - to look over code (like pep8.py)

pro-django is a decent book, but not written by Bennet.

Testing - Eric Holscher

Django 1.1 encourages you to test by auto-creating tests.py.
Support for:
- Unittests
- Doctest
- Tests done in a db transacation
Test Driven Documentation (TDD + DDD)
Doctest
- easy
- can't use PDB
- Hides certain failures
Unittests via Django TestCase
- XUnit
- setup/Teardown
- adds db fixtures
- assertions
- mail testing/inbox testing
- url testing
TestCase
- Browserless Request/Response testing
- Similar to sfBrowser in symfony
Google Summer of Code (for Django 1.2)
- Coverage reports!
I need to learn PDB

Deploying Django -

Run mod_wsgi in daemon mode.

Day 2

Keynote - Ian Bicking

GNU Manifest:

I consider that the golden rule requires that if I like a program I must share it with other people who like it. Software sellers want to divide the users and conquer them, making each user agree not to share with others. I refuse to break solidarity with other users in this way. I cannot in good conscience sign a nondisclosure agreement or a software license agreement. ...

So that I can continue to use computers without dishonor, I have decided to put together a sufficient body of free software so that I will be able to get along without any software that is not free.

GNU manifesto was the idea of sharing software amongst friends
GNU has purpose - BSD, etc is just a rule - free to share
Free is not just the absense of copyright
Free is not a reaction to existing rules, but a golden rule
Not just a fight against MS
Need to find morality (the why) within the practical (the law, or what you can do)
Open sourcing closed source code isn't building open source
This might apply to Mozilla... as webkit has taken off more than Gecko.
Open source is person to person not company to company - despite sponsorship.

Using Django in Non-standard ways - Eric Florenzano

Django loosely coupled
Replace templating with Jinja 2
Copy Django methods into djangoext to easily customize Django behavior
Not using django.contrib.auth
- reasons: writing a fb app - no auth needed
- no shoehorning needed - saves time - less overhead
skip the orm?
- legacy dbs
- non standard or db (or non-relational database)
- no database
wsgi middleware has some cool shit
- repose.bitblt: autoscales images
- repose.squeeze: will concat js/css on the fly based on statistical analysis
non standard Django based apps
- YARDBird - IRCBot framework
- djng micro framework
- Jngo- singlefile cms
using admin in a nonstandard way is hard/impossible coupled with ORM and auth

Real-time web and other Buzzwords - Chris Wanstrath

more than just getting your rss feeds faster
push vs. pull
1 persisting connection vs polling
comet/flash-xml/or html5 web socket
orbitted - open source python comet server
zeddicus - does the business logic
orbitted has its own js libs - its a simple port/socket thing for your server code to deal with - not request/response.
all connections are persisting browser/orbitted orbitted/zeddicus
You can even use orbitted to connect straight to IRC and write a client in JS
Jetty also is good for comet

Also: * see webhooks * see pubsubhubub

Pluggable, Reusable Django Apps: A Use Case and Proposed Solution - Shawn Rider and Nowell Strite

PBS moved from perl to django - build a lot of reusable apps
convincing your superiors
- need a good story -
- existing base of python helped
- With Django easy to do things right without doing things slow
- be really good...
built a lot of apps to be very reusable, and pluggable based on requirements PBS had

Day 3

Keynote - Ted Leung - Sun

Django jobs are a growing market
Preferred by startups
Bespin/wave - cool
APIs are big... still
Physically impossible to create purely server-side interactions that are usable enough - rely on rest/comet/ajax/etc to bridge gap

Scaling Django Mike Malone

MM from Pownce (now sixapart)
Slides started out as "Building Scalable Web Applications"
Django didn't get in the way too much when it came to scaling
Django had tons of caching support
Cached objects by hand (memcached) and object ID lists
Use memache for sessions too
use signals to signal cache invalidation
race conditions...
Queue shit... gearman, rabbit mq, etc.
Memecached incr/decr operators are awesome
See gh/mmalone/django-caching
See gh:.../django-multidb
to combat slavelag use a memcache key to alternate between master or slave

Gearman - working later - Chris Heisel

Gearman - a work later alt to rabbit mq
Makes the most sense for something like cesium, with a bazillion worker ~~bees~~ foxes feeding off a single queue

Also at the con, I talked to someone about rebuilding large apps... and they took a PHP app and used URL rewriting to and a lot of PHP/Python glue code to build a seamless transitory app. The rule is, all new functionality was done up in python while the old app was in maintenance mode.

More talks here!

Resolving Django dumpdata errors

2009-09-13T00:00:00-07:00

Recently I recieved this wonderful piece of news when I ran ./manage.py dumpdata for the first time:

Error: Unable to serialize database: User matching query does not exist.

I knew this might not work out since I was dealing with a legacy database, but the resolution is quite simple. First I had to narrow it down to which app was causing this. Naturally I assumed it was one of the two apps I had, either common or restaurant. So I ran: ./manage.py dumpdata common and ./manage.py dumpdata restaurant. The latter had no problem whatsoever.

This made sense, since my common application was the only one that made any reference to a User. By looking in my models.py for that application, I narrowed it down to my Profile object. Sure enough, commenting it out meant I could get my data.

It ended up being a foreign key mismatch between the profile and user tables. Since this is legacy data, this mismatch made sense. A simple SELECT id,userid FROM profile WHERE userid NOT IN (SELECT id FROM auth_user) gave me a list of bad profiles. Removing them allowed me to create my Django fixtures.

Automating Django Redirects

2009-03-07T00:00:00-08:00

Django has a very simple redirects system. Simple as in it's easy to understand. If someone encounters a 404, Django Redirects catches this and does a final lookup to see if there's an entry to a new URL in the redirects table.

The real win, is when your slug fields change (and thus the get_absolute_url() of your objects), you can simply automate the creation of a redirect:

Note you must have enabled the redirects app before trying this. You also need a get_absolute_url() function defined for your MyObjectClass. This is the true value of Django Middleware. Let those truly repetitive site-wide tasks get done in one spot.

A stitch in Fabric saves time

2009-03-02T00:00:00-08:00

Each day as I grow as a developer I pick up better habits. One is automating anything that I'll have to do more than once.

Therefore, deploying important projects has evolved:

Edit files live on the site.
FTP files from my local machine onto the live site and do tweaking as needed.
Develop using a cross-platform capable framework and deploy live via rsync.
One step deployment.

I looked at a few methods for performing one-step deployment. Shell scripts seemed to basic, and things like Puppet seemed to large a scale. So I looked into Capistrano and Fabric, and settled on Fabric, because it was barebones and python.

I'm at a stage with my project where I need to test it on an external server, so rather than uploading it and winging it I actually thought about a few things.

The task

I was thinking the ideal situation is a single command line that I can type that will upload my files to the server. Furthermore it should upload first to a new directory and then symlink and restart the server to do the cutover. I can easily write another script to rollback.

The way I setup my Nginx and Apache servers was they would look for the symlinks: staging, test, production to serve up the respective environments (I'm using a single host for this example).

Directory Structure

I use a very particular directory structure:

$WWW_APPS/mysite.com/ - is my root app deployment directory.
- releases/ - each code push is stored in a unique directory
  - 1/
  - 2/
  - ...
  - 100/ - the directory name is the svn revision, but is a series of precise svn exports, not a checkout.
- staging - this symbolic link links to a specific revision directory that has been designated for staging purposes. This might be the last directory in releases/, e.g. $WWW_APPS/mysite.com/releases/100/
  - config/ - any config files go here (e.g. nginx or apache)
  - mysite.com/ - the actual Django project
  - scripts/ - any utility scripts, mostly database model changes
  - site-packages/ - any related libraries that I need, this isn't the best approach I need to investigate virtualenv
  - static/ - all my static assets go here
- staging.rollback - the former staging symlink is demoted to staging.rollback
$WWW/mysite.com/
- staging - This is where the static assets get served from according to nginx. It's a symbolic link to $WWW_APPS/staging/static. This link does not change during deployment.
- uploads_staging/ - This directory contains uploads (user data). It requires manual adjustment when there are data changes (deletes, adds, updates). For the most part it can stay unchanged from one deployment to the next.

Subversion

I'm currently using subversion to maintain my code. If you are starting a project anew, I suggest using git as it is designed with branching in mind. The script below should be adaptable for git instead of svn.

That being said, when I deploy code, I use the revision number as the directory name for that deployment (e.g. $WWW_APPS/mysite.com/releases/100/ for r100) .

This forces me to commit my changes and not deploy code that is not checked in.

Other viable alternatives would be time-stamped directories. Anything that is unique between deployments will be sufficient.

The `fabfile.py`

Here's the fabfile.py I constructed. It's by no means final

def staging():
    "Pushes current code to staging, hups Apache"
    # get the build number    
    local('svn up mysite.com')
    
    config.svn_version   = svn_get_version()
    
    if not config.svn_version:
        abort()
    
    config.static_path   = '/var/www/static.mysite.com'
    config.svn_path      = 'http://svn.mysite.com/trunk'
    config.svn_export    = 'svn export -q -r %(svn_version)s'
    
    run('mkdir %(path)s', fail='abort')
    
    # svn export mysite.com to path 
    run('%(svn_export)s %(svn_path)s/mysite.com %(path)s/mysite.com', fail='abort')
    
    # svn export site-packages to site-packages
    run('%(svn_export)s %(svn_path)s/site-packages %(path)s/site-packages', fail='abort')
    
    # svn export mysite.com to path 
    run('%(svn_export)s %(svn_path)s/scripts %(path)s/scripts', fail='warn')
    
    # svn export configs
    run('%(svn_export)s %(svn_path)s/config %(path)s/config', fail='abort')
    
    # export /var/www/static.mysite.com/releases/%(svn_version) 
    run('%(svn_export)s %(svn_path)s/static %(path)s/static', fail='abort')
    
    # symlink to images from /var/www/static.mysite.com/staging/images/menuitems/* new release dir
    run("rm -r %(path)s/static/images/menuitems", fail=abort)
    run("ln -s %(static_path)s/menuitems_staging %(path)s/static/images/menuitems", fail=abort)

# rotate "staging" symlinks
    run('rm %(releases_path)s/staging.rollback', fail='warn')
    run('mv %(releases_path)s/staging  %(releases_path)s/staging.rollback', fail='warn')

# staging sym to new destination
    run('ln -s %(path)s %(releases_path)s/staging', fail='abort')
    
    # server is hup'd
    invoke(hup)

def rm_cur_rev():
    config.svn_version   = svn_get_version()
    run('rm -rf %(path)s', fail='abort')

def hup():
    sudo('/etc/init.d/apache2 restart')
    sudo('/etc/init.d/nginx restart')
    
    
def svn_get_version():
    from subprocess import Popen, PIPE
    output = Popen(["svn", "info", "mysite.com"], stdout=PIPE).communicate()[0]
    return output.partition('Revision: ')[2].partition('\n')[0]

config.fab_hosts = ['mysite.com']
config.fab_user = 'builder'
config.releases_path = '/var/www_apps/mysite.com'
config.path          = '%(releases_path)s/releases/$(svn_version)'

Final Thoughts

There's room for improvement, but this is a start. virtualenv looks like another avenue I might want to explore (especially when it comes time to upgrade Django, python, MysqlDB or PIL). This also doesn't take care of priming the server with necessary packages - again, something virtualenv could make easier for me.

However, this file was not too difficult to create. I sat down, thought about the ideal solution, implemented the steps and iterated on that until the script worked. I have pushed 19 revisions of code so far, mostly to test this process, but also to make necessary adjustments. I've probably saved countless hours of logging in and doing things manually.

How do you deploy your apps?

Resizing Image on upload in Django

2009-02-21T00:00:00-08:00

I had trouble wrapping my head around Django ORM's handling of Images.

The first hurdle was realizing that there is no easy way to store images in the database. I couldn't be too upset with that, it was a flawed concept at best. I wrote a dump script to fix this, and converted my app to read from disk. One of those "good" problems.

When it came to uploading new content... I seemed to hit every roadblock. So I thought this guide might be of use for people (other than future-me). What I want to do is upload an image, resize it per the requirements of my app, and save the resized image.

The Template

The template was fairly easy, yet I forgot to specify the enctype attribute. Here's what you need:

<h2>Upload a new photo</h2>

<form action="" method="POST" class="cmxform" enctype="multipart/form-data">
  <fieldset>
    <ul>
      <li>
        
        <label for="id_image">Photo</label>
        
      </li>

    </ul>
  </fieldset>
  <input type="submit" value="Submit" class="submit"/>
</form>

Resizing an image

Resizing an image was straightforward. The Python Imaging Library is very easy to work with, and very similar to the PHP GD Library. There was one caveat...

def handle_uploaded_image(i):
    # resize image
    imagefile  = StringIO.StringIO(i.read())
    imageImage = Image.open(imagefile)

    (width, height) = imageImage.size
    (width, height) = scale_dimensions(width, height, longest_side=240)

    resizedImage = imageImage.resize((width, height))

    imagefile = StringIO.StringIO()
    resizedImage.save(imagefile,'JPEG')
    # ...

I'm using OS X and didn't have libjpeg installed. The recommended way was to install it via fink.

Generating a filename

I like using MD5 to generate filenames for images because MD5 almost guarantees uniqueness ("unique enough").

   ...
    filename = hashlib.md5(imagefile.getvalue()).hexdigest()+'.jpg'
    ...

Saving the file via the model

The model is supposed to put the files in the right spot and populate the meta data.

You can supposedly do this via:

   ...
    my_object = MyDjangoObject()
    my_object.photo.save(filename, content)
    ...

That will also take care of my_object.save(). The trick is that content needs to be a django.core.files.File object or things will never work. Further more, you can't use a StringIO for your File object. A true file object is best. Hence rewriting the JPEG:

    imagefile = open(os.path.join('/tmp',filename), 'w')
    resizedImage.save(imagefile,'JPEG')
    imagefile = open(os.path.join('/tmp',filename), 'r')
    content = django.core.files.File(imagefile)

Note: I also had to reopen the imagefile.

All together now

@login_required
def my_view(request, slug, item_slug):

    if request.method == 'POST':
        form = NewImageForm(request.POST, request.FILES)

        if form.is_valid():
            handle_uploaded_image(request.FILES['image'])
            return HttpResponseRedirect(elsewhere)
    else:
        form       = NewImageForm()

    return render_to_response(somewhere, locals(),
                              context_instance=RequestContext(request))

def handle_uploaded_image(i):
    # resize image
    imagefile  = StringIO.StringIO(i.read())
    imageImage = Image.open(imagefile)

    (width, height) = imageImage.size
    (width, height) = scale_dimensions(width, height, longest_side=240)

    resizedImage = imageImage.resize((width, height))

    imagefile = StringIO.StringIO()
    resizedImage.save(imagefile,'JPEG')
    filename = hashlib.md5(imagefile.getvalue()).hexdigest()+'.jpg'

    # #save to disk
    imagefile = open(os.path.join('/tmp',filename), 'w')
    resizedImage.save(imagefile,'JPEG')
    imagefile = open(os.path.join('/tmp',filename), 'r')
    content = django.core.files.File(imagefile)

    my_object = MyDjangoObject()
    my_object.photo.save(filename, content)

I'm sure there's room to optimize this flow. But this gets the job done. I do wish I could have used a StringIO instead of an actual file to save the image. Writing to disk twice seems silly.

Database versus files for Images

2009-02-18T00:00:00-08:00

This is a dead topic for sure, but one of the bad web development habits I had picked up was that storing everything in a database made things easier.

I actually put some effort into thinking this through. For me it was a case of management. I didn't want to have to worry about two data stores for application-generated data. In otherwords, the data in the database was all generated via my web application (either by myself or others who worked on the product, or by end users). Having to also remember that there were select files as well seemed like a disaster.

Most frameworks, correctly assume you'll upload files to a filesystem. I never fully understood this until I thought about how I'd speed things up when the time comes to speed things up. Django forces you to think that way.

Almost from the start Django encourages you to:

Upload any binary content to the filesystem with pointers in a database
Have a separate server, or even machine serve static content.

Furthermore in a cached environment or even an environment that utilizes a CDN, putting static content in one spot, including user-generated content, was a big win.

I've been porting an app from symfony to django for some time, and I had been serving images via the database. Immediately when I switched to the filesystem I saw a huge benefit. Not just a drop in database connections, but overall "zippiness" in the site.

We'll see how well this performs in the real world, but I am quite sure that I learned my lesson.

Versioning Django Models

2009-02-13T00:00:00-08:00

In symfony, versioning a model was not terribly difficult. I had my own specialized brute-force way of doing this.

In my experience it's been a lot easier to write python code than PHP code, so naturally I figured this would be an easy task. It was not. I suspect that it was not easy because I have a naive understanding of Django and my symfony knowledge was fairly well grounded.

I tried looking for a generic implementation of django model versioning, but failed. So I came up with a specific solution.

How I think of versions

For my app I chose to use a sparse versioning system. I'd have one interface for interacting with a model. For example, I have a Restaurant model and a RestaurantVersion model.

Restaurant would directly store immutable elements like name or rating or approved and it would encapsulate versioned elements like description by proxying to RestaurantVersion.

This means I only need to interface with one class and let the versioning happen behind the scenes.

The Django implementation of `RestaurantVersion`

RestaurantVersion held the data that I thought might suffer from corruption, and therefore require reversion. Therefore there's nothing surprising in this table:

The only thing of note is that I'm enforcing a 1:M relation between Restaurant and RestaurantVersion.

The Django implementation of `Restaurant`

Restaurant is where the magic happens. It needs to do a few things:

Do what a normal object does.
Proxy attributes to the corresponding attribute of a RestaurantVersion
Generate a new RestaurantVersion on demand.
(optional) Manage which version is "active".

I say "4" is optional because I don't do this myself. I built versioning into a few models because it was easier to do upfront than worry about it down the road.

Here's the code I use:

class Restaurant(models.Model):
    name           = models.CharField(max_length=765, blank=True)
    stripped_title = models.CharField(max_length=384, blank=True)
    approved       = models.IntegerField(null=True, blank=True)
    version        = models.ForeignKey(RestaurantVersion, related_name="the_restaurant")
    updated_at     = models.DateTimeField(auto_now=True)
    created_at     = models.DateTimeField(auto_now_add=True)
    new_version    = None
    
    def save(self, force_insert=False, force_update=False):
        if not self.stripped_title:
            self.stripped_title = slugify(self.name)
        
        super(Restaurant, self).save(force_insert, force_update)
        if self.new_version:
            self.new_version.restaurant = self
            self.new_version.save()
            self.version = self.new_version
            super(Restaurant, self).save(force_insert, force_update)        
        
    def __setattr__(self, name, value):
        if name == 'description':
            self.get_new_version().description = value
        
        elif name == 'url':
            self.get_new_version().url = value
        
        else:
            object.__setattr__(self, name, value)

def __getattr__(self, name):
        try:
            if name == 'description':
                return self.version.description

elif name == 'url':
                return self.version.url
        except RestaurantVersion.DoesNotExist:
            return ''
            
        models.Model.__getattribute__(self, name)
    
    def get_new_version(self):
        if self.new_version == None:
            try:
                rv    = self.version
                rv.id = None
            except RestaurantVersion.DoesNotExist:
                rv = RestaurantVersion()
            
            self.new_version = rv

return self.new_version
            
    class Meta:
        db_table     = u'restaurant'

It's not rocket science, but it wasn't necessarily easy either. Let's look at requirements 2 and 4 in a bit more detail.

Proxy attributes to the corresponding `RestaurantVersion` attributes

Proxying attributes is a way of masking the whole versioning infrastructure from the developer. We do this with __getattr__ and __setattr__ methods.

The __getattr__ determins if you are looking for description or url attributes of a Restaurant. If you are then it uses the current RestaurantVersion's attribute.

__setattr__ is very similar, except since we're changing a versionable attribute we're going to make a request to a method, get_new_version(), which we will detail later. It does what it says, though, it gets the "new" RestaurantVersion of the Restaurant.

Generate a new `RestaurantVersion` on demand

As you can see from the above code, I am "auto-versioning". This is done mostly via get_new_version(). We also have to do a few tricks to make sure the bidirectional relationship gets maintained on save.

get_new_version() either returns the current "new version", a brand new "new version" or a "copy" of the current version.

The "copy" is done simply by setting the id attribute of the new version to None. Django takes care of assigning it a new id on save, thus preserving the old version.

Note that when we create a brand new RestaurantVersion, we don't immediately set it's restaurant attribute. That's because we haven't saved the current restaurant yet. It's a "chicken and the egg" problem that gets solved in our save() method:

We first save the Restaurant, then we save a new version if there is one. If we save a new version we want to update the Restaurant a second time. This ensures that there's a 1:1 relationship between Restaurant and the current RestaurantVersion and it also establishes a 1:M relationship between Restaurant and all RestaurantVersions.

Conclusion and Drawbacks

While, I think that this setup works in my particular situation, I feel like it has some major flaws.

The code is quite messy and very specific to this model. My goal was to take care of this quickly, not necessarily in a reusable manner. I also was not familiar enough with the Django model to create some sort of extension.

This way of versioning does not allow for versioned attributes to be set upon instantiation of the model:

r=Restaurant(description="Great place")

won't work. You'll have to do:

r=Restaurant()
r.description="Great place"

I figured this was acceptable.

Lastly I'm not entirely happy with having to explicitly save the Restaurant model twice, but I think my bidirectional relation requires this.

All in all this works in my particular situation. I'm hoping that this can be simplified. I'm curious to hear about other versioning schemes.

Mimic propel's update_at and created_at in Django models

2008-11-29T00:00:00-08:00

One "trick" that propel offers you is tables with fields created_at and updated_at.

The fields are self explanatory and within Propel you just define them and they just do their business.

To simulate this in Django just do:

updated_at = models.DateTimeField(auto_now=True)
created_at = models.DateTimeField(auto_now_add=True)

auto_now automatically sets a field to the current time, and auto_now_add only does this if the object is being added (not updated).

Django models: saving markdown

2008-11-29T00:00:00-08:00

I love markdown. I write my blogs in markdown. For most user text areas in my web apps I support markdown and save both the markdown and the formatted text into my data store.

To do this I had to install markdown for python. For my Django projects I prefer downloading (or externally linking) to source code versus easy_install.

Then you override your save model like so:

from markdown import markdown

class MyNote(models.Model):
    def save(self, force_insert=False, force_update=False):
        self.html_note = markdown(self.note)
        super(MyNote, self).save(force_insert, force_update)

Easy.

Custom error messages for Django forms

2008-11-28T00:00:00-08:00

For some reason, it was difficult for me to find the documentation for this. If your Django form field is required you'll by default get an error stating that 'This field is required.' You can easily replace that when defining your form like so:

class ReviewForm(forms.Form):
    rating = forms.ChoiceField(choices=STARS, error_messages={'required': 'Please choose a star rating'})
    note   = forms.CharField(widget=forms.Textarea(),)

error_messages is just a simple dictionary of validation messages that override the default.

While this is properly documented it was not quickly searchable. But the django form fields reference page documents this well.

sfGuardUser -> django.contrib.auth

2008-11-20T00:00:00-08:00

If you find yourself moving from symfony to Django, here's how you sf_guard_user's user table to django.contrib.auth user table:

INSERT INTO 
    auth_user (id, username, password, is_active, last_login, date_joined)
SELECT 
    id, 
    username, 
    CONCAT(algorithm,'$', salt, '$', password), 
    1, 
    last_login, 
    created_at 
FROM sf_guard_user;

DROP TABLE sf_guard_user;

Luckily, django uses a similar salting and encryption strategy as symfony, so it's easy to go back and forth.

Django Admin and Cookies

2008-11-15T00:00:00-08:00

I was dusting off an old Django project and everything was working except the admin site:

Looks like your browser isn't configured to accept cookies. Please enable cookies, reload this page, and try again.

The culprit was SESSION_COOKIE_DOMAIN = "onyxfoundation.org" being set.

The project wasn't loading into Development mode (it's triggered by hostname) so it set the cookie domain to what I require in production. Once remedied, everything worked as expected.

Why code rewrites are often coupled with redesigns

2008-08-13T00:00:00-07:00

I've done some rewrites of code, and they usually are coupled with redesigns.

Redesigns and rewrites are tricky. With web sites existing users tend to prefer incremental changes with each. Changing a design element or a feature are pretty much equivalent when its done incrementally.

Usually small changes are like surgery. You make a small change, you possibly announce it, and people look at it and usually say yes, this change is better. But mostly a small change can go unnoticed.

Tiny atomic changes are nice and they improve the product.

A rewrite of code however is a discrete change. They take forever. Second system and all that. They inevitably involve a redesign as well. They tend to be received with mixed reviews.

Generally a design of a site is grafted onto code. Sometimes by templating languages, sometimes in a tightly coupled system. In either case there's usually two options:

Make a faithful port of the design.
Redesign

The latter is generally preferred since this saves a lot of time. Generally there are bugs with a design, and there is a design backlog. A faithful port of the design would mean porting every broken HTML code, every non user-friendly element, etc.

Unfortunately this leaves web developers with a scary situation: Launching a product which changes peoples functional and UI experience.

The good thing is redesigns usually mean evolutionary leaps. Meaning, iterations can happen faster, both in design and functionality. We can easily please new customers, and we can quickly bring current customers up to speed.

In my spare time, I've been rewriting code that I wrote in symfony (PHP) into django (python). It's a tiresome long process that can easily kill a side project.

nginx proxying to apache

2008-04-16T00:00:00-07:00

I gave up on fastcgi with NginX and django. Too many things just didn't work, so I decided to keep Apache, but lock it down and thrown NginX in the front to serve static content and to prevent max client issues.

I also applied a similar approach for symfony. Server configurations after the jump...

Apache

You should configure Apache as you would if you were serving directly from Apache. In otherwords, if you are already using Apache, you don't need to do anything different, save changing a port.

The one tweak I made to a standard configuration is to limit the allowed hosts to just the ones I want.

Here's my django app's apache.conf:

NameVirtualHost *
<VirtualHost *>
    ServerName onyxfoundation.org

    SetEnvIf X-Magic-Header ^secret$ let_me_in
    <Location "/">
            SetHandler python-program
            PythonHandler django.core.handlers.modpython
            SetEnv DJANGO_SETTINGS_MODULE onyx.settings
            PythonDebug On
            PythonPath "['/var/www/django/'] + sys.path"
            Order Deny,Allow
            Deny from all
            Allow from env=let_me_in
    </Location>
</VirtualHost>

Note the SetEnvIf it can be set however you want, I prefer to match a specific header that I send from nginx.

If you are using symfony, something like this will be more appropriate:

<VirtualHost *>
        ServerName reviewsby.us
        DocumentRoot /var/www/reviewsby.us/web
        DirectoryIndex index.php
       SetEnvIf X-Magic-Header ^secret$ let_me_in

        <Directory /var/www/reviewsby.us/web>
                Options Indexes FollowSymLinks

                RewriteEngine On

                RewriteRule ^$ index.html [QSA]
                RewriteRule ^([^.]+)$ $1.html [QSA]

                RewriteCond %{REQUEST_FILENAME} !-f
                RewriteRule ^(.*)$ index.php [QSA]
                Order Deny,Allow
                Deny from all
                Allow from env=let_me_in
        </Directory>
</VirtualHost>

You'll also need to listen on a different port (preferably one that is not exposed to the outside world):

Listen 8080

NginX

Now NginX needs to be configured to do 2 things: 1. Serve all static content 2. Pass requests that need interpreting to django or symfony

The Django conf of Nginx is very easy:

server {
  listen   80;
  server_name onyxfoundation.org;

  location ^~ /static/ {
    alias        /var/www/django/onyx/static/;
    access_log   off;
    expires      30d;
  }

  location / {
    proxy_pass      http://127.0.0.1:8080/;
    proxy_redirect  off;

    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

    proxy_set_header        X-Magic-Header "secret";
    client_max_body_size       10m;
    client_body_buffer_size    128k;

    proxy_connect_timeout      90;
    proxy_send_timeout         90;
    proxy_read_timeout         90;

    proxy_buffer_size          4k;
    proxy_buffers              4 32k;
    proxy_busy_buffers_size    64k;
    proxy_temp_file_write_size 64k;

  }
}

The settings are trivial. Everything that's /static gets served straight from NginX and the django application is served over Apache.

The same thing is a bit trickier in symfony:

server {
  listen   80;
  server_name  reviewsby.us;

  charset utf-8;

  location  /sf/ {
    alias  /var/www/reviewsby.us/lib/vendor/symfony/data/web/sf/;
  }
  location /css/ {
    alias /var/www/reviewsby.us/web/css/;
  }
  location /images/ {
    alias /var/www/reviewsby.us/web/images/;
  }

  location /js/ {
    alias /var/www/reviewsby.us/web/js/;
  }

  location / { 
    proxy_pass http://reviewsby.us:8080/;
    proxy_redirect off;

    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

    proxy_set_header        X-Magic-Header "secret";
    client_max_body_size       10m;
    client_body_buffer_size    128k;

    proxy_connect_timeout      90;
    proxy_send_timeout         90;
    proxy_read_timeout         90;

    proxy_buffer_size          4k;
    proxy_buffers              4 32k;
    proxy_busy_buffers_size    64k;
    proxy_temp_file_write_size 64k;
  }
}

The added difficulty with symfony is specific to this app (and most default apps) since static content is split amongst the web root (/js, /css, /images, etc) versus the suggested django approach of throwing everything in /static. I now adopt this approach for symfony as well. It makes everything more flexible in the long-run.

Conclusion

If you've been hesitating about putting a reverse proxy in front of Apache, hopefully this can help you out. NginX is a lightweight fast server that can handle far more requests than Apache. By putting it on the front-lines you can give your app that extra inch it needs to keep churning out requests.

Let me know if this configuration works for you.

Star Rating using YUI (and Django)

2008-03-04T00:00:00-08:00

I have a very good star rater on reviewsby.us, but it was written using some sloppy prototype code. I wanted to redo star raters in a well thought out manner and I wanted to use YUI. In this particular tutorial I will use Django although it is not a requirement.

For some background information on star raters, see this Yahoo! Design Pattern. Our pattern is more of a join star rater, similar to what's found on Netflix: you see an average rating for a restaurant or dish unless you yourself have rated it.

This was a thought out design decision for our reviewsby.us redesign. Our site is primarily a personal utility that answers the question, "What dishes do I like at a particular restaurant?" If you haven't rated something the website can only offer up an average and you can use that as a decision as to whether you should eat something or not.

If you have eaten something however, that average rating is irrelevant. You don't need fellow meal advisors to tell you that you liked Chicken Makhani, you already know that for yourself. Therefore we show only your rating unless you haven't rated something.

Working backwards

I like to "work backwards" as it were. Meaning, I like to just write the code that I ultimately will use to output a star rater. From there I will work on the supporting code that is necessary. I find by using this strategy, I can keep my code fairly clean and organized.

The template

So ultimately I want this:

{\% star 'mything' 3 4 '/path/to/script' %}

To show up as this:

Unfortunately Django templates doesn't seem to have named attributes for template tags, so I'll need to explain my syntax:

star: is the template tag which we define below
'mything': is an id string we will use for this rater and its associated objects
3: this is the second argument to star, it will be the users current rating, it can also be None
4.1: this is the third argument, it will be the average rating, it can also be None
/path/to/script: is the form that will process our rating

The HTML we want

Another developer had a good approach for handling star ratings and for handling Javascript in general. Create an underlying Javascript-free system, and then let the Javascript make it pretty. This is way to degrade gracefully.

Ultimately, I had my own approach to this problem, I wanted much of the visual lifting to happen on the CSS layer. So, we'll use the following code:

<span class="joint_star_rater"> <form id="rater_restaurant" method="post" action="/restaurant/pizza-luce/rate/"> <fieldset> <legend>Rating</legend> <ul> <li style="width: 100px;" title="5" class="current meta">Current Rating: 5</li> <li title="Poor" class="star_1 star"> <label for="restaurant_rating_1">Poor</label> <input type="radio" name="rating" value="1" id="restaurant_rating_1"/> </li> <li title="Fair" class="star_2 star"> <label for="restaurant_rating_2">Fair</label> <input type="radio" name="rating" value="2" id="restaurant_rating_2"/> </li> <li title="Good" class="star_3 star"> <label for="restaurant_rating_3">Good</label> <input type="radio" name="rating" value="3" id="restaurant_rating_3"/> </li> <li title="Very Good" class="star_4 star"> <label for="restaurant_rating_4">Very Good</label> <input type="radio" name="rating" value="4" id="restaurant_rating_4"/> </li> <li title="Excellent" class="star_5 star"> <label for="restaurant_rating_5">Excellent</label> <input type="radio" name="rating" value="5" id="restaurant_rating_5"/> </li> </ul> </fieldset> <input type="submit" name="rate" value="rate it" class="submit"/> </form> </span>

A couple things to note in our HTML. Our unique string is restaurant. It's got an ID that is as unique as you want: rater_restaurant where restaurant was the first argument to our template tag. We use restaurant to create some other unique IDs as well.

Also, this rating form makes a lot of sense semantically. While this form in its current state is a far cry from some ajaxy goodness, it makes clear sense as to what is going on.

The template tag

Well we know what we want from the HTML side, so let's start coding our star tag:

@register.simple_tag def star(id_string, current, average, path, spanfree=False): meta = None if current != None: meta = """ <li class="current meta" title="%d" style="width:%dpx">Current Rating: %d</li> """ % (int(current), int(current)*20, int(current)) else: meta = """ <li class="average meta" title="%.1f" style="width:%dpx">Average Rating: %.1f</li> """ % (average, average*20,average) stars = '' ratings = ['Poor', 'Fair', 'Good', 'Very Good', 'Excellent'] for i in range(1,6): stars = stars + """ <li class="star_%d star" title="%s"> <label for="%s_rating_%d">%s</label> <input id="%s_rating_%d" type="radio" value="%d" name="rating"/> </li> """ % (i, ratings[i-1],id_string, i, ratings[i-1], id_string, i, i) html = """ <form action="%s" method="post" id="rater_%s"> <fieldset> <legend>Rating</legend> <ul> %s %s </ul> </fieldset> <input type="submit" class="submit" value="rate it" name="rate"/> </form> """ % (path, id_string, meta,stars) if spanfree: return html else: return """<span class="joint_star_rater">%s</span>""" % html return html

The CSS

A lot of work will happen via CSS. The CSS will remove quite a bit of the textual data that can be interpreted graphically with stars.

The strategy we use is to:

fix the UL at a certain width with a background of grey stars
decorate the LI.average and LI.current with repeating stars (blue and orange respectively) with a z-index of 1
decorate the LI.average:hover and LI.current:hover with a transparent background
decorate LI:hover input as a colored in star and a z-index of 2

This might not make sense now, until you see the CSS in full action. Also for the stars we'll use a sprite of 3 stars. A grey defunct star as the default background, a blue star if it's the average rating for an item and an orange star if it's what the user wants.

I use the following sprite:

The following CSS will do some magic:

.joint_star_rater {display:inline-block;} .joint_star_rater ul{width:100px;position:relative;height:20px;background:url(../images/icons/stars.png) repeat-x 0 0} .joint_star_rater li.meta{position:absolute;text-indent:-9999px;display:block;z-index:1;} .joint_star_rater ul:hover li.meta{display:none;} .joint_star_rater li.current{background:url(../images/icons/stars.png) repeat-x 0 -40px} .joint_star_rater li.average{background:url(../images/icons/stars.png) repeat-x 0 -20px} .joint_star_rater li{height:20px;width:20px;position:absolute;text-indent:-9999px;z-index:3;} .joint_star_rater li.star_2{left:20px;} .joint_star_rater li.star_3{left:40px} .joint_star_rater li.star_4{left:60px} .joint_star_rater li.star_5{left:80px} .joint_star_rater li.star_1:hover{width:20px} .joint_star_rater li.star_2:hover{width:40px} .joint_star_rater li.star_3:hover{width:60px} .joint_star_rater li.star_4:hover{width:80px} .joint_star_rater li.star_5:hover{width:100px} .joint_star_rater li.star:hover{background:url(../images/icons/stars.png) repeat-x 0 -40px;z-index:2;left:0;} .joint_star_rater input.submit{display:none;}

The inline-block value for display is not supported very well. I recently switched to Firefox 3 Beta and it renders as expected. Firefox 2 has problems with it. I may revise the CSS later to accommodate it.

The Javascript

The fundamental drawback to the design here, is that it really only works well with the Javascript on. In fact, with the CSS on and Javascript off, this code will not work very well for the end user. This too will be revised in the future.

Our Javascript needs to do something very simple: * extract the star value you clicked on * send it to the server * redraw the stars

It's a very simple operation, but I honestly think other libraries have an advantage to YUI in this regard.¹ Here's some unobtrusive code I came up with:

Note: I intentionally left out irrelevant pieces of code, like the function definition of MA.is_authenticated(), this code isn't meant for cutting and pasting, it's meant for cutting-pasting and then some careful editing.

The callback view

The callback script is what you specify when you call {\% star ... %}. The view I use is as follows:

@login_required def rate(request, slug): MyObject = get_my_object() value = request['value'] MyObject,rate(value); return render_to_response("rating.html", locals(), context_instance=RequestContext(request))

That code is oversimplified... you have to write your own logic as it applies to our site. The rating.html is simply the call to your star tag:

{\% load tags %}
{\% star 'mything' restaurant.current_rating restaurant.average_rating restaurant.get_rating_url 1 %}

Note the 1 at the end. It's a flag to turn off the outer span so we can just insert the guts back into the original span.

Final Thoughts

The star-rater is really a large problem that's hard to tackle in one sitting and quite frankly is not documented well anywhere. The code I've provided is a shadow of the real code I'll be using, but hopefully it's enough to get you started.

I definitely will update my production code to solve a few outstanding issues, as I mentioned above. I'll try to update this tutorial at the same time. If there are questions about the examples given, feel free to ask and I'll attempt to answer.

YUI has the Selector and that might alleviate some problems, but jQuery has very nice selection capabilities. In this future version I keep promising, I'll use the YUI Selector ↩

Unicode or How to deal with f'd up text in Django

2008-02-11T00:00:00-08:00

So I've been going out of my mind trying to figure out why something like:

Pham's would look like Phamâ€™s

After searching as much as I could into django and unicode, I found this article which worked.

As near as my feeble USA-centric "cafes not cafés" understood it, my data was encoded incorrectly, and PHP's mysql ignores encoding and expects UTF-8 whereas django tries to understand the encoding.

Enabling a debug.css in django

2008-02-11T00:00:00-08:00

I like for extra magical stuff to occur when I am in development/debug mode. One of those magical things is a a magic debug.css style sheet.

Django sets a debug variable when the following conditions are met:

django.core.context_processors.debug is listed under TEMPLATE_CONTEXT_PROCESSORS and your IP address (127.0.0.1 if your using the standard development server) is listed under INTERNAL_IPS.

Then in your base.html just add something like this:

For my CSS I like to use the following:

As I am developing my site, if I feel that a block of content isn't fully finished for whatever reason, I mark it with a class of incomplete. This results in a purple color (which I don't use on my site), the purpler it is... the more incomplete the content.

It's like eating that stuff that makes your teeth red if you haven't brushed them enough. Once all the purple's gone the site is ready for production.

Eventually we can use all these hooks into the template to implement a symfony-style debug bar... later.

the magic of django: get_callable

2008-02-08T00:00:00-08:00

symfony is an amazing framework simply because of the uphill battle it has with PHP. There's a lot of behind the scenes magic that ameliorates the frustrations one might have with PHPs lack of syntactical magic.

A lot of work is done to make sure certain things are autoloaded and that certain things behave the way you expect.

I hadn't had a chance to peek under Django's hood too much myself, but it appears that despite the inherent magic of python, there were some much needed additions. My savior for today was get_callable.

I have a very particular vision of interacting with OpenID. Basically I like to decouple as much of the OpenID interactions away from the inner workings of my web site. I do, however, want a hook where I can supply some of my own logic after OpenID has been verified.

It's roughly how I implimented the sfOpenIDPlugin. Luckily this time, I was able to rely on the OpenID 2.1 libraries from OpenIDEnabled.

I did ditch their example code, because I wanted a sexy package. The package isn't quite ready for primetime, but you can have a gander (and yes, I do welcome collaborators and am open to merging my work with the myriad of work that's out there).

Getting to the point

Anyway, get_callable is what allowed me to write my "hook" into my OpenID logic.

In my settings.py I defined:

OPENID_SUCCESS = 'myproject.myapp.views.openid_handler'

The way my OpenID library works, is it does it's business and when it verifies a url belongs to someone it then delivers he request to the OPENID_SUCCESS hook. Here's how:

if settings.OPENID_SUCCESS:
    view = get_callable(settings.OPENID_SUCCESS)
    return view(request, result)

get_callable works some python magic by splitting myproject.myapp.views.openid_handler into the module: myproject.myapp.views and the attribute openid_handler. It uses this to produce a callable object. In the example above I gave it parameters of request and result as they were required by my app.

sfGuardUser to django.contrib.auth.models.User

2008-02-07T00:00:00-08:00

Let's pretend that your assignment was to convert a symfony app that used sfGuardAuth to a django-based system. Wouldn't it be great if someone just gave you a bunch of SQL that you could use to convert the symfony sf_guard_user table to a django auth_user table?

Here you go:

REPLACE INTO 
  auth_user

SELECT 
  p.id AS id, 
  replace(username, 'http://', '') AS username,
  null AS first_name,
  null AS last_name,
  email,
  concat(algorithm,'$',salt,'$',password),
  0 as is_staff,
  1 as is_active,
  0 as is_superuser, 
  last_login,
  u.created_at as date_joined


FROM 
  `sf_guard_user` u, profile p 

WHERE
  p.userid = u.id

Django circular model references

2008-02-03T00:00:00-08:00

I'm used to circular references in my model. Often I do a versioning of an Item with an ItemVersion. Item will link to the latest ItemVersion and ItemVersion will link to the relevant Item.

Here's how you can define your appropriate django models:

Note in the first model, Item, we reference ItemVersion in quotes because ItemVersion is not yet defined.

Why Django Templating is awesome and why I get smarty again

2008-02-01T00:00:00-08:00

I get Smarty thanks to django... yeah, it's weird.

Back to my original comment about templating, smarty really is trying to limit the scope of PHP in a good way. Too often I see a lot of heavy-lifting in the templates. It's so bad it makes my MVC'd brain explode.

Django templates are very limited, based on a philosophy of keeping view logic and only view logic in the templates.

This is what smart tries to do and it's a reasonable solution to the fact that PHP is a templating language with a kitchen sink. It's saying, okay... well let's treat PHP as a programming language, and keep Smarty as the template.

symfony of course says (well not really, or in any official capacity, but would you believe... frameworks talk to me), some PHP is view and some is model/controller. We'll suggest which ones are which, but really we're not getting in the way of making your life a living hell by sticking complicated business logic inline.

Girl Geek Dinner

2008-02-01T00:00:00-08:00

Katie and I went to Girl Geek Dinner. The best thing about being a Geek Husband is having a Geek Wife to bring you as a guest.

The panel talk was pretty good. Featuring:

Irene Au, Director of User Experience, Google
Rashmi Sinha, CEO, SlideShare
Leah Culver, Lead Developer & Co-Founder, Pownce
Sumaya Kazi, Entrepreneur & Social Media Manager, Sun Microsystems
Katherine Barr (moderator), Partner, Mohr Davidow Ventures

What really made the talk was it had a lot of good advice in general. Writing in a blog for example is how a lot of people got their credibility. It's how I got mine, and why I now live in the bay area and ultimately why I work on del.icio.us.

The talk was really reaffirming of my move out here. It also makes me want to code-like-mad.

Following the Django theme... Pownce was written in Django ;)

reusability

2008-01-30T00:00:00-08:00

[tags]django, plugins, apps, projects, symfony[/tags]

A project is a collection of settings for an instance of Django, including database configuration, Django-specific options, and application-specific settings.

The Django Book, Chapter 2:

A few people have been asking for more comparisons between symfony and Django. For me it's a great way of understanding Django and python as well as symfony and PHP.

Reusability is at the core of Django, not an afterthought. The only unique part of an app is the settings and the views. Everything else is an application that can exist independently of your app. It's nice and decoupled.

This wouldn't be impossible to do in symfony. Each module could be designed from the start as a plugin. Complete with its own set of models and default templates. The configuration of a project/app could then make the web app unique.

Right now the bulk of my symfony models are tightly coupled to their apps. It's a little confusing, but there isn't a direct correlation between Django projects, Django apps and symfony Projects, apps and modules. Each kind of overlaps one another.

Templating

2008-01-28T00:00:00-08:00

[tags]smarty, symfony, php, python, django[/tags]

I used to use the Smarty templating system quite heavily in my PHP apps. Then after switching over to symfony, I broke that habit.

PHP is a templating language for better or worse. Smarty was a way of saying that "well things could be easier", but that's against the grain of what PHP fundamentally is.

One frustration of PHP is this. A "pure" PHP file with no HTML looks like this:

PHP is an HTML templating system, by default everything in PHP is just stuff waiting to be sent untouched to the browser. So in essence, PHP web apps aren't really web apps... they are very complicated templates all talking to each other.

So you'll note in the above example, even a pure PHP file must begin with <?php, and signal that we're actually doing code. In fact, to prevent any unintended whitespace being sent to the browser, a PHP only file will by convention omit the closing ?>.

So my shiny new Django framework is a breath of fresh air. There's no <?py tags, it's all just python. And template files are explicitly template files parsed by a template engine built on top of a scripting language.

I can easily see PHP evolving into something similar one day. Where all PHP files are just PHP by default. It would be more in line with the way MVC development is going.

django experiment: day 2 templates, and that's it?

2008-01-28T00:00:00-08:00

[tags]django, symfony[/tags]

[Note: This really does have something to do with symfony.]

So I finished the tutorials on Django, but the Django tutorial is nothing compared to the Askeet Advent Calendar. Askeet was 24 one hour tutorials that go in depth to each nook and cranny of symfony, whereas I still feel like there's got to be more to Django.

I get Templating now, and I really like the flexibility offered in Django. The one major difference is the layout structure.

symfony has this nice structure of defining a general layout, and then all following templates automatically inherit from the default layout (unless otherwise specified). Django on the other hand requires you to explicitly extend other layouts and define regions within those layouts.

I like the symfony style because I inherit almost always from the general template.
The Django style however offers the flexibility of having subsections (which inherit from the base style) which you can inherit from. E.g. you may have a site which has 1. a blog and 2. a gallery. Both the blog and gallery sections share common elements from a main template, but all the pages within blog all have some unique bloggish features that the gallery lacks.

I've got a lot more to learn with Django, but I am enjoying it so far.

I...

~~still have to learn how to create templates.~~
~~want better understanding of the available field types.~~
want to find a good CMS package
want to know if there's the idea of environments in Django (like in symfony)
want to override .save() on my models
want to learn how to have self-referencing models (E.g. a hierarchical category)

python: relative paths

2008-01-27T00:00:00-08:00

So I started yesterday with Django, and I decided I didn't want to futz with creating another mysql database that I'd need to manage, etc. Instead I'll just use sqlite.

I wanted to keep my sqlite database within my project regardless of where I might move my project later. So I did this:

I confused a lot of people on IRC, but it's really quite easy:

__file__ is the filename of the current script, very similar to PHP's __FILE__
os.path.abspath calculates the absolute path, hence the absolute path of the current file
os.path.join does all the nasty business of joining paths together and figuring out what type of slashes are needed, etc.
'data/db.sqlite' is a string

So really all we were doing is creating a relative path, but setting it absolutely.

GeoDjango

2008-01-27T00:00:00-08:00

[tags]geodjango, django, gis[/tags]

I just discovered GeoDjango. The existence of this is awesome.

The GIS layer of reviewsby.us is incredibly complicated, and if I do another project of that magnitude, it'll be nice to know that a framework exists to tackle problems. It looks functional, but not quite documentation complete for usage, but if I endeavor to learn more about GIS, I'm going to dive-in.

django experiment: Day 1 and Model Inheritence

2008-01-27T00:00:00-08:00

[tags]django, inheritance[/tags]

I spent a few hours diving into Django. I knew enough python to get around, and also to be dangerous.

Here's a summary of what I've learned.

Model Inheritence

So far I've learned that ModelInheritence is not quite ready. In other words, you can't define in your models.py something like this:

The model itself will work, and it will save, but things like Page.objects.all() will return empty lists.

Where I'm still shaky

I... * still have to learn how to create templates. * want better understanding of the available field types. * want to find a good CMS plugin * want to know if there's the idea of environments in Django (like in symfony) * want to override .save() on my models * want to learn how to have self-referencing models (E.g. a hierarchical category)

I'm going to keep on plugging along, it's really easy to figure out django.

Dave Dash

Plus Addressing

The wrong validation

The URL issue

Call to Action

Naming things and a recursion

The typical python project

MOZtrosity

So?

So what did we learn?

Choosing a New Web Stack

DjangoCon Testing Tutorial

How we slug at Mozilla

Testing Redis in Django

Faceted Search on Input

Trimming Whitespace in Django Forms

The Perils of One Giant Fixture

Your objects, and all their friends

Test Driven Confidence

Making our tests run thrice as fast

Why speed matters

Testing Sphinx search with TransactionTestCase

Making our own TransactionTestCase

Fast tests are good

λ^2: safely doing class based views in Django

λ

λ λ

django-fixture-magic: Testing issues with real data.

Serializing alone isn't enough.

Too much serial

Retrieving elements in a specific order in Django and mySQL

Django: Model Inheritance or Related Tables wrt AMO

DjangoCon wrapup

Day one

Keynote - Avi Bryant

UR doing it wrong - James Bennet

Testing - Eric Holscher

Deploying Django -

Day 2

Keynote - Ian Bicking

Using Django in Non-standard ways - Eric Florenzano

Real-time web and other Buzzwords - Chris Wanstrath

Pluggable, Reusable Django Apps: A Use Case and Proposed Solution - Shawn Rider and Nowell Strite

Day 3

Keynote - Ted Leung - Sun

Scaling Django Mike Malone

Gearman - working later - Chris Heisel

Resolving Django dumpdata errors

Automating Django Redirects

A stitch in Fabric saves time

The task

Directory Structure

Subversion

The fabfile.py

Final Thoughts

Resizing Image on upload in Django

The Template

Resizing an image

Generating a filename

Saving the file via the model

All together now

Database versus files for Images

Versioning Django Models

How I think of versions

The Django implementation of RestaurantVersion

The Django implementation of Restaurant

Proxy attributes to the corresponding RestaurantVersion attributes

Generate a new RestaurantVersion on demand

Conclusion and Drawbacks

Mimic propel's update_at and created_at in Django models

Django models: saving markdown

Custom error messages for Django forms

sfGuardUser -> django.contrib.auth

Django Admin and Cookies

Why code rewrites are often coupled with redesigns

nginx proxying to apache

Apache

NginX

Conclusion

Star Rating using YUI (and Django)

Testing Sphinx search with `TransactionTestCase`

Making our own `TransactionTestCase`

The `fabfile.py`

The Django implementation of `RestaurantVersion`

The Django implementation of `Restaurant`

Proxy attributes to the corresponding `RestaurantVersion` attributes

Generate a new `RestaurantVersion` on demand