Pylons vs Pyramid

If you weren't already aware of it, the Pylons project has recently sprouted an all-new Python Web framework called Pyramid. Pyramid is kinda like Pylons 2.0 - it is a completely new codebase with no code-level backwards compatibility. It is actually based on repoz.bfg. However, many of the concepts are very similar to the older Pylons 1.x. Pyramid is where all the new development is happening, and it has fantastic code test coverage and documentation.

You can learn more about Pyramid in the Pyramid FAQ.

What Is Akhet

On its own, Pyramid is just a framework - a set of libraries you can use. Projects are most easily started from a template. A number of different templates are included, offering different persistence options, URL mappers and session implementations.

Akhet is a Pyramid project template (AKA scaffold) which tries to give you a more Pylons 1.x-like environment. Personally, I'm much more familiar with Pylons than Pyramid and so Akhet is a great place to start.

It is well worth your while reading the Akhet documentation. However, the subject of this post, working with MongoDB, takes advantage of a few changes I have made myself to my Akhet fork on BitBucket. I am hoping Mike Orr will accept my changes upstream, but for now you will need to build the egg from my sources. I will explain how in the next section.

Starting A Pyramid Project With MongoDB

First you need to create a virtual env for your project. Depending on your platform, you should be able to install the virtualenv tool through a package manager (sudo port install virtualenv-2.7 in Mac Ports, for example).

Install Pyramid

Once you have the virtual env tool installed, create the virtualenv, install Pyramid and its dependencies into it:

$ mkdir ~/projects/ahket-example
$ virtualenv --no-site-packages myenv
$ cd myenv
$ source bin/activate
$ easy_install pyramid

Install Akhet

Now you have Pyramid and all its dependencies installed. However, you still need Akhet and its dependencies like PyMongo etc. We are going to use my fork of Akhet, because it has MongoDB support at the moment.

In a new terminal, type the following (you will need Mercurial installed to fetch the sources from BitBucket):

$ cd ~/projects/akhet-example
$ hg clone https://bitbucket.org/niallohiggins/akhet
$ cd akhet
$ python setup.py sdist

Now you have the Akhet with MongoDB support module available. Install it in your virtual env (go back to original terminal, or re-source ~/projects/akhet-example/myenv/bin/activate):

$ pip install ~/projects/akhet-example/akhet/dist/Akhet-1.0.1.tar.gz

Create Pyramid Project with Akhet and MongoDB Support

Okay, great - you can create your Pyramid project now! We shall call it "mongofu".

$ cd ~/projects/akhet-example
$ paster create -t akhet mongofu
Selected and implied templates:
  Akhet#akhet  A Pylons-like Pyramid project

Enter project name: mongofu
Variables:
  egg:      mongofu
  package:  mongofu
  project:  mongofu
Enter sqlalchemy (Include SQLAlchemy configuration? (y/n)) [True]: n
Enter mongodb (Include MongoDB configuration? (y/n)) [False]: y

Install Akhet Dependencies

You are just about ready to go. You need to make sure your virtual environment has all the dependencies needed by your mongofu project (e.g. the PyMongo driver). To do so run the following:

$ cd mongofu ; python setup.py develop

Assuming all went well, you should now have a directory called "mongofu" with a whole bunch of stuff in it. The default configuration files tell Pyramid to connect to a MongoDB server on localhost, and a database called mydb. If you need to change that, simply edit the mongodb.url and mongodb.db_name settings in the INI-files.

Start The Development Server

Now try starting up a development server:

$ paster serve development.ini

Starting server in PID 6020.
serving on http://127.0.0.1:5000

Talking to MongoDB with Pyramid/Akhet

Now you are ready to write some MongoDB queries! Edit the file mongofu/handlers/main.py.
In the Main class, we can add a simple query to the "index" action method, referencing the request.db property - which is a handle to our MongoDB database:

class Main(base.Handler):
    @action(renderer="index.html")
    def index(self):
        # Do some logging.
        log.debug("testing logging; entered Main.index()")
        # MongoDB Query
        # can just as easily write request.db.mycollection.find()
        records = request.db['mycollection'].find()
        # Push a flash message if query param 'flash' is non-empty.
        if self.request.params.get("flash"):
            import random
            num = random.randint(0, 999999)
            message = "Random number of the day is:  %s." % num
            self.request.session.flash(message)
            # Normally you'd redirect at this point but we have nothing to
            # redirect to.
        # Return a dict of template variables for the renderer.
        return {"project":"mongofu"}

As you can see, we specify a collection in that database to query against. We're not doing anything with the results, but that is enough to demonstrate how to talk to MongoDB with Pyramid and Akhet.

I've been playing with the recently-released HTTP API for accessing the Best Buy product catalog. While its a little strange to use at first, its actually pretty useful. One of the things I am interested in is online retail, specifically how to make Internet shopping easier. Lets imagine I am looking for information on a particular digital camera - the Nikon Coolpix S210.

A Python skeleton
First, lets get our little Python test harness together. Also, you are going to need your own Best Buy Remix API key. Here is a skeletal Python HTTP client:

import httplib

API_KEY=''

QUERY="/v1/products(name=Coolpix*&modelNumber=S210)"
OPTS="?sort=name.desc&show=all&format=json"
c = httplib.HTTPConnection('api.remix.bestbuy.com')
c.request('GET', "%s%s&apiKey=%s" %(QUERY, OPTS, API_KEY))
r = c.getresponse()
data = r.read()
print data

Our first Best Buy query
Now lets try to write a sample query for the Nikon Coolpix S210. Although the Best Buy Remix API docs are a bit sparse, we can guess that items must have an attribute called 'name'. In fact they do! So lets try searching for the camera by name.

# Same code as above, but we change the value of QUERY:
QUERY="/v1/products(name=Nikon Coolpix S210)"

$ python bb.py
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>400 - Bad Request</title>
 </head>
 <body>
  <h1>400 - Bad Request</h1>
 </body>
</html>

Gimme everything!
It turns out that Best Buy don't name their products in the most intuitive way. Lets try a wildcard on just `Coolpix' instead:

# Same code as above, but we change the value of QUERY:
QUERY="/v1/products(name=Coolpix*)"

This time, we are going to get tons of data back, in JSON format. Best Buy remix defaults to XML, but I prefer JSON so I added the format=json parameter to the query. Ok, so now we have an overwhelming amount of data on Coolpix cameras - but we really just want information for the S210.

Best Buy's quirky product schema
Well, there is a solution. Best Buy don't store the model number in the `name' attribute - instead they store it in a separate `modelNumber' attribute. If we query for name=Coolpix* AND modelNumber=S210, we should get the expected result, finally:

# Same code as above, but we change the value of QUERY:
QUERY="/v1/products(name=Coolpix*&modelNumber=S210)"

Et voila! Now Best Buy gives us back all the information it has about the Nikon Coolpix S210. This is pretty detailed stuff, including all those details like compatible memory formats, digital zoom, along with the price and availability. Very cool! Just for kicks, lets show the whole script to send a query to Best Buy, parse the JSON response, and finally print the price:

import httplib
import json

API_KEY=''
QUERY="/v1/products(name=Coolpix*&modelNumber=S210)"
OPTS="?sort=name.desc&show=all&format=json"
c = httplib.HTTPConnection('api.remix.bestbuy.com')
c.request('GET', "%s%s&apiKey=%s" %(QUERY, OPTS, API_KEY))
r = c.getresponse()
data = r.read()

camera_info = json.loads(data)

print "price: %s"%(camera_info['products'][0]['regularPrice'])

$ python bb.py
price: 89.99

SQLObject vs. SQLAlchemy
I actually much prefer SQLAlchemy to SQLObject. SQLAlchemy has a more explicit divide between its various components - you aren't forced to use the ORM stuff if you don't want to. It can be used just as a handy database abstraction layer with programmatic SQL and connection pooling and so on if you want. And if you truly want to go the full ORM mapping route, they provide that too. For truly tricky things, SQLAlchemy will be happy to provide you with a DB-API 2.0 cursor so that you can execute whatever custom SQL you wish.

Monolithic vs. modular
This is my main problem with SQLObject - its very difficult to figure out how to get at the underlying database connection. I don't know how its possible to use the connection pooling and programmatic SQL builder without using the ORM but perhaps it is doable. The documentation for SQLObject is far inferior to the documentation of SQLAlchemy I'm sorry to say. Just try to figure out how to use transactions reliably with SQLObject! Even when I managed to put together some code which according to the documentation should work, SQLObject decided to interleave the actions in separate transactions. With SQLAlchemy I never had this problem.

Getting at the cursor
While seemingly undocumented, it is in fact possible to get the underlying driver's connection object, and from there grab a DB-API cursor. The pattern is:

# Set up the SQLObject connection as usual
connection = connectionForURI('sqlite:/:memory:')
# Grab the database connection
dbobj = connection.getConnection()
# Get a cursor from the low-level driver
cursor = dbobj.cursor()
# 
cursor.close()

import sqlite3
def open_db():
    conn = sqlite3.connect(config['sqlite_db_file'])
    return conn

[app:main]
use = egg:MyApp
full_stack = true
cache_dir = %(here)s/data
beaker.session.key = mykey
beaker.session.secret = somesecret
sqlite_db_file = /path/to/your/sqlite.db

var o = { type:'/music/album' };
$(id).freebaseSuggest({ac_param:o});
$(id).bind("fb-select", suggest_cb);

var o = { type:'/music/album', 
          type:'/film/film',
          type:'/tv/tv_program' };
$(id).freebaseSuggest({ac_param:o});
$(id).bind("fb-select", suggest_cb);

[{ 
    'id': /* id of topic to find image for */, 
    '/common/topic/image': [{}]
}]

if (acre.environ.path_info == "/") {
    acre.start_response(400);
    acre.exit();
}
var query = [{ "id" : acre.environ.path_info, "/common/topic/image":[{}]}];
var r = acre.freebase.MqlRead(query);
acre.start_response(302, {
        'Location': acre.freebase.imgurl(r.result[0]['/common/topic/image'][0]['id'].substring(1))
});