I was at the MongoSF 2011 conference in May. Very much enjoyed the 1-day conference, made some great connections and the 10gen folks hooked me up with O'Reilly to work on a book "MongoDB and Python", which I hope to have out in a couple of months.

My talk: MongoDB with Python, Pylons, and Pyramid

I was very happy to be invited to speak at MongoSF. My talk covers using Python with MongoDB, goes into a little bit of detail on the Web frameworks Pylons and Pyramid. Watch the video.

Pylons vs Pyramid

If you weren't already aware of it, the Pylons project has recently sprouted an all-new Python Web framework called Pyramid. Pyramid is kinda like Pylons 2.0 - it is a completely new codebase with no code-level backwards compatibility. It is actually based on repoz.bfg. However, many of the concepts are very similar to the older Pylons 1.x. Pyramid is where all the new development is happening, and it has fantastic code test coverage and documentation.

You can learn more about Pyramid in the Pyramid FAQ.

What Is Akhet

On its own, Pyramid is just a framework - a set of libraries you can use. Projects are most easily started from a template. A number of different templates are included, offering different persistence options, URL mappers and session implementations.

Akhet is a Pyramid project template (AKA scaffold) which tries to give you a more Pylons 1.x-like environment. Personally, I'm much more familiar with Pylons than Pyramid and so Akhet is a great place to start.

It is well worth your while reading the Akhet documentation. However, the subject of this post, working with MongoDB, takes advantage of a few changes I have made myself to my Akhet fork on BitBucket. I am hoping Mike Orr will accept my changes upstream, but for now you will need to build the egg from my sources. I will explain how in the next section.

Starting A Pyramid Project With MongoDB

First you need to create a virtual env for your project. Depending on your platform, you should be able to install the virtualenv tool through a package manager (sudo port install virtualenv-2.7 in Mac Ports, for example).

Install Pyramid

Once you have the virtual env tool installed, create the virtualenv, install Pyramid and its dependencies into it:

$ mkdir ~/projects/ahket-example
$ virtualenv --no-site-packages myenv
$ cd myenv
$ source bin/activate
$ easy_install pyramid

Install Akhet

Now you have Pyramid and all its dependencies installed. However, you still need Akhet and its dependencies like PyMongo etc. We are going to use my fork of Akhet, because it has MongoDB support at the moment.

In a new terminal, type the following (you will need Mercurial installed to fetch the sources from BitBucket):

$ cd ~/projects/akhet-example
$ hg clone https://bitbucket.org/niallohiggins/akhet
$ cd akhet
$ python setup.py sdist

Now you have the Akhet with MongoDB support module available. Install it in your virtual env (go back to original terminal, or re-source ~/projects/akhet-example/myenv/bin/activate):

$ pip install ~/projects/akhet-example/akhet/dist/Akhet-1.0.1.tar.gz

Create Pyramid Project with Akhet and MongoDB Support

Okay, great - you can create your Pyramid project now! We shall call it "mongofu".

$ cd ~/projects/akhet-example
$ paster create -t akhet mongofu
Selected and implied templates:
  Akhet#akhet  A Pylons-like Pyramid project

Enter project name: mongofu
Variables:
  egg:      mongofu
  package:  mongofu
  project:  mongofu
Enter sqlalchemy (Include SQLAlchemy configuration? (y/n)) [True]: n
Enter mongodb (Include MongoDB configuration? (y/n)) [False]: y

Install Akhet Dependencies

You are just about ready to go. You need to make sure your virtual environment has all the dependencies needed by your mongofu project (e.g. the PyMongo driver). To do so run the following:

$ cd mongofu ; python setup.py develop

Assuming all went well, you should now have a directory called "mongofu" with a whole bunch of stuff in it. The default configuration files tell Pyramid to connect to a MongoDB server on localhost, and a database called mydb. If you need to change that, simply edit the mongodb.url and mongodb.db_name settings in the INI-files.

Start The Development Server

Now try starting up a development server:

$ paster serve development.ini

Starting server in PID 6020.
serving on http://127.0.0.1:5000

Talking to MongoDB with Pyramid/Akhet

Now you are ready to write some MongoDB queries! Edit the file mongofu/handlers/main.py.
In the Main class, we can add a simple query to the "index" action method, referencing the request.db property - which is a handle to our MongoDB database:

class Main(base.Handler):
    @action(renderer="index.html")
    def index(self):
        # Do some logging.
        log.debug("testing logging; entered Main.index()")
        # MongoDB Query
        # can just as easily write request.db.mycollection.find()
        records = request.db['mycollection'].find()
        # Push a flash message if query param 'flash' is non-empty.
        if self.request.params.get("flash"):
            import random
            num = random.randint(0, 999999)
            message = "Random number of the day is:  %s." % num
            self.request.session.flash(message)
            # Normally you'd redirect at this point but we have nothing to
            # redirect to.
        # Return a dict of template variables for the renderer.
        return {"project":"mongofu"}

As you can see, we specify a collection in that database to query against. We're not doing anything with the results, but that is enough to demonstrate how to talk to MongoDB with Pyramid and Akhet.

I was at the awesome PyCon 2011 last week, where I had a fantastic time. Really great conference - tons of incredible talks and wonderful conversation. Definitely the best technical conference I've attended. Everyone I met was super friendly, down-to-earth and just all-around fun.

Had the pleasure to hang out with the Pylons/Pyramid crew who are a great bunch. I will certainly be going in 2012, when the event is scheduled to be held in Santa Clara. Plan to stay for the sprints next time!

My talk: MongoDB + Pylons at Catch.com

I was very happy to be able to present a talk on some of my work at Catch.com, where I designed and implemented the platform backend using Pylons and MongoDB. The video of my talk is embedded below:

MasterSlaveConnection: Works in Pymongo 1.6
In my previous article Python, MongoDB and Pylons, Connection handles and all that lark I described the code and configuration we use at Snaptic for working with Pylons and MongoDB. We use the MasterSlaveConnection class to automatically handle read/write splitting, because we expect to be running a single master/multiple slaves configuration in production.

Broken in Pymongo 1.7
Well, yesterday I upgraded from Pymongo 1.6 to Pymongo 1.7 (just released) and found that unfortunately all the MongoDB stuff breaks, throwing TypeError exceptions when issuing even a simple find_one() query. You will see an error like this one: TypeError: 'Database' object is not callable. If you meant to call the 'document_class' method on a 'Collection' object it is failing because no such method exists.

But fixed in tip
After talking with Mike Dirolf (Pymongo author) on the MongoDB IRC channel, he came up with a fix within a few minutes. Although I don't think its in a release yet, if you grab this commit to Pymongo from github, MasterSlaveConnection should work again. He also filed this bug to come up with a more general solution. Cheers to Mike for his super quick turnaround on fixing the bug!

I've been doing a bunch of hacking with Pylons and MongoDB recently for some backend stuff at Snaptic. Right now we are using Paster as the webserver and the Pymongo driver. This all works fine and is pretty straightforward to set up - but there are a couple of subtleties.

If you've ever used Pylons with SQLAlchemy, then you've probably noticed the integration is quite good. All of the glue to get access via the global Session object is done for you. With MongoDB and Pylons, the integration isn't quite there yet. You have to set this up yourself.

MongoDB doesn't give you the same kind of single-server durability guarantees that a RDBMS like MySQL or PostgreSQL does, so you pretty much have to use some kind of replication in production. I'm expecting a master/slave configuration, and Pymongo has native support for read/write splitting (writes go to the master, reads go to the slave(s)), so I've used that in my Pylons integration code.

The basic idea is that you define a list of masters and slaves in your Pylons config file, and some code in lib/app_globals.py sets up a global handle to the connection object.

Here is what I have at the moment, in the __init__ method in lib/app_globals.py:

# in production, we will write to the master and read from the slaves
mongo.master_host = localhost
mongo.master_port = 27017
# to supply multiple slaves, use comma as the separator.
mongo.slave_hosts = localhost
mongo.slave_ports = 27017

mongo.db = foo

db = self._py_object.app_globals.db
# now I can go wild and run:
# db.fooCollection.find_one()

db_conn = self._py_object.app_globals.db_conn
# now I can run:
# db_conn.end_request() if I want.

The Pieces
Autocomplete is not too hard to understand. It is comprised of two pieces:

• Client-side javascript
• Backend web service

Autocomplete JavaScript Widget
The client-side JavaScript code is the thing that watches user input to a given text-entry field, and sends queries to the backend web service. If the backend has some suggestions for the user, the JavaScript then displays those hints and lets the user pick one.

Although not hard to understand, the display code and query timing bits of the client-side component make it not utterly trivial to implement. For this reason, we decided to use an existing jQuery autocomplete widget, instead of writing our own.

Choosing one was a little bit confusing, because there are around four or five distinct jQuery autocomplete widgets floating around. It took some investigation to find one which fit our needs. We ended up picking Ajax Autocomplete for jQuery by Tomas Kirda.

We chose this one because it seemed to be the only widget which easily enabled you to pass arbitrary metadata to the client JavaScript from the backend, in addition to mere text completion. So, if you need to pass along IDs or other values with your suggestions, I recommend this widget instead of some of the simpler ones.

$("#sometextfield").autocomplete({
            serviceUrl:'/api/product_autocomplete',
            onSelect: function(val, data) {
              /* Handle data here */
            },
});

Backend Web Service
Writing an autocomplete web service is pretty simple. Your entrypoint is going to accept a string of text (query) and return a set of results to be displayed to the user by the autocomplete widget. Depending on your usage, you may also wish to include some metadata along with your results - for example, the database ID of each completion, or something like that.

The service is especially trivial if you are using a database which supports an analog to the SQL LIKE/ILIKE operator which does basic wild-card text matching. I believe that all databases supported by SQLAlchemy will have this feature.

Although I happened to be using SQLAlchemy and a fairly traditional RDBMS (PostgreSQL) for snagmachine.com, something similar should be quite possible with Tokyo Tyrant and the like.

We are using Pylons and SQLAlchemy for snagmachine.com, but again, it shouldn't be much more complicated with some other web framework:

    @rest.restrict('GET')
    @jsonify
    def tag_autocomplete(self):
        if 'query' not in request.params:
            abort(400)
        fragment = request.params['query']
        keywords = fragment.split()
        searchstring = "%%".join(keywords)
        searchstring = '%%%s%%' %(searchstring)
        try:
            ac_q = Session.query(Tag)
            res = ac_q.filter(Tag.name.ilike(searchstring)).limit(10)
            return dict(query=fragment,
                    suggestions=[r.name for r in res],
                    data=["%s" %(r.name) for r in res])
        except NoResultFound:
            return dict(query=fragment, suggestions=[], data=[])

tag_table = sa.Table("tag", meta.metadata,
    sa.Column("id", types.Integer, sa.schema.Sequence('taq_seq_id',
        optional=True), primary_key=True),
    sa.Column("name", types.Unicode(50), nullable=False, unique=True),
    sa.Column("extra", types.String),
)
class Tag(object):
    pass
orm.mapper(Tag, tag_table)

We also use the Pylons rest decorator and the Pylons jsonify decorator for convenience.

• Use the ilike operator
• Use wildcards at the beginning and end of the string
• Replace whitespace with wildcards

We've found this mode to give us the best user experience, however there are performance implications. PostgreSQL at least can only utilise text indexes for LIKE, and furthermore only if the wildcards are suffixes [This email from Tom Lane has the details].

While using the index does yield about an order of magnitude difference in query response time, we are talking about 0.1 ms vs 1.0 ms with our dataset. For our use case, this is perfectly acceptable!

So, thats pretty much everything there is to it. Hope this article helps!

class ImageController(BaseController):

    def index(self, id):
        ''' Stream local image contents to client '''
        try:
            imgf = open("%s/%s" %(config['image_data_dir'],
                os.path.basename(id)), 'r')
        except IOError:
            abort(404)
        def stream_img():
            chunk = imgf.read(1024)
            while chunk:
                yield chunk
                chunk = imgf.read(1024)
            imgf.close()
        return stream_img()

import httlib
import json
import logging
import socket
import time
import urllib

SEARCH_HOST="search.twitter.com"
SEARCH_PATH="/search.json"


class TagCrawler(object):
    ''' Crawl twitter search API for matches to specified tag.  Use since_id to
    hopefully not submit the same message twice.  However, bug reports indicate
    since_id is not always reliable, and so we probably want to de-dup ourselves
    at some level '''

    def __init__(self, max_id, tag, interval):
        self.max_id = max_id
        self.tag = tag
        self.interval = interval
        
    def search(self):
        c = httplib.HTTPConnection(SEARCH_HOST)
        params = {'q' : self.tag}
        if self.max_id is not None:
            params['since_id'] = self.max_id
        path = "%s?%s" %(SEARCH_PATH, urllib.urlencode(params))
        try:
            c.request('GET', path)
            r = c.getresponse()
            data = r.read()
            c.close()
            try:
                result = json.loads(data)
            except ValueError:
                return None
            if 'results' not in result:
                return None
            self.max_id = result['max_id']
            return result['results']
        except (httplib.HTTPException, socket.error, socket.timeout), e:
            logging.error("search() error: %s" %(e))
            return None

    def loop(self):
        while True:
            logging.info("Starting search")
            data = self.search()
            if data:
                logging.info("%d new result(s)" %(len(data)))
                self.submit(data)
            else:
                logging.info("No new results")
            logging.info("Search complete sleeping for %d seconds"
                    %(self.interval))
            time.sleep(float(self.interval))

    def submit(self, data):
        pass

June meet-up
Anyway, the first meet-up went extremely well - we had 15 people show up, which was a perfect number for the space. Shannon -jj Behrens gave an excellent talk on building RESTful Web services and Marius Eriksen - in fact a colleague from the OpenBSD project - gave an awesome talk on GeoDjango. Slides for both talks are online, of course.

July meet-up
Metaweb Technologies presenting a comparison of Django and Pylons. Then we have Alec Flett, another Metaweb'er, speaking about all the issues involved in scaling Python web applications.

Check it out
If you are interested in checking out the event, its July 28th, 6pm @ SF Main Public Library’s Stong Room. Full details can be found at pywebsf.org. Or if you are interested in giving a talk, just let me know.

I read today that Turbo Gears 2.0 has been released - at long last! I used Turbo Gears 1 briefly in 2007 for a small project then switched to Pylons.

Pylons is pretty neat because its really a framework for building a framework. You can pick and choose WSGI middleware and slot it all together with whatever templating engine or database abstraction layer you like. Pylons just gives you most of the glue you'd need - stuff like unit and functional test harnesses, request routing, caching, and various handy decorators. The Pylons approach has some disadvantages however, since its not quite as well integrated as say Django, which has more of a one-size-fits-all, monolithic approach. Pylons also has excellent documentation.

Its interesting that Turbo Gears 2.0 is built on top of Pylons. It seems to aim to provide you with a more consistent out-of-the-box solution - they standardise on one templating language (Genshi), one database abstraction layer (SQLAlchemy), etc.

It will be interesting to see where Turbo Gears goes from here. It could be quite compelling if the community catches on, but it seems to me that its a little late to the game. I'm not certain what jumping to Turbo Gears 2.0 from Pylons would buy me at this point, and I'd imagine many people feel the same way.