Python, MongoDB and Pylons – Connection handles and all that lark
I’ve been doing a bunch of hacking with Pylons and MongoDB recently for some backend stuff at Snaptic. Right now we are using Paster as the webserver and the Pymongo driver. This all works fine and is pretty straightforward to set up – but there are a couple of subtleties.
MongoDB vs SQLAlchemy
If you’ve ever used Pylons with SQLAlchemy, then you’ve probably noticed the integration is quite good. All of the glue to get access via the global Session object is done for you. With MongoDB and Pylons, the integration isn’t quite there yet. You have to set this up yourself.
Gotta have some replication
MongoDB doesn’t give you the same kind of single-server durability guarantees that a RDBMS like MySQL or PostgreSQL does, so you pretty much have to use some kind of replication in production. I’m expecting a master/slave configuration, and Pymongo has native support for read/write splitting (writes go to the master, reads go to the slave(s)), so I’ve used that in my Pylons integration code.
Getting a handle to the database
The basic idea is that you define a list of masters and slaves in your Pylons config file, and some code in lib/app_globals.py sets up a global handle to the connection object.
Here is what I have at the moment, in the __init__ method in lib/app_globals.py:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | # ... non-pymongo imports omitted from pymongo import Connection from pymongo.master_slave_connection import MasterSlaveConnection from pymongo.errors import ConnectionFailure # ... class definition, __init__ method etc omitted for brevity ... # assumption is a master/slave setup for initial production deployment. # writes should go to the master # reads should go to the slave(s) # with sharding, this may change. def make_conn(host, port, database, username=None, password=None, slave_okay=False): try: conn = Connection(host, port, slave_okay=slave_okay) except ConnectionFailure: raise Exception('Unable to connect to MongoDB') if username and password: auth = conn.authenticate(username, password) if not auth: raise Exception('Authentication to MongoDB failed') return conn username = None password = None if 'mongo.username' and 'mongo.password' in config: username = config['mongo.username'] password = config['mongo.password'] master_conn = make_conn(config['mongo.master_host'], int(config['mongo.master_port']), username, password) slaves = [] for i, slave_host in enumerate(config['mongo.slave_hosts'].split(',')): slave_port = int(config['mongo.slave_ports'].split(',')[i]) slave_conn = make_conn(slave_host.strip(), slave_port, username, password, slave_okay=True) slaves.append(slave_conn) self.db_conn = MasterSlaveConnection(master_conn, slaves) self.db = self.db_conn[config['mongo.db']] |
Then in development.ini I have:
1 2 3 4 5 6 7 8 | # in production, we will write to the master and read from the slaves mongo.master_host = localhost mongo.master_port = 27017 # to supply multiple slaves, use comma as the separator. mongo.slave_hosts = localhost mongo.slave_ports = 27017 mongo.db = foo |
Now, to get a handle to the db object from a controller context, you just do the following:
1 2 3 | db = self._py_object.app_globals.db # now I can go wild and run: # db.fooCollection.find_one() |
If you need a handle to the connection object instead, no problem:
1 2 3 | db_conn = self._py_object.app_globals.db_conn # now I can run: # db_conn.end_request() if I want. |
There might be more elegant ways to do this, but this seems to work fine for us. I will write about connection pooling and end_request() next time. Feel free to comment if I left anything out or you have questions!
Thanks for this post, it’s very helpful!
I just wanted to point out one thing though that could use clarification: Your first code block is labeled with “Here is what I have at the moment, in the __init__ method in lib/app_globals.py:” but in that code block the comments say that the __init__ method and class definition are omitted for brevity.
So is all of this code actually IN the __init__ method and you just leave out the declaration of the method/class or is all of this code just left outside the class completely (as suggested by the indentation level)? I think it’d be more helpful if this was either made clear by the text before the code block, or maybe if all the code was included for clarity.
Thanks again for the post!
Yes you place the code in the __init__() method.