tmux, a BSD alternative to GNU Screen

June 04, 2009 at 08:25 PM | categories: Technical, C, UNIX | View Comments |

I started using tmux today. It's a terminal multiplexer / task switcher for UNIX-likes, very much in the same vein as GNU Screen. However, it's a from-scratch implementation, designed to be clean, sane and easy to configure. The more liberal 3-clause BSD license is a plus also, since it means that OpenBSD has been able to integrate it into the source tree, so that it's available out of the box.

Comparison with GNU Screen
I've been a heavy screen user for many years - almost all my work is done on remote screen sessions. However, screen configuration has always been essentially black magic to me. For this reason, tmux and its nice manual page is a breath of fresh air. `tmux list-commands' is very straight forward and easy to grok. Furthermore, I like that everything in tmux is scriptable from the command line - you can run commands like `tmux resize-pane-up -t comms' to resize the pane on a session called 'comms'.

The other thing I really like about tmux is its default status bar. Some people might hate this, but I find it very useful to have a clock and a list of windows along with the process executing in them. This took quite some work to set up to my liking in GNU screen, but the default in tmux is great.

My config
One thing I don't much like is the default of C-b as the 'prefix' command. I suppose this makes some sense, since the author doesn't want to clobber GNU screen key bindings. Perhaps he will consider changing it to C-a, like in GNU screen, in the future. In any case, this isn't hard to change. Also, I am constantly using C-a C-a to switch back to the previous window - the default for this action in tmux is C-b l. Much less friendly in my opinion - of course, it's also easy to change!

So here are the contents of my $HOME/.tmux.conf:

set -g prefix C-a
bind-key C-a last-window

Getting tmux
I'm sure that packages exist for most operating systems. You can grab the source from http://tmux.sourceforge.net/. On OpenBSD, you can simply run `pkg_add -i tmux' to get the binary on your system.

UPDATE
Since OpenBSD 4.6, tmux is part of the base system. This means that if you are running OpenBSD 4.6 or later, you don't need to install any packages in order to get tmux.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

Turbo Gears 2.0 Released

May 27, 2009 at 06:30 PM | categories: Technical, Python | View Comments |

Turbo Gears 2.0

I read today that Turbo Gears 2.0 has been released - at long last! I used Turbo Gears 1 briefly in 2007 for a small project then switched to Pylons.

Pylons is pretty neat because its really a framework for building a framework. You can pick and choose WSGI middleware and slot it all together with whatever templating engine or database abstraction layer you like. Pylons just gives you most of the glue you'd need - stuff like unit and functional test harnesses, request routing, caching, and various handy decorators. The Pylons approach has some disadvantages however, since its not quite as well integrated as say Django, which has more of a one-size-fits-all, monolithic approach. Pylons also has excellent documentation.

Its interesting that Turbo Gears 2.0 is built on top of Pylons. It seems to aim to provide you with a more consistent out-of-the-box solution - they standardise on one templating language (Genshi), one database abstraction layer (SQLAlchemy), etc.

It will be interesting to see where Turbo Gears goes from here. It could be quite compelling if the community catches on, but it seems to me that its a little late to the game. I'm not certain what jumping to Turbo Gears 2.0 from Pylons would buy me at this point, and I'd imagine many people feel the same way.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

I've been playing with the recently-released HTTP API for accessing the Best Buy product catalog. While its a little strange to use at first, its actually pretty useful. One of the things I am interested in is online retail, specifically how to make Internet shopping easier. Lets imagine I am looking for information on a particular digital camera - the Nikon Coolpix S210.

A Python skeleton
First, lets get our little Python test harness together. Also, you are going to need your own Best Buy Remix API key. Here is a skeletal Python HTTP client:

import httplib

API_KEY=''

QUERY="/v1/products(name=Coolpix*&modelNumber=S210)"
OPTS="?sort=name.desc&show=all&format=json"
c = httplib.HTTPConnection('api.remix.bestbuy.com')
c.request('GET', "%s%s&apiKey=%s" %(QUERY, OPTS, API_KEY))
r = c.getresponse()
data = r.read()
print data
Save the above to a file like bb.py.

Our first Best Buy query
Now lets try to write a sample query for the Nikon Coolpix S210. Although the Best Buy Remix API docs are a bit sparse, we can guess that items must have an attribute called 'name'. In fact they do! So lets try searching for the camera by name.

# Same code as above, but we change the value of QUERY:
QUERY="/v1/products(name=Nikon Coolpix S210)"
Looks pretty reasonable. Unfortunately, Best Buy is going to give us back a 400 error:

$ python bb.py
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>400 - Bad Request</title>
 </head>
 <body>
  <h1>400 - Bad Request</h1>
 </body>
</html>

Gimme everything!
It turns out that Best Buy don't name their products in the most intuitive way. Lets try a wildcard on just `Coolpix' instead:

# Same code as above, but we change the value of QUERY:
QUERY="/v1/products(name=Coolpix*)"

This time, we are going to get tons of data back, in JSON format. Best Buy remix defaults to XML, but I prefer JSON so I added the format=json parameter to the query. Ok, so now we have an overwhelming amount of data on Coolpix cameras - but we really just want information for the S210.

Best Buy's quirky product schema
Well, there is a solution. Best Buy don't store the model number in the `name' attribute - instead they store it in a separate `modelNumber' attribute. If we query for name=Coolpix* AND modelNumber=S210, we should get the expected result, finally:

# Same code as above, but we change the value of QUERY:
QUERY="/v1/products(name=Coolpix*&modelNumber=S210)"

Et voila! Now Best Buy gives us back all the information it has about the Nikon Coolpix S210. This is pretty detailed stuff, including all those details like compatible memory formats, digital zoom, along with the price and availability. Very cool! Just for kicks, lets show the whole script to send a query to Best Buy, parse the JSON response, and finally print the price:

import httplib
import json

API_KEY=''
QUERY="/v1/products(name=Coolpix*&modelNumber=S210)"
OPTS="?sort=name.desc&show=all&format=json"
c = httplib.HTTPConnection('api.remix.bestbuy.com')
c.request('GET', "%s%s&apiKey=%s" %(QUERY, OPTS, API_KEY))
r = c.getresponse()
data = r.read()

camera_info = json.loads(data)

print "price: %s"%(camera_info['products'][0]['regularPrice'])
And we run it:
$ python bb.py
price: 89.99
Whee!

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

On ORMs
It so happens that I end up dealing with the Python ORM SQLObject pretty often. I don't really like ORMs very much, since in my experience they make those 80% of database things that are already easy to do with plain SQL easier, while making the other 20% of database things which are already hard impossible. They do save some boiler-plate, and let you express your schema and queries in Python (or whatever programming language you are using) instead of SQL - but this tends to break down at a certain level of complexity, and just gets in the way. You end up doing a huge amount of wrangling with the ORM to do something which is very simple in plain old SQL. Fundamentally, SQL was designed as a declarative query language for the relational model and not to represent object hierarchies in the same way programming languages do, hence ORMs are always going to be a nasty hack in my opinion.

SQLObject vs. SQLAlchemy
sqlalchemy logo I actually much prefer SQLAlchemy to SQLObject. SQLAlchemy has a more explicit divide between its various components - you aren't forced to use the ORM stuff if you don't want to. It can be used just as a handy database abstraction layer with programmatic SQL and connection pooling and so on if you want. And if you truly want to go the full ORM mapping route, they provide that too. For truly tricky things, SQLAlchemy will be happy to provide you with a DB-API 2.0 cursor so that you can execute whatever custom SQL you wish.

Monolithic vs. modular
monolith This is my main problem with SQLObject - its very difficult to figure out how to get at the underlying database connection. I don't know how its possible to use the connection pooling and programmatic SQL builder without using the ORM but perhaps it is doable. The documentation for SQLObject is far inferior to the documentation of SQLAlchemy I'm sorry to say. Just try to figure out how to use transactions reliably with SQLObject! Even when I managed to put together some code which according to the documentation should work, SQLObject decided to interleave the actions in separate transactions. With SQLAlchemy I never had this problem.

Getting at the cursor
While seemingly undocumented, it is in fact possible to get the underlying driver's connection object, and from there grab a DB-API cursor. The pattern is:

# Set up the SQLObject connection as usual
connection = connectionForURI('sqlite:/:memory:')
# Grab the database connection
dbobj = connection.getConnection()
# Get a cursor from the low-level driver
cursor = dbobj.cursor()
# 
cursor.close()
Et voila.

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

Lots of people have a small home network. Usually you have a combo box which acts as a router/firewall/file server. Then you have a couple of other machines hooked up, and you share the Internet using NAT. A private DNS server is helpful in this kind of scenario for two reasons:

  • Recursive resolver cache can speed up common DNS lookups.
  • Private authoritative resolver lets you easily refer to machine in your home by name, instead of remembering IP addresses.
The DNS Dichotomy For many years there has been a dichotomy in the DNS server implementation world, pretty much between the ISC's BIND and just about everything else. The essence of this dichotomy is that BIND integrates both djbdns, has one tool - tinydns - for the authoritative portion while another - dnscache - implements the caching recursive resolver functionality. Convenience costs you security The monolithic BIND approach has certain limited benefits - mainly that it is convenient to configure and install a private DNS server which acts both as a cache and as an authority for the private domain. Unfortunately, this design has severe implications for the robustness of the software. It serves both to increase complexity within a single process while ignoring the principle of least privilege. Essentially, BIND is a horribly complicated beast, with serious security vulnerabilities being found pretty often - and even the smallest security flaw can result in major problems due to the single process design. Alternative approaches [caption id="attachment_487" align="aligncenter" width="250" caption="Unbound: A modern, secure DNS server"]Unbound[/caption] While djbdns might be one of the better-known BIND alternatives, I recently came across Unbound, a BSD licensed recursive resolver. One of the authors of Unbound is also an OpenBSD developer, which inspires confidence in the security of the software. Unbound also does simple authoritative resolution One of the nifty features of Unbound is that you can very simply configure it to act as an authority for your private domains. Due to this feature, you can have a single daemon on your home network router acting as both a cache and server for your local domain. This is very nice. In fact, I have found the Unbound configuration format to be considerably nicer to deal with than that of BIND. Setup under OpenBSD This describes how I set up Unbound on my OpenBSD machine - it should be a pretty similar procedure on most other operating systems.
# install the package
$ sudo pkg_add -i unbound
Now you have the binaries on disk, you can edit the configuration to set up your private domain. Unbound runs as a recursive resolver out of the box, so this is just about all the configuration you'll need to do.
# edit the config
$ sudo vi /var/unbound/etc/unbound.conf
For a single machine, add the following under 'server', replacing 'inet' with the desired name of your local domain, and 'joust' with the name of your machine:
    local-zone: "inet." static
    local-data: "joust.inet. IN A 192.168.1.1"
Since you want the DNS server to be accessible from other machines, you probably want it to listen on 0.0.0.0 (all available interfaces). Make sure you have some kind of firewall in place before you do this, though - you don't want to let random Internet hosts query your DNS server:
    interface: 0.0.0.0
    # Make sure you have a packet filter to block queries from the Internet.
    # Alternatively, set this only for your local network.
    access-control: 0.0.0.0/0 allow
Now you can start up Unbound:
$ sudo /usr/local/sbin/unbound
And of course you probably want it to come up on boot, so follow these instructions:
$ pkg_info -D unbound
Information for inst:unbound-1.2.1p0

Install notice:
You should add:

    syslogd_flags="${syslogd_flags} -a /var/unbound/dev/log"

to /etc/rc.conf.local to create a syslog socket in the unbound chroot.

You may also want to add the following to /etc/rc.local to start unbound
at boot:

        if [ -x /usr/local/sbin/unbound ]; then
                echo -n ' unbound'; /usr/local/sbin/unbound 
        fi

Niall O'Higgins is an author and software developer. He wrote the O'Reilly book MongoDB and Python. He also develops Strider Open Source Continuous Deployment and offers full-stack consulting services at FrozenRidge.co.

Read and Post Comments

« Previous Page -- Next Page »