Tuesday, January 11, 2011

How To Speed Up Django Tests on Postgresql

I had a problem with a Django project that took forever to run its unit tests. The test database took an enormous amount of time to run, upwards of ten to fifteen minutes each. I didn't have a lot of ways around this, because I had to use a base Model that pulled in lots of cascading requirements and I couldn't avoid the dozens of applications it needed to build tables for. This was really hindering my ability to develop, as I rely heavily on constantly running tests in my own pathetic attempt at Continuous Integration.

After some poking around the PG forums, I eventually worked out this script, which I now run on startup.


#!/usr/bin/env bash
service postgresql stop
mount -t tmpfs -o size=500m tmpfs /mnt/pg_data_mem/
cp -R /var/lib/postgresql/8.4/main/ /mnt/pg_data_mem/
mount --bind /var/lib/postgresql/8.4/main/pg_xlog /mnt/pg_data_mem/main/pg_xlog
chown -R postgres:postgres /mnt/pg_data_mem/
sudo -u postgres /usr/lib/postgresql/8.4/bin/pg_resetxlog -f /mnt/pg_data_mem/main/service postgresql start

I also set data_directory = '/mnt/pg_data_mem/main' in postgresql.conf.

This works in development, where I don't need my DB to persist between reboots. If I did want to keep it around, I could just copy from /mnt/pg_data_mem/main/ to /var/lib/postgresql/8.4/main/ and keep it on disc. For now, my one-way solution works.

4 comments:

Stephen said...

On most modern Linux systems, you already have a tmpfs volume mounted on /dev/shm. Just make a directory there (/dev/shm/postgres), or put your directory right in /dev/shm. That should make your setup a bit easier.

akaihola said...

Also make sure you have "fsync = off" in your postgresql.conf. Makes a big difference.

stefantalpalaru said...

I agree with akaihola, "fsync = off" provides roughly the same speed up at the same price (the risk of data corruption on hardware failure).

gorsky said...

@akaihola - brilliant tip; easy to do, speedup is significant. Thanks!

I write here about programming, how to program better, things I think are neat and are related to programming. I might write other things at my personal website.

I am happily employed by the excellent Caktus Group, located in beautiful and friendly Carrboro, NC, where I work with Python, Django, and Javascript.