Tuesday, January 11, 2011

How To Speed Up Django Tests on Postgresql

I had a problem with a Django project that took forever to run its unit tests. The test database took an enormous amount of time to run, upwards of ten to fifteen minutes each. I didn't have a lot of ways around this, because I had to use a base Model that pulled in lots of cascading requirements and I couldn't avoid the dozens of applications it needed to build tables for. This was really hindering my ability to develop, as I rely heavily on constantly running tests in my own pathetic attempt at Continuous Integration.

After some poking around the PG forums, I eventually worked out this script, which I now run on startup.


#!/usr/bin/env bash
service postgresql stop
mount -t tmpfs -o size=500m tmpfs /mnt/pg_data_mem/
cp -R /var/lib/postgresql/8.4/main/ /mnt/pg_data_mem/
mount --bind /var/lib/postgresql/8.4/main/pg_xlog /mnt/pg_data_mem/main/pg_xlog
chown -R postgres:postgres /mnt/pg_data_mem/
sudo -u postgres /usr/lib/postgresql/8.4/bin/pg_resetxlog -f /mnt/pg_data_mem/main/service postgresql start

I also set data_directory = '/mnt/pg_data_mem/main' in postgresql.conf.

This works in development, where I don't need my DB to persist between reboots. If I did want to keep it around, I could just copy from /mnt/pg_data_mem/main/ to /var/lib/postgresql/8.4/main/ and keep it on disc. For now, my one-way solution works.

Sunday, January 02, 2011

Soft Announcement: Trapdoor Prototype

Previously I made a little preview of something I'm announcing today, but it is a soft announcement as I've had it on github for some time, but I haven't done anything with the project.

This is just a prototype, an experiment, and it is called Trapdoor. I don't remember why I named it Trapdoor.

Trapdoor has a simple concept, along the lines of Mozilla Prism. I wanted to take things I use to build web applications and test how they could be applied to building desktop applications. This meant two things:

1) I need to wrap up a web application in something that can be run locally.

2) I need some way to extend a web application in ways that normally only desktop applications can provide.

This has been done in the simplest way I could find, at the time, and is available on github.

extensions:
 - calculator.Calculator
js:
 - calculator.js
plugins:
 - trapdoor.contrib.windowmanager

Applications are configured through this simply manifest, where they are given extensions, plugins, and javascript. Honestly, extensions and plugins can probably be merged. They are both defined by a Python class and the object exposes methods to the Javascript, which gets loaded and initialized by the runtime. The current version includes two plugins:

trapdoor.contrib.windowmanager

This is a basic plugin that provides a simple createWindow method, through which the Javascript can create new windows with (in the future) more control than the standard Javascript APIs would otherwise provide. One thing I'll be adding to this is a fullscreen API and other properties to control the window appearance, such as borders and which controls are visible.

trapdoor.contrib.nodes

This is my favorite of the two standard plugins. Each Node is an isolated Javascript runtime, in its own global space. The application is loaded in the first node, but it can create and initialize others, allowing it to run untrusted Javascript safely. If I continue development, I hope to use this to test ways to allow community additions to software without worrying about what they are running. This is similar to how extensions and user scripts work in Firefox and Chrome.

Now, I don't know if more will come of this. I think, if it does, it should probably be evaluated if I should rewrite it based on Chromium and V8, rather than Qt and whatever Javascript engine it is running on. It is also lacking a solid use case, for me, that I can use to drive my desire to improve it. However, I had fun writing it for the thought experiment, and I do hope to do more with it in the near future.

Please, fork it and tell me what you think.
I write here about programming, how to program better, things I think are neat and are related to programming. I might write other things at my personal website.

I am happily employed by the excellent Caktus Group, located in beautiful and friendly Carrboro, NC, where I work with Python, Django, and Javascript.