Python tools for local continuous integration

The Python testing ecosystem is rich and diverse, with tools and frameworks available for unit testing, load testing, acceptance testing, code coverage analysis, code quality and standards compliance checking, mocking, and almost anything you can think of. Many of these tools work nicely together, and can be used at different stages of the development and life cycle of your application to ensure that your software works correctly.

This post is about a particular combination of these tools that I've found to be useful for day-to-day development: nose, tdaemon and nosegrowl2.

Test discovery and notification with nose

To actually run tests, I use nose. Nose makes it trivially easy to run tests, and helps with writing them too. It comes with a command-line tool (nosetests) which searches through your project's source tree, looking for tests. It collects and runs unittest.TestCase subclasses (unittest is Python's standard testing framework), but it will also collect anything that looks like a test, based on class, method or function name. So, for example, if you write a simple function called test_my_code_works(), nose will find it and run it for you.

As well as finding and running your tests, nose gives you some low-level tools for controlling test execution, setup and teardown. Most importantly, it has an easy-to-use plugin system (and plenty of built-in plugins) which allow you to "hook in" to any part of the test collection and running process. More on plugins later.

Django and nose

If you're working on a Django project, your testing requirements are a little more complex. Django's testing infrastructure does things like setting up test databases and installing fixtures before running your tests. Thanks to django-nose, you can take advantage of nose's niceties in your Django project. Once you've added django_nose to your INSTALLED_APPS, and set TEST_RUNNER to django_nose.NoseTestSuiteRunner, then running django-admin.py test (or manage.py test) will use nose to find and run your tests. This gives you all of nose's plugins, tools and test discovery benefits.

Continuous integration

Most developers are familiar with the idea of continuous integration, but usually this concept is applied at a very high level in the development process. It often takes the form of an automated build tool such as Jenkins which can automatically run your tests on a regular basis (perhaps nightly, or after every push into your main repository). This is an extremely valuable piece of infrastructure, but the same concept of continuous test running can also be applied on a "micro" scale, as part of the minute-to-minute workflow of an individual developer.

Local continuous integration with tdaemon

Whether you're doing test-driven development or just writing the tests after the code, it's important that those tests actually get run. The sooner they are run, the sooner you can identify problems in the code you're writing, and catch any regressions. Ideally, your tests should be run every time you make a change, and that's exactly what tdaemon accomplishes.

It works by constantly monitoring the files in your project's source tree for changes (much like Django's development server). Whenever it detects a change, it runs your tests. It's a very simple concept, but I've found it an extremely valuable addition to my workflow. It integrates nicely with nose and Django, as well as pytest and other tools. If I'm working on a Django project (with django-nose installed, see above) I run tdaemon -t django. If I'm working on a non-Django project (or something like a reusable app that doesn't need to use Django's testing infrastructure) I run tdaemon -t nose.

Running a subset of tests

If your project is very large and complex, your full test suite may take several minutes (or more) to run. In this case, you probably don't want to run every test every time you save a file. There are a few ways to avoid this, but the simplest is probably to use nose's attribute plugin. This allows you to mark some of your tests as, say, "slow":

You can then use tdaemon's custom args flag to tell nose to ignore these slow tests: tdaemon -t nose --custom-args="-a '!slow'". Of course, your higher-level CI tool (eg Jenkins) should run all the tests, however long they take.

Added bonus: test notifications with Growl

So, tdaemon is running in the background, monitoring your source tree and running your tests every time it detects a change. But what if your editor is running full-screen, and you can't see the output? Or perhaps tdaemon is in another terminal tab, and so you miss a new test failure? You really need to be told straight away about the status of your tests.

If you're a Mac user, you're probably familiar with Growl, a desktop notification framework for OS X. Wouldn't it be nice if we could use Growl to shout every time our tests run, and let us know whether they pass or fail?

It turns out that this isn't a new idea. Unfortunately, the existing NoseGrowl plugin has packaging problems and seems to have been abandoned on PyPI. Even worse, it depends on PyGrowl, which is fiendishly complicated, and has also been abandoned on PyPI. The existing version uses the md5 module which has been deprecated in favour of hashlib, and so throws warnings in recent versions of Python. A new version exists in the main Growl repository but must be downloaded and installed manually, which is a pain.

Sidestepping the Python packaging ecosystem problems raised by all this, I decided to fix things, and so I wrote a new nose plugin, nosegrowl2. This is much simpler and smaller than the original NoseGrowl, and can be installed cleanly with pip install nosegrowl2. Its only dependency is the command-line growlnotify tool. You can easily install this with Homebrew by typing brew install growlnotify.

You can then run nosetests --with-growl and get lovely notifications on your desktop:

To have tdaemon show these, you can either run tdaemon -t nose --custom-args="--with-growl", or you can put with-growl=1 in your global ~/.noserc config file.

Notifications on other platforms

On Ubuntu, you can use nose-notify for desktop notifications. I'm sure there are similar tools for other desktop environments - and if not, it's fairly easy to create your own nose plugin to do exactly what you want.

Conclusion

This combination of nose, tdaemon and Growl makes it fun and easy to test your code constantly, even during development. Personally, I've found it to be incredibly helpful to my workflow. If one extra bug is caught somewhere in the world thanks to this article, the time spent writing it was entirely worthwhile!