Thursday 22 October 2009

Custom Doctest Syntax with Manuel

Python doctests allow you to execute code within a single document. The special syntax '>>>' is used to denote python statements that are to be executed within the interpreter, and '...' is used to denote the body of compound statements. Using these tools, you can create anything that is available to you in a standard Python interpreter session:

>>> 2 + 2
4
>>> def func():
... pass
>>>

That said, there is no way of easily defining a module in a Python doctest, and having that namespace available in subsequent doctest interactions. Many packages (or maybe meta-packages??) themselves operate on modules, one good example is martian which 'scans' them for configuration information.

The martian project's solution was to write the module's body within a class that inherited from 'FakeModule'. The FakeModule's metaclass then extracted the namespace out of your class, and pushed it into a module (updating all the __module__ variables on the way). A cut down version of this (from the martian README):

>>> class templating(FakeModule):
...
... class InterpolationTemplate(Template):
... "Use %(foo)s for dictionary interpolation."
... extension('.txt')
... def __init__(self, text):
... self.text = text
...
...
... # the registry, empty to start with
... extension_handlers = {}
...
... def render(data, extension, **kw):
... # this hasn't changed
... template = extension_handlers[extension](data)
... return template.render(**kw)
>>> from martiantest.fake import templating

As you can see, this is quite ugly!

Fortunately, Benji York has written Manuel, a system for defining and executing your own custom doctest syntax. As a simple example, take a look at the footnote example:

Here we reference a footnote. [1]_

>>> x
42

.. [1] This is a test footnote definition.

>>> x = 42

So this is a doctest with executable footnotes! Manuel scans the document for footnotes defined with syntax looking like:

'[?]_'

and then executes the matching footnote's body! Check out the manuel page for details of other extensions and syntax.

Fake Modules

Using this system, I've written an extension to Manuel that recognises module blocks and makes them available in the rest of your doctests. We use the 'module-block' directive to define our modules. Take a look at the doctest example:

.. module-block:: some_module
import pprint

a = ['10', '20', '30']

def some_function(bar):
pprint.pprint(bar)

def get_my_module():
print get_my_module.__module__

An unindented line like this marks the end of the module
definition. We can, of course, mix classic doctest tests
in with our module-blocks. Let's test that our module
was set up correctly:

>>> some_module.some_function(some_module.a)
['10', '20', '30']

We'll have a poke around and see what's in `some_module`:

>>> dir(some_module)
['__builtins__', '__doc__', '__file__', '__name__',
'a', 'get_my_module', 'pprint', 'some_function']

Modules must have a file location, this is just a dummy/
unique namespace::

>>> print some_module.__file__
/manueltest/fake/some_module

Are functions aware of their module?::

>>> print some_module.some_function.__module__
manueltest.fake.some_module

Externally, yes. What about from within the module::

>>> some_module.get_my_module()
manueltest.fake.some_module

As you can see, this is a much cleaner and readible way of defining modules within your doctests!

I intend to release this as a separate plugin package 'manuelpi.fake_module'.

Thursday 8 October 2009

Grok 1.0 Released!!

Grok 1.0 has been released today!

Over the past 9 months I've been watching the development of Grok progress, and it has been a great learning experience. I've only been able to contribute a single module and some documentation, but I've written quite a bit of code upon it, and have been mightily impressed with its power and versatility.

Hopefully over the next year I'll have more time to dedicate to it....