Will's Journal

We Have a Rachel

2007-12-30T20:34:00Z

Loading C Plugins for PureDocs

2007-09-01T13:25:00Z

I finally had a chance to sit back down with PureDocs, after getting the build process straightened out, and figure out how to load C plugins. Turns out, it's pretty simple with ctypes. Let's imagine I have a C library, compiled and linked as libtest, containing the following (a simple modification of my original example):

#include <stdio.h>
#include "puredocs.h"

Resource * test_function() {
    Resource * r = puredocs_createResource("foo");
    Resource * c = puredocs_createResource("bar");

    puredocs_setDocString(r, "omg some docs lol");
    puredocs_setDocString(c, "some bar docs omg lol");

    puredocs_setProperty(r, "feh", "omg");

    if ((puredocs_addChild(r, c)) < 0) {
        printf("OH NOES!\n");
    }

    return r;
}

This function can easily be called when loading the libary with``ctypes``'s PyDLL type (won't release the Python interpreter's GIL, which is important if we're creating Python objects we want to use). The restype of the function needs to be set to ctypes.py_object, since we're returning a Python object. The whole process looks like this:

>>> from ctypes import *
>>> t = pydll.LoadLibrary("libtest.dylib")
>>> t.test_function.restype=py_object
>>> r = t.test_function()
>>> r
<Resource foo [<Resource bar []>]>

The whole point of this exercise is basically to make it possible to write C libraries that serve as plugins for PureDocs that don't necessarily have any knowledge of Python. They'll be linked against a PureDocs library that provides the glue-code. So, for PureDocs's purpose, I now just need to wrap this around some library loading and registration code, and we'll be ready to go!

Peanut Thumbing Her Nose

2007-08-21T05:54:35Z

Building PureDocs, Redux

2007-08-11T03:58:00Z

After much trial and tribulation with Pyhton's distutils and setuptools, I decided it wasn't really appropriate to be building a C library with the Python tools. While the capability exists in distutils, it's pretty obvious that it suffers from some pretty substantial code rot.

So, I've gone the autoconf route. I didn't really want to, but really there's no better solution that I can find that doesn't involve setting PureDocs aside while I start yet another project.

In any case, the autoconf'ing isn't complete, and it doesn't quite work, but I've committed what I have so far. Once I get it all working well on multiple platforms, I'll finish off the C API. Then Kate should be able to add some C-based plugins using the SID parser generator.

We Have a Peanut

2007-05-15T23:34:00Z

Building PureDocs

2007-04-25T00:45:43Z

PureDocs seems reasonably unique in being a Python system that provides an external C library, the kind of thing that would probably be happiest with autoconf and make. Yes, I dislike autoconf, automake, and everything in between, but I'm writing an API doc-generation system, I don't want to maintain a build system.

But, I naturally want a unified build system for the entire kit and kaboodle. The C library should come with the Python system, and the Python system could come with the C library. This includes Python egg files, which can serve as the basis for other package files. So, anything I choose needs to integrate with the existing distutils/setuptools-based Python distribution system.

Transforming xar Files with XSLT

2007-04-14T05:23:00Z

One of the more interesting things about xar is the representation of the the Table of Contents (toc) in XML, with the ability to store "subdocs", or additional XML trees, within the toc. You can easily add your own metadata to a xar file, in addition to its support for a variety of new metadata, including Extended-Attributes, both POSIX and Mac OS X's.

Anyway, I've been playing with prototyping a package manager based on XSL transformations of xar files. Basically registering extension functions that can perform the necessary actions on the xar file as it is transformed. It's a very interesting concept, and one I'm looking forward to exploring in greater detail.

I'll write some more about this as I have time to play with it.

C Plugins for PureDocs

2007-03-18T01:05:00Z

I had the opportunity to sit down and design a C API for creating PureDocs Resource objects this weekend. For the C API, Resource is an opaque type, and the API provides the ability to create a Resource, add properties, children, etc, without having to worry about it being a Python type.

The reason behind C plugin support for an API documentation generation utility that parses files based on grammars should be pretty obvious. Because most higher-level languages, at the very least, have implementations in C, there are readily available parsers. Having to reinvent the wheel just because we're using Python, or at the very least, write Python-C support code, would be painful and reduce the usefulness of PureDocs. Having a simple C API that can be used to wrap one of these existing parsers seems to be the optimal situation.

The C API itself is defined in a library, libpuredocs, that links against Python to provide access to the Python data types. C plugins will be loadable modules linked against libpuredocs. They will be dyanmically loaded by PureDocs (perhaps by a C Python module) into the same address space, so that Resource objects ca easily be passed back and forth between PureDocs and the C plugin.

It took me a little while to come up with just the right way to do this to avoid C plugin authors having to have any interaction with Python at all. I still haven't figured out how loading and registration of C plugins will work, but I think the datatypes were the hard part. I'll write more about this once the entire thing is finalized. For testing purposes, I have passed a Resource object back and forth between a pure-C loadable module and a C python module, with success.

A quick test showing that Resource objects created using the libpuredocs library can be used in Python looks something like the following, implemented in C:

#include <Python.h>
#include <stdio.h>
#include "puredocs.h"

PyObject * test_function(PyObject * self, PyObject * args) {
    Resource * r = puredocs_createResource("foo");
    Resource * c = puredocs_createResource("bar");

    puredocs_setDocString(r, "omg some docs lol");
    puredocs_setDocString(c, "some bar docs omg lol");

    puredocs_setProperty(r, "feh", "omg");

    if ((puredocs_addChild(r, c)) < 0) {
        printf("OH NOES!\n");
    }

    return (PyObject *)r;
}

The C API (along with the Python API) is documented in The PureDocs Plugin Architecture. Again, I'm going to provide more concrete examples once I have loading and registration worked out, and have some working C plugins as examples.

I Haven't Forgotten TracGantt

2007-02-13T22:34:00Z

TracGantt isn't forgotten, but it's definitely neglected. I'm thoroughly convinced that putting tickets themselves on a Gantt chart, by using extra fields, was a fundamentally bad idea. The correct thing to do would be to use Milestones, which already have a lot of the implementation details that we were adding to Tickets.

I plan to rewrite TracGantt at some point in the future, but I don't know when that will be.

Implementing pyxar With ctypes

2007-01-16T00:14:00Z

I've been exploring implementing pyxar with the ctypes FFI module for Python, which is included in Python2.5. Currently, pyxar is a compiled C module for Python built with Pyrex. Ultimately, ctypes might be easier to maintain than Pyrex, and it certainly requires less on the user's end, now that it is included with Python.

I'm hoping to have something working soon.