further_extensions

= Python - Further Extensions using C =

  



Any code that you write using any compiled language like C, C++ or Java can be integrated or imported into another Python script. This code is considered as an "extension". A Python extension module is nothing more than a normal C library. On Unix machines, these libraries usually end in **.so** (for shared object). On Windows machines, you typically see **.dll** (for dynamically linked library).

Pre-Requisite:
To start writing your extension you are going to need the Python header files. > > Additionally, it is assumed that you have good knowledge of C or C++ to write any Python Extension using C programming.
 *  On Unix machines, this usually requires installing a developer-specific package such as python2.5-dev.
 *  Windows users get these headers as part of the package when they use the binary Python installer.

First look at a Python extension:
For your first look at a Python extension module, you'll be grouping your code into four parts: > > > > = The header file //Python.h// = Start including //Python.h// header file in your C source file, which will give you access to the internal Python API used to hook your module into the interpreter. Be sure to include Python.h before any other headers you might need. You'll follow the includes with the functions you want to call from Python. = The C functions: = The signatures of the C implementations of your functions will always take one of the following three forms: code static PyObject *MyFunction( PyObject *self, PyObject *args );
 *  The header file //Python.h//.
 * <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> The C functions you want to expose as the interface from your module.
 * <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> A table mapping the names of your functions as Python developers will see them to C functions inside the extension module.
 * <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> An initialization function.

static PyObject *MyFunctionWithKeywords(PyObject *self,                                PyObject *args,                                 PyObject *kw);

static PyObject *MyFunctionWithNoArgs( PyObject *self ); code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Each one of the preceding declarations returns a Python object. There's no such thing as a //void//function in Python as there is in C. If you don't want your functions to return a value, return the C equivalent of Python's **None** value. The Python headers define a macro, Py_RETURN_NONE, that does this for us. <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">The names of your C functions can be whatever you like as they will never be seen outside of the extension module. So they would be defined as //static// function. <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Your C functions usually are named by combining the Python module and function names together, as shown here: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">static PyObject *//module_func//(PyObject *self, PyObject *args) { /* Do your stuff here. */  Py_RETURN_NONE; } code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">This would be a Python function called //func// inside of the module //module//. You'll be putting pointers to your C functions into the method table for the module that usually comes next in your source code. = The method mapping table: = <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">This method table is a simple array of PyMethodDef structures. That structure looks something like this: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">struct PyMethodDef { char *ml_name; PyCFunction ml_meth; int ml_flags; char *ml_doc; }; code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Here is the description of the members of this structure: > > > <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">This table needs to be terminated with a sentinel that consists of NULL and 0 values for the appropriate members.
 * <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> **ml_name:** This is the name of the function as the Python interpreter will present it when it is used in Python programs.
 * <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> **ml_meth:** This must be the address to a function that has any one of the signatures described in previous seection.
 * <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> **ml_flags:** This tells the interpreter which of the three signatures ml_meth is using.
 * This flag will usually have a value of METH_VARARGS.
 * This flag can be bitwise or'ed with METH_KEYWORDS if you want to allow keyword arguments into your function.
 * This can also have a value of METH_NOARGS that indicates you don't want to accept any arguments.
 * <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> **ml_doc:** This is the docstring for the function, which could be NULL if you don't feel like writing one

Example:
<span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">For the above defined function, we would have following method mapping table: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">static PyMethodDef //module//_methods[] = { { "//func//", (PyCFunction)//module_func//, METH_NOARGS, NULL }, { NULL, NULL, 0, NULL } }; code || = The initialization function: = <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">The last part of your extension module is the initialization function. This function is called by the Python interpreter when the module is loaded. It's required that the function be named**init//Module//**, where //Module// is the name of the module. <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">The initialization function needs to be exported from the library you'll be building. The Python headers define PyMODINIT_FUNC to include the appropriate incantations for that to happen for the particular environment in which we're compiling. All you have to do is use it when defining the function. <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Your C initialization function generally has the following overall structure: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">PyMODINIT_FUNC init//Module// { Py_InitModule3(//func//, //module//_methods, "docstring..."); } code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Here is the description of //Py_InitModule3// function: > > > <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Putting this all together looks like the following: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">#include <Python.h>
 * <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> **func:** This is the function to be exported.
 * <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> **//module//_methods:** This is the mapping table name defined above.
 * <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> **//docstring://** This is the comment you want to give in your extension.

static PyObject *//module_func//(PyObject *self, PyObject *args) { /* Do your stuff here. */  Py_RETURN_NONE; }

static PyMethodDef //module//_methods[] = { { "//func//", (PyCFunction)//module_func//, METH_NOARGS, NULL }, { NULL, NULL, 0, NULL } };

PyMODINIT_FUNC init//Module// { Py_InitModule3(//func//, //module//_methods, "docstring..."); } code ||

Example:
<span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">A simple example that makes use of all the above concepts: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">#include <Python.h>

static PyObject* helloworld(PyObject* self) {   return Py_BuildValue("s", "Hello, Python extensions!!"); }

static char helloworld_docs[] = "helloworld: Any message you want to put here!!\n";

static PyMethodDef helloworld_funcs[] = { {"helloworld", (PyCFunction)helloworld, METH_NOARGS, helloworld_docs}, {NULL} };

void inithelloworld(void) {   Py_InitModule3("helloworld", helloworld_funcs,                   "Extension module example!"); } code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Here the //Py_BuildValue// function is used to build a Python value. Save above code in hello.c file. We would see how to compile and install this module to be called from Python script. = Building and Installing Extensions: = <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">The //distutils// package makes it very easy to distribute Python modules, both pure Python and extension modules, in a standard way. Modules are distributed in source form and built and installed via a setup script usually called //setup.py// as follows. <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">For the above module, you would have to prepare following setup.py script: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">from distutils.core import setup, Extension setup(name='helloworld', version='1.0', \      ext_modules=[Extension('helloworld', ['hello.c'])]) code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Now use the following command, which would perform all needed compilation and linking steps, with the right compiler and linker commands and flags, and copies the resulting dynamic library into an appropriate directory: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">$ python setup.py install code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">On Unix-based systems, you'll most likely need to run this command as root in order to have permissions to write to the site-packages directory. This usually isn't a problem on Windows = Import Extensions: = <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Once you installed your extension, you would be able to import and call that extension in your Python script as follows: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">#!/usr/bin/python import helloworld

print helloworld.helloworld code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">This would produce following result: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">Hello, Python extensions!! code || = Passing Function Parameters: = <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Because you'll most likely want to define functions that do accept arguments, you can use one of the other signatures for your C functions. For example, following function, that accepts some number of parameters, would be defined like this: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">static PyObject *//module_func//(PyObject *self, PyObject *args) { /* Parse args and do something interesting here. */  Py_RETURN_NONE; } code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">The method table containing an entry for the new function would look like this: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">static PyMethodDef //module//_methods[] = { { "//func//", (PyCFunction)//module_func//, METH_NOARGS, NULL }, { "//func//", //module_func//, METH_VARARGS, NULL }, { NULL, NULL, 0, NULL } }; code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">You can use API //PyArg_ParseTuple// function to extract the arguments from the one PyObject pointer passed into your C function. <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">The first argument to PyArg_ParseTuple is the args argument. This is the object you'll be//parsing//. The second argument is a format string describing the arguments as you expect them to appear. Each argument is represented by one or more characters in the format string as follows. code <span style="font-family: 'Courier New',monospace; font-size: 12px;">static PyObject *//module_func//(PyObject *self, PyObject *args) { int i;  double d;   char *s;

if (!PyArg_ParseTuple(args, "ids", &i, &d, &s)) { return NULL; }

/* Do something interesting here. */  Py_RETURN_NONE; } code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Compiling the new version of your module and importing it will enable you to invoke the new function with any number of arguments of any type: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">module.func(1, s="three", d=2.0) module.func(i=1, d=2.0, s="three") module.func(s="three", d=2.0, i=1) code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">You can probably come up with even more variations.

The //PyArg_ParseTuple// Function:
<span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Here is the standard signature for **PyArg_ParseTuple** function: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">int PyArg_ParseTuple(PyObject* tuple,char* format,...) code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">This function returns 0 for errors, and a value not equal to 0 for success. tuple is the PyObject* that was the C function's second argument. Here //format// is a C string that describes mandatory and optional arguments. <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Here is a list of format codes for **PyArg_ParseTuple** function: = Returning Values: = <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">//Py_BuildValue// takes in a format string much like //PyArg_ParseTuple// does. Instead of passing in the addresses of the values you're building, you pass in the actual values. Here's an example showing how to implement an add function: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">static PyObject *foo_add(PyObject *self, PyObject *args) { int a;  int b;
 * ~ Code ||~ C type ||~ Meaning ||
 * c || char || A Python string of length 1 becomes a C char. ||
 * d || double || A Python float becomes a C double. ||
 * f || float || A Python float becomes a C float. ||
 * i || int || A Python int becomes a C int. ||
 * l || long || A Python int becomes a C long. ||
 * L || long long || A Python int becomes a C long long ||
 * O || PyObject* || Gets non-NULL borrowed reference to Python argument. ||
 * s || char* || Python string without embedded nulls to C char*. ||
 * s# || char*+int || Any Python string to C address and length. ||
 * t# || char*+int || Read-only single-segment buffer to C address and length. ||
 * u || Py_UNICODE* || Python Unicode without embedded nulls to C. ||
 * u# || Py_UNICODE*+int || Any Python Unicode C address and length. ||
 * w# || char*+int || Read/write single-segment buffer to C address and length. ||
 * z || char* || Like s, also accepts None (sets C char* to NULL). ||
 * z# || char*+int || Like s#, also accepts None (sets C char* to NULL). ||
 * (...) || as per ... || A Python sequence is treated as one argument per item. ||
 * | ||  || The following arguments are optional. ||
 * : ||  || Format end, followed by function name for error messages. ||
 * ; ||  || Format end, followed by entire error message text. ||

if (!PyArg_ParseTuple(args, "ii", &a, &b)) { return NULL; }  return Py_BuildValue("i", a + b); } code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">This is what it would look like if implemented in Python: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">def add(a, b): return (a + b) code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">You can return two values from your function as follows, this would be cauptured using a list in Python. code <span style="font-family: 'Courier New',monospace; font-size: 12px;">static PyObject *foo_add_subtract(PyObject *self, PyObject *args) { int a;  int b;

if (!PyArg_ParseTuple(args, "ii", &a, &b)) { return NULL; }  return Py_BuildValue("ii", a + b, a - b); } code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">This is what it would look like if implemented in Python: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">def add_subtract(a, b): return (a + b, a - b) code ||

The //Py_BuildValue// Function:
<span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Here is the standard signature for **Py_BuildValue** function: code <span style="font-family: 'Courier New',monospace; font-size: 12px;">PyObject* Py_BuildValue(char* format,...) code || <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Here //format// is a C string that describes the Python object to build. The following arguments of//Py_BuildValue// are C values from which the result is built. The //PyObject*// result is a new reference. <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Following table lists the commonly used code strings, of which zero or more are joined into string format. <span style="background-color: #ffffff; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: justify;">Code {...} builds dictionaries from an even number of C values, alternately keys and values. For example, Py_BuildValue("{issi}",23,"zig","zag",42) returns a dictionary like Python's {23:'zig','zag':42}.
 * ~ Code ||~ C type ||~ Meaning ||
 * c || char || A C char becomes a Python string of length 1. ||
 * d || double || A C double becomes a Python float. ||
 * f || float || A C float becomes a Python float. ||
 * i || int || A C int becomes a Python int. ||
 * l || long || A C long becomes a Python int. ||
 * N || PyObject* || Passes a Python object and steals a reference. ||
 * O || PyObject* || Passes a Python object and INCREFs it as normal. ||
 * O& || convert+void* || Arbitrary conversion ||
 * s || char* || C 0-terminated char* to Python string, or NULL to None. ||
 * s# || char*+int || C char* and length to Python string, or NULL to None. ||
 * u || Py_UNICODE* || C-wide, null-terminated string to Python Unicode, or NULL to None. ||
 * u# || Py_UNICODE*+int || C-wide string and length to Python Unicode, or NULL to None. ||
 * w# || char*+int || Read/write single-segment buffer to C address and length. ||
 * z || char* || Like s, also accepts None (sets C char* to NULL). ||
 * z# || char*+int || Like s#, also accepts None (sets C char* to NULL). ||
 * (...) || as per ... || Builds Python tuple from C values. ||
 * [...] || as per ... || Builds Python list from C values. ||
 * {...} || as per ... || Builds Python dictionary from C values, alternating keys and values. ||

<span style="background-color: #ffffff; color: #900b09; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> <span style="background-color: #ffffff; color: #900b09; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;"> <span style="background-color: #ffffff; color: #900b09; font-family: verdana,helvetica,arial,sans-serif; font-size: 11px; text-align: left;">