Reporting on a project I worked on for the last few weeks – porting the excellent Gesture Recognition Toolkit (GRT) to Python.
Right now it’s still a pull request: https://github.com/nickgillian/grt/pull/151.
Not exactly porting, rather I’ve simply added Python bindings to GRT that allow you to access the GRT C++ APIs from Python.
Did it using the wonderful SWIG project. Such a wondrous tool, SWIG is. Magical.
Here are the deets
I know GRT from my time in the MIT Media Lab, where I met Nick Gillian.
Back then, and seems right now as well, GRT is one of the only general purpose gesture recognition frameworks out there.
Sure, machine learning pipelining tools abound (e.g. scikit-learn), but nothing is specifically targeting gesture recognition.
Too bad GRT is written in C++ and not part of the modern machine learning universe, dominated by Python.
Well not any more!
Now you can write sweet Python GRT like this:
print("Loading dataset...")
training_data = GRT.ClassificationData()
training_data.load(filename)
print("Data Loaded")
# Print out some stats about the training data
training_data.printStats()
print("Splitting data into training/test split...")
test_data = training_data.split(80)
# Create a new Gesture Recognition Pipeline
pipeline = GRT.GestureRecognitionPipeline()
# Add a KNN classifier to the pipeline with a K value of 10
knn = GRT.KNN(10)
pipeline.setClassifier(knn)
print("Training model...")
pipeline.train(training_data)
print("Testing model...")
pipeline.test(test_data):
# Print some stats about the testing
print("Pipeline Test Accuracy: %.3f"%(pipeline.getTestAccuracy()))
(the c++ equivalent of this code is twice as long… 🙂
Actually, wrapping GRT with a Python binding using SWIG was not a very difficult task.
Some things were difficult, and I’ll tell you about them in a minute, but GRT itself is written so well and builds very smoothly that adding SWIG was mostly very simple.
As a way to start covering the entire GRT API I set out to replicate all the GRT examples, instead of just going h-file-by-h-file.
This proved like a good method.
Some considerations:
- vectors of INT (e.g. lists of IDs, categorical) should translate to Python lists
- floating point vectors, matrices and other tensors should translate to numpy arrays
- the API should work in Python the same way it works in C++ (no API breaks), just better and simpler
Starting easy with a simple vector
For the most part, things were easy.
However since GRT implements its own templated Vector<T> class (that inherits from std::vector), and is mostly used throughout in form of its Vector<UINT> – I had to introduce some glue C code in swig to translate between Vector<UINT> (and its various forms) and PyListObject.
This is not difficult to do in just a few lines in the SWIG .i file:
//...
%include "std_vector.i"
namespace std {
%template(IntVector) vector<int>;
}
//...
%apply int { UINT }; // type aliasing
%apply int { GRT::UINT };
//...
%template(UINTVector) std::vector<UINT>; // initialize a template for the inheritance
%include "../GRT/DataStructures/Vector.h"
%template(VectorTUINT) GRT::Vector<UINT>; // initialize the inheritance
//...
%typemap(out) GRT::Vector<UINT>, const GRT::Vector<UINT> %{
$result = PyList_New($1.size());
for (int i = 0; i < $1.size(); ++i) {
PyList_SetItem($result, i, PyInt_FromLong($1[i]));
}
%
And going back from Python list to Vector<UINT>: (a %typemap(in) directive, since it’s going in C++)
%typemap(in) GRT::Vector< UINT >,
const GRT::Vector< UINT >
{
Py_ssize_t size = PyList_Size($input); //get size of the list
for (int i = 0; i < size; i++) {
PyObject *s = PyList_GetItem($input,i);
if (!PyInt_Check(s)) {
PyErr_SetString(PyExc_ValueError, "List items must be integers");
return NULL;
}
$1.push_back((int)PyLong_AsLong(s)); //put the value into the array
}
}
For good measure, I added a type check for accepting lists as arguments for functions that accept Vector<UINT>:
// Make sure vector arg is a python list
%typecheck(SWIG_TYPECHECK_POINTER) GRT::Vector< UINT >,
const GRT::Vector< UINT >
%{
$1 = PyList_Check($input);
%}
The way it works is that SWIG adds a wrapper around each C++ function, e.g.
SWIGINTERN PyObject *_wrap_new_KMeansFeatures(PyObject *self, PyObject *args) { ... }
That wraps the new KMeansFeatures call.
In python it’d be kmeansfeatures = KMeansFeatures(), which would create a new KMeansFeatures object. Under the hood SWIG will call this wrapper function.
In the wrapper SWIG adds code to check/validate the arguments, so that each of them matches the expected input.
new KMeansFeatures as specified here will accept a Vector<UINT> as the first parameter.
So SWIG, in c++, will add the following code, copying verbatim from our %typecheck in the .i file:
//...
if (argc == 1) {
int _v;
_v = PyList_Check(argv[0]); // copied verbatim
if (_v) {
return _wrap_new_KMeansFeatures__SWIG_2(self, args);
}
}
//...
Now – Numpy!
So far so good.
Now comes the real pain, handling the numpy arrays.
I sought out to make it very easy and seamless for programming in Python, so any translations between C++ GRT and Python types would work smoothly.
This turns out to require a lot of foot work in SWIG.
First some of the numpy .h headers are needed in the SWIG .i:
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION #include "numpy/npy_common.h" #include "numpy/ndarrayobject.h" #include "numpy/ndarraytypes.h" #include "numpy/arrayobject.h"
There are a lot of places (functions) in GRT that accept Vector<Float>. Some are by-val, some by-ref or by-pointer.
SWIG makes a distinction between the two (three) cases.
All by-vals are handled by val, and all by-ref and by-pointer are handled by pointer.
This means I had to create two version for the C++-to-Python translator:
%include "../GRT/DataStructures/VectorFloat.h"
// From GRT::VectorFloat to numpy.array
%typemap(out) GRT::Vector<Float>, // by-vals
GRT::VectorFloat,
const GRT::Vector<Float>,
const GRT::VectorFloat%{
{
npy_intp dims[1]{(npy_intp)($1.size())}; // use '.' for members
$result = PyArray_SimpleNew(1, dims, NPY_FLOAT);
PyArrayObject* arr_ptr = reinterpret_cast<PyArrayObject*>($result);
for (size_t i = 0; i < $1.size(); ++i) {
float* d_ptr = static_cast<float*>(PyArray_GETPTR1(arr_ptr, i));
*d_ptr = $1[i];
}
}
%}
%typemap(out) GRT::VectorFloat&, // by-refs or pointers
GRT::VectorFloat*,
const GRT::VectorFloat&,
const GRT::VectorFloat* %{
{
npy_intp dims[1]{(npy_intp)($1->size())}; // use '->' for members
$result = PyArray_SimpleNew(1, dims, NPY_FLOAT);
PyArrayObject* arr_ptr = reinterpret_cast<PyArrayObject*>($result);
for (size_t i = 0; i < $1->size(); ++i) {
float* d_ptr = static_cast<float*>(PyArray_GETPTR1(arr_ptr, i));
*d_ptr = $1->operator[](i); // operator invocation with '->'
}
}
%}
The back channel (Python-to-C++) is just as painful:
%typemap(in)
GRT::Vector<Float>&, // the by-ref/by-pointer version
const GRT::Vector<Float>&,
GRT::VectorFloat&,
const GRT::VectorFloat&,
GRT::VectorFloat const& %{
{
PyArrayObject* arrayobj = reinterpret_cast<PyArrayObject*>($input);
npy_intp size = PyArray_SIZE(arrayobj); //get size of the 1d array
$1 = new VectorFloat();
for (npy_intp i = 0; i < size; i++) {
void* itemptr = PyArray_GETPTR1(arrayobj, i);
PyObject *s = PyArray_GETITEM(arrayobj, reinterpret_cast<char *>(itemptr));
if (!PyFloat_Check(s)) {
PyErr_SetString(PyExc_ValueError, "List items must be floats");
return NULL;
}
$1->push_back((float)PyFloat_AsDouble(s)); //put the value into the array
}
}
%}
But – notice we have a new operator in the glue code.
Remember this is a temporary code in the wrapper that takes a Python object and feeds it into C++.
So the new VectorFloat() object doesn’t need to exist beyond just the wrapped function call.
Therefore we must delete it right after the invocation, using a %typemap(freearg) directive:
%typemap(freearg) GRT::VectorFloat&, const GRT::VectorFloat&, GRT::VectorFloat const& {
if ($1 != 0) {
delete($1);
}
}
It also makes sense to add some typechecks so we avoid any input to these functions that makes no sense.
For example if the function expects a numpy array of floats but we give it ints – it should throw an error.
%typecheck(SWIG_TYPECHECK_POINTER)
GRT::Vector<Float>,
GRT::VectorFloat,
const GRT::VectorFloat,
GRT::VectorFloat&,
const GRT::VectorFloat&,
GRT::VectorFloat const&
{
PyArrayObject* arrayobj = reinterpret_cast<PyArrayObject*>($input);
$1 = PyArray_Check(arrayobj) && PyArray_ISFLOAT(arrayobj) && (PyArray_NDIM(arrayobj) == 1);
}
The GRT::MatrixFloat and GRT::Vector<GRT::VectorFloat> I translated to numpy arrays (and back) in the same way, just with two accessors to the vector data.
I remembered to add %typecheck(SWIG_TYPECHECK_POINTER) and %typemap(freearg) as needed.
Just that (vectors and matrices) got me covered for ~90% of the C++ API in Python, and GRT was singing.
I transcoded many C++ examples to Python to prove the point, and they all worked exactly the same.
Building
To build I used CMake’s SWIG bindings, since GRT anyway builds with CMake.
This little CMake script create an importable Python library, but not yet a pip installable package. That’s future work.
You get a couple of files: GRT.py and GRT_.so.
So if you’re running Python from the directory with these files you can simply:
import GRT # and then get to work... pipeline = GRT.GestureRecognitionPipeline()
Simple as that.
That’s all I have for you today! Enjoy using Python-GRT (PyGRT I call it, original right? 🙂
Best
Roy.