A Kinect browser plugin with FireBreath [w/ code]

Hi,
Just reporting on a small achievement, part of a big project: Creating a browser plugin to display the Kinect depth map on screen.
The integration was fairly easy, which leads me to think that both FireBreath and OpenNI/Nite are pretty neat framework that are robust..
So let’s see how it’s done

From a template FireBreath plugin to an OpenGL plugin

FireBreath is kind of an amazing project. They aim to be able to write a single source that will create plugins for all browsers and all operating systems. A daunting feat by my book. But building a MacOS Safari/Firefox plugin using their framework proved very simple…
So I started here: http://www.firebreath.org/display/documentation/Mac+Video+Tutorial
It’s a video tutorial of how to create a plugin from template, build it, install it and run it. Follow their instructions and you’ll have your plugin ready in 10 minutes.
The next step will be to make our plugin display an OpenGL scene, which is what OpenNI/NITE use to display their depth map. This was also easy, borrowing code from the FireBreath OpenGL example.
However I ended up with a smaller source since I threw away most of the stuff…

class tutorialpluginMac : public tutorialplugin {
public:
    tutorialpluginMac();
	~tutorialpluginMac();
    BEGIN_PLUGIN_EVENT_MAP()
	EVENTTYPE_CASE(FB::AttachedEvent, onWindowAttached, FB::PluginWindowMac)
	EVENTTYPE_CASE(FB::DetachedEvent, onWindowDetached, FB::PluginWindowMac)
	PLUGIN_EVENT_MAP_CASCADE(tutorialplugin)
    END_PLUGIN_EVENT_MAP()
    virtual bool onWindowAttached(FB::AttachedEvent *evt, FB::PluginWindowMac*);
    virtual bool onWindowDetached(FB::DetachedEvent *evt, FB::PluginWindowMac*);
protected:
private:
    void* m_layer;
};
void glutDisplay (void); //this is implemented in the NITE code
@interface MyCAOpenGLLayer : CAOpenGLLayer {
    GLfloat m_angle;
}
@end
@implementation MyCAOpenGLLayer
- (id) init {
    if ([super init]) {
        m_angle = 0;
    }
    return self;
}
- (void)drawInCGLContext:(CGLContextObj)ctx pixelFormat:(CGLPixelFormatObj)pf forLayerTime:(CFTimeInterval)t displayTime:(const CVTimeStamp *)ts {
    //m_angle += 1;
    GLsizei width = CGRectGetWidth([self bounds]), height = CGRectGetHeight([self bounds]);
    GLfloat halfWidth = width / 2, halfHeight = height / 2;
    glViewport(0, 0, width, height);
	glutDisplay(); //let NITE draw it's stuff
    [super drawInCGLContext:ctx pixelFormat:pf forLayerTime:t displayTime:ts];
}
@end
tutorialpluginMac::tutorialpluginMac() : m_layer(NULL) {}
tutorialpluginMac::~tutorialpluginMac()
{
    if (m_layer) {
        [(CALayer*)m_layer removeFromSuperlayer];
        [(CALayer*)m_layer release];
        m_layer = NULL;
    }
}
bool tutorialpluginMac::onWindowAttached(FB::AttachedEvent* evt, FB::PluginWindowMac* wnd)
{
	cout << "tutorialpluginMac::onWindowAttached" << endl;
    if (FB::PluginWindowMac::DrawingModelCoreAnimation == wnd->getDrawingModel() ||
		FB::PluginWindowMac::DrawingModelInvalidatingCoreAnimation == wnd->getDrawingModel())
	{
        cout << " Setup CAOpenGL drawing. "<<endl;
        MyCAOpenGLLayer* layer = [MyCAOpenGLLayer new];
        layer.asynchronous = (FB::PluginWindowMac::DrawingModelInvalidatingCoreAnimation == wnd->getDrawingModel()) ? NO : YES;
        layer.autoresizingMask = kCALayerWidthSizable | kCALayerHeightSizable;
        layer.needsDisplayOnBoundsChange = YES;
        m_layer = layer;
        if (FB::PluginWindowMac::DrawingModelInvalidatingCoreAnimation == wnd->getDrawingModel())
            wnd->StartAutoInvalidate(1.0/30.0);
        [(CALayer*) wnd->getDrawingPrimitive() addSublayer:layer];
    }
    return tutorialplugin::onWindowAttached(evt,wnd);
}
bool tutorialpluginMac::onWindowDetached(FB::DetachedEvent* evt, FB::PluginWindowMac* wnd)
{
    return tutorialplugin::onWindowDetached(evt,wnd);
}

(You guys will have to fill in the gaps… includes, etc.)
This goes in a new file, a new subclass of the generic plugin, only for Mac. For windows, you should subclass again and create the OpenGL context using WIN32 API or equivalent.
CMakeLists.txt files are also affected. Check out the repo.

NITE OpenGL rendering

Now that the plugin will just draw whatever NITE is drawing, half the battle is done. So for the drawing code I took the simple NiPointViewer example from the NITE library (get it here).
But, since we have need no windows management in the OpenNI, again we can make everything more simple. I took the code exactly as it is, and changed really just a small bit.
I added

#undef USE_GLUT
#undef USE_GLES
, which pretty much makes that code compile to a very lean code (without window management etc.).
And I rescued the glOrtho call in glutDisplay()

//#ifdef USE_GLUT
glOrtho(0, mode.nXRes, mode.nYRes, 0, -1.0, 1.0);
#if defined(USE_GLES)
glOrthof(0, mode.nXRes, mode.nYRes, 0, -1.0, 1.0);
#endif

But the rest is pretty much identical.
One more thing, we should start the Kinect driver and OpenNI stack from somewhere in the plugin loading steps. In the main.cpp file from NITE I changed the main() function to kinect_main().
I did that by adding it here, in the generic plugin (not the Mac subclass because it should be called from all OSs):

bool tutorialplugin::onWindowAttached(FB::AttachedEvent *evt, FB::PluginWindow *)
{
    // The window is attached; act appropriately
	kinect_main(0, 0);
	cout << "tutorialplugin::onWindowAttached" << endl;
    return true;
}

It now will fire when a window is attached to the plugin. The OpenGL calls will start running after the OGL context is up and starts rendering in a loop.

Source and stuff

Get the source for the Kinect-FireBreath plugin at GitHub: https://github.com/royshil/KinectPlugin
This is how it looks:

Cool.
Roy