Hi
Another quicky on how to use Kinect (libfreenect) with OpenCV 2.1. I already saw people do it, but havn’t seen code.
UPDATE (12/29): OpenKinect posted very good C++ code of using libfreenect with OpenCV2.X APIs: here it is. Plus, their git repo now has a very clean C code: here it is.
So here it goes
Before I started, I got libfreenect off the OpenKinect git repo: https://github.com/OpenKinect/libfreenect
Which comes bundeled with “glview.c”, an example of how to use it with OpenGL, that I take some stuff from.
So get past the hurdle of compiling, and if glview runs – you’re home free. Play around and familiarize yourself with the code, this is going to be fast…
The only tricky thing is that OpenCV wants the “imshow”s to be on the main thread. This I learned empirically, form trial and error. But, in the glview.c example they do the “freenect_process_events” on the main thread, and the rendering is done on the GL thread. I flipped things around a bit.
So in the main function I have the initialization stuff and a pthread_create to make a freenect thread:
Mat depthMat(Size(640,480),CV_16UC1),
rgbMat(Size(640,480),CV_8UC3,Scalar(0));
pthread_t fnkt_thread;
freenect_device *f_dev;
pthread_mutex_t buf_mutex = PTHREAD_MUTEX_INITIALIZER;
freenect_context *f_ctx;
pthread_cond_t frame_cond = PTHREAD_COND_INITIALIZER;
int main(int argc, char **argv)
{
int res;
g_argc = argc;
g_argv = argv;
if (freenect_init(&f_ctx, NULL) < 0) {
printf("freenect_init() failed\n");
return 1;
}
freenect_set_log_level(f_ctx, FREENECT_LOG_INFO);
int nr_devices = freenect_num_devices (f_ctx);
printf ("Number of devices found: %d\n", nr_devices);
int user_device_number = 0;
if (argc > 1)
user_device_number = atoi(argv[1]);
if (nr_devices < 1)
return 1;
if (freenect_open_device(f_ctx, &f_dev, user_device_number) < 0) {
printf("Could not open device\n");
return 1;
}
freenect_set_tilt_degs(f_dev,freenect_angle);
freenect_set_led(f_dev,LED_RED);
freenect_set_depth_callback(f_dev, depth_cb);
freenect_set_rgb_callback(f_dev, rgb_cb);
freenect_set_rgb_format(f_dev, FREENECT_FORMAT_RGB);
freenect_set_depth_format(f_dev, FREENECT_FORMAT_11_BIT);
freenect_start_depth(f_dev);
freenect_start_rgb(f_dev);
res = pthread_create(&fnkt_thread, NULL, freenect_threadfunc, NULL);
if (res) {
printf("pthread_create failed\n");
return 1;
}
while (!die) {
fr++;
imshow("rgb", rgbMat);
depthMat.convertTo(depthf, CV_8UC1, 255.0/2048.0);
imshow("depth",depthf);
char k = cvWaitKey(5);
if( k == 27 ) break;
}
printf("-- done!\n");
destroyWindow("rgb");
destroyWindow("depth");
pthread_join(fnkt_thread, NULL);
pthread_exit(NULL);
}
The freenect thread is simply
void *freenect_threadfunc(void* arg) {
cout << "freenect thread"<<endl;
while(!die && freenect_process_events(f_ctx) >= 0 ) {}
cout << "freenect die"<<endl;
return NULL;
}
And the two callbacks need also to write into the OpenCV buffers:
void depth_cb(freenect_device *dev, freenect_depth *depth, uint32_t timestamp)
{
pthread_mutex_lock(&buf_mutex);
//copy to ocv buf...
memcpy(depthMat.data, depth, FREENECT_DEPTH_SIZE);
got_frames++;
pthread_cond_signal(&frame_cond);
pthread_mutex_unlock(&buf_mutex);
}
void rgb_cb(freenect_device *dev, freenect_pixel *rgb, uint32_t timestamp)
{
pthread_mutex_lock(&buf_mutex);
got_frames++;
//copy to ocv_buf..
memcpy(rgbMat.data, rgb, FREENECT_RGB_SIZE);
pthread_cond_signal(&frame_cond);
pthread_mutex_unlock(&buf_mutex);
}
And that’s all you’ll need.
Now go crazy with your concoction of computer vision algorithms!
Roy.
25 replies on “Kinect and OpenCV 2.1”
Hi there, greetings from Finland.
Could you post full code? kthxbai
Hi Joni
Actually the code is all up here in the post, there are no other important things in the file… Just copy-paste and make sure everything is declared, follow the compiler errors and fix them.
Also, add these include directives:
#include
using namespace std;
#include <cv.h>
#include <highgui.h>
#include <cvaux.h>
using namespace cv;
#include <iostream>
#include <stdio.h>
#include <string.h>
#include <libusb.h>
#include "libfreenect.h"
#include <pthread.h>
Good luck,
Roy
Thanks for the reply.
I don’t even have the device yet – setting up things beforehand. But as I’m an OpenCV n00b, this example helped a lot.
I was not able to compile your example, I’m sorry.
I used the following code for Kinect+OpenCV but I do not see RGB or DEPTH images…. even if all callback are working.
Have you please any idea how to get right OpenCV working?
#————-CODE———–#
/*
g++ -O2 -g -Wall -fmessage-length=0 -bind_at_load `pkg-config –cflags opencv` -I../libusb/libusb/libusb-1.0 KinectOpenCVFreeBox3.c -o KinectOpenCVFreeBox3 `pkg-config –libs opencv` -lfreenect
*/
#include “highgui.h”
#include
#include
#include
#include “libusb.h”
#include “libfreenect.h”
#include
#define CV_NO_BACKWARD_COMPATIBILITY
#include
#include
#define FREENECTOPENCV_WINDOW_D “Depthimage”
#define FREENECTOPENCV_WINDOW_N “Normalimage”
#define FREENECTOPENCV_RGB_DEPTH 3
#define FREENECTOPENCV_DEPTH_DEPTH 1
#define FREENECTOPENCV_RGB_WIDTH 640
#define FREENECTOPENCV_RGB_HEIGHT 480
#define FREENECTOPENCV_DEPTH_WIDTH 640
#define FREENECTOPENCV_DEPTH_HEIGHT 480
#define CV_NO_BACKWARD_COMPATIBILITY
IplImage* frame = 0;
IplImage *image = 0, *hsv = 0, *hue = 0, *mask = 0, *backproject = 0, *histimg = 0;
CvCapture* capture = 0;
//
cv::Mat depthMat(cvSize(640,480),CV_16UC1),
rgbMat(cvSize(640,480),CV_8UC3,cvScalar(0));
pthread_t fnkt_thread;
pthread_mutex_t buf_mutex = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t frame_cond = PTHREAD_COND_INITIALIZER;
int got_frames;
//
IplImage* depthimg = 0;
IplImage* brgbimg = 0;
IplImage* rgbimg = 0;
IplImage* tempimg = 0;
pthread_mutex_t mutex_depth = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t mutex_rgb = PTHREAD_MUTEX_INITIALIZER;
pthread_t cv_thread;
freenect_context *f_ctx;
freenect_device *f_dev;
int freenect_angle = 0;
int freenect_led;
volatile int die = 0;
pthread_cond_t gl_frame_cond = PTHREAD_COND_INITIALIZER;
int got_rgb = 0;
int got_depth = 0;
// callback for depthimage, called by libfreenect
void depth_cb(freenect_device *dev, void *depth, uint32_t timestamp)
{
// freenect_depth *depth = v_depth;
cv::Mat depth8;
cv::Mat mydepth = cv::Mat( FREENECTOPENCV_DEPTH_WIDTH,FREENECTOPENCV_DEPTH_HEIGHT, CV_16UC1, depth);
mydepth.convertTo(depth8, CV_8UC1, 1.0/4.0);
pthread_mutex_lock( &mutex_depth );
memcpy(depthimg->imageData, depth8.data, 640*480);
pthread_mutex_unlock( &mutex_depth );
// printf(“\ndepth_cb…\n”);
}
// callback for rgbimage, called by libfreenect
void rgb_cb(freenect_device *dev, freenect_pixel *rgb, uint32_t timestamp)
{
pthread_mutex_lock( &mutex_rgb );
freenect_set_rgb_buffer(f_dev, rgb);
memcpy(rgbimg->imageData, rgb, FREENECTOPENCV_DEPTH_WIDTH*FREENECTOPENCV_DEPTH_HEIGHT*FREENECTOPENCV_RGB_DEPTH);
pthread_mutex_unlock( &mutex_rgb );
// printf(“\nrgb_cb…\n”);
}
/*
* thread for displaying the opencv content
*/
void *cv_threadfunc (void *ptr) {
cvNamedWindow( FREENECTOPENCV_WINDOW_D, CV_WINDOW_AUTOSIZE );
cvNamedWindow( FREENECTOPENCV_WINDOW_N, CV_WINDOW_AUTOSIZE );
depthimg = cvCreateImage(cvSize(FREENECTOPENCV_DEPTH_WIDTH, FREENECTOPENCV_DEPTH_HEIGHT), IPL_DEPTH_8U, FREENECTOPENCV_DEPTH_DEPTH);
rgbimg = cvCreateImage(cvSize(FREENECTOPENCV_RGB_WIDTH, FREENECTOPENCV_RGB_HEIGHT), IPL_DEPTH_8U, FREENECTOPENCV_RGB_DEPTH);
tempimg = cvCreateImage(cvSize(FREENECTOPENCV_RGB_WIDTH, FREENECTOPENCV_RGB_HEIGHT), IPL_DEPTH_8U, FREENECTOPENCV_RGB_DEPTH);
freenect_start_depth(f_dev);
freenect_start_rgb(f_dev);
printf(“\nstart…\n”);
while(1)
{
// printf(“\nloop…\n”);
pthread_mutex_lock( &mutex_depth );
cvCvtColor(depthimg,tempimg,CV_GRAY2BGR);
cvCvtColor(tempimg,tempimg,CV_HSV2BGR);
cvShowImage(FREENECTOPENCV_WINDOW_D,tempimg);
pthread_mutex_unlock( &mutex_depth );
pthread_mutex_lock( &mutex_rgb );
cvCvtColor(rgbimg,tempimg,CV_BGR2RGB);
cvShowImage(FREENECTOPENCV_WINDOW_N, tempimg);
pthread_mutex_unlock( &mutex_rgb );
if( cvWaitKey( 15 )==27 )
break;
}
printf(“\nshutting down streams…\n”);
freenect_stop_depth(f_dev);
freenect_stop_rgb(f_dev);
freenect_close_device(f_dev);
freenect_shutdown(f_ctx);
printf(“– done!\n”);
return NULL;
pthread_exit(NULL);
}
int main(int argc, char **argv)
{
int res = 0;
int die = 0;
printf(“Kinect camera test\n”);
if (freenect_init(&f_ctx, NULL) 1)
user_device_number = atoi(argv[1]);
if (nr_devices < 1)
return 1;
if (freenect_open_device(f_ctx, &f_dev, user_device_number) = 0 );
printf(“init done\n”);
}#————-–––––––––––––––––––#
… I’m sorry missing include are (due to ):
#include stdio.h
#include string.h
#include math.h
#include pthread.h
#include cv.h
#include highgui.h
I haven’t succeeded in running OpenCV
imshoworcvShowImageon a different thread (like you do).So I put the libfreenect on a thread, and the OpenCV on the main thread – then it worked fine.
I believe it’s a problem with OpenCV’s GUI threading.
Thanks Roy, now it’s working well….
Thanks for the example!
I’m fairly new to OpenCV…
I’m looking to dump some image data to get my own cal of the depth sensor. Is it possible to convert the 11-bit input up to a properly-scaled CV_16UC1 or a scaled CV_32FC1 image (rather than scale to 8-bit) to preserve the data?
I’ve tried to use the convertTo method to properly scale my 16-bit image but my output looks the same as the 8-bit. I also don’t know of any image formats supporting single-precision floating points (CV_32FC1). It would be very useful if I could use cvSaveImage() could save a CV_32FC1 image. Any insight would be appreciated!
Thanks in advance!
Hi,
I am interested in using openCV with OpenKinect too. I tried from the source code given by Claudio but got some errors. As I’m a noob with thread, is it possible to see your implementation using libfreenect on a thread and OpenCV on the main thread ?
Thanks in advance
Hey I’m trying to use libfreenect sources in a larger project with cmake, how do i properly include the library, after
In the CMakeLists.txt i added :
include_directories(/usr/local/include/libfreenect)
target_link_libraries(test_webcam dear-webcam libreenect)
So it sees the heards in
#include
but when running i get
:: error: cannot find -llibreenect
This site is a great resource thanks!
Try linking just “freenect” and not “libfreenect”, the linker adds the “lib” prefix on its own.
R.
That worked! Except now its giving me some errors from the header file:
/usr/local/include/libfreenect/libfreenect.hpp:59: error: ‘Freenect::FreenectTiltState::m_state’ will be initialized after
/usr/local/include/libfreenect/libfreenect.hpp:57: error: ‘freenect_tilt_status_code Freenect::FreenectTiltState::m_code’
/usr/local/include/libfreenect/libfreenect.hpp:46: error: when initialized here
/home/mike/dear/dear/modules/webcam/tests/test_webcam.cpp:: In constructor ‘MyFreenectDevice::MyFreenectDevice(freenect_context*, int)’:
/home/mike/dear/dear/modules/webcam/tests/test_webcam.cpp:95: error: ‘MyFreenectDevice::m_new_depth_frame’ will be initialized after
/home/mike/dear/dear/modules/webcam/tests/test_webcam.cpp:30: error: when initialized here
@Robert: AFAIK, you can’t store 32F images directly using cvSaveImage. What I do is breaking down the float into its 4 bytes and store them in a w x 4*h 8UC1 image (where w and h are the original width and height of the image), then save it as png. The other option is to save it as a binary file.
HI Roy,
I am a newbiee,
can you pls explain me what do u mean by “I haven’t succeeded in running OpenCV imshow or cvShowImage on a different thread (like you do).
So I put the libfreenect on a thread, and the OpenCV on the main thread – then it worked fine.”
Can you pls explain me how to get this kinect and opencv working.
t123,
You better stick to the examples by OpenKinect:
http://openkinect.org/wiki/C%2B%2BOpenCvExample
https://github.com/OpenKinect/libfreenect/blob/master/wrappers/opencv/cvdemo.c
Roy.
HI rob,
when i run the C# wrapper from OpenKinect, I get the following error, do you have any idea abt it :
Unable to find an entry point named ‘freenect_init’ in DLL ‘freenect’.
sorry i meant Hi Roy …
Hi t123,
did u get any solution out of the problem stated above “freenect_init’ in DLL ‘freenect’.” if Yes, then kindly share solution of it . I am badly stuck in it.
Thanks.
Hi,
There is no motor manager in freenect lib, isn’t it?
hello,
Thanks for the example. This site is great!
I could compile the openkinect I am able to run the samples like glview but I could not compile the sample code in http://openkinect.org/wiki/C%2B%2BOpenCvExample with the given makefile. I am getting libfreenect.h no such file or directory message. I will be happy if you can help me solve this issue.
By the way, this solves some of the problems for the c++ wrapper related to “is not a template” error from another page which I cant put here as the system takes it as spam 🙂
that I have to change:
> Freenect::Freenect freenect;
> MyFreenectDevice& device = freenect.createDevice(0);
> into:
> Freenect::Freenect freenect;
> MyFreenectDevice& device = freenect.createDevice(0);
but I still could not solve the “FREENECT_DEPTH_11BIT_SIZE” not found problem
Sorry for 3 consecutive posts but I was feeling so frustrated after long hours now I solved it here is how:
I replaced the
m_buffer_depth(FREENECT_DEPTH_11BIT_SIZE),m_buffer_rgb(FREENECT_VIDEO_RGB_SIZE)
with
m_buffer_depth((640*480*sizeof(uint16_t))),m_buffer_rgb((640*480*3)
and yay! it solved other problems as well with my previous posts.
good luck with kinect to everyone.
Thanks for your post.
The openkinect c++ wrapper does not include (#include ) which was causing frame to be dropped. By having a look through your code I was able to use abit of trial and error. So thank you very much!!!!!!
Troy
Hello,
Thanks for the link. I am a beginner in this field. I tried to follow your instructions and built libfreenect using cmake and then tried to compile the program given by c++opencv example.
but i get following errors. could you please tell me my mistake. thanks.
10 IntelliSense: cannot open source file “libfreenect.h”
11 IntelliSense: cannot open source file “pthread.h”
12 IntelliSense: identifier “freenect_raw_tilt_state”
13 IntelliSense: identifier “freenect_tilt_status_code”
14 IntelliSense: identifier “freenect_raw_tilt_state”
15 IntelliSense: identifier “freenect_context” is undefined
16 IntelliSense: identifier “freenect_led_options”
17 IntelliSense: identifier “freenect_video_format”
18 IntelliSense: identifier “freenect_resolution”
19 IntelliSense: identifier “FREENECT_RESOLUTION_MEDIUM”
20 IntelliSense: identifier “freenect_video_format”
21 IntelliSense: identifier “freenect_resolution”
22 IntelliSense: identifier “freenect_depth_format”
23 IntelliSense: identifier “freenect_resolution”
24 IntelliSense: identifier “FREENECT_RESOLUTION_MEDIUM”
25 IntelliSense: identifier “freenect_depth_format”
26 IntelliSense: identifier “freenect_resolution”
27 IntelliSense: identifier “uint32_t” is undefined
28 IntelliSense: identifier “uint32_t” is undefined
29 IntelliSense: identifier “freenect_device”
30 IntelliSense: identifier “freenect_video_format”
31 IntelliSense: identifier “freenect_depth_format”
32 IntelliSense: identifier “freenect_resolution”
33 IntelliSense: identifier “freenect_resolution”
34 IntelliSense: identifier “freenect_device” i
35 IntelliSense: identifier “uint32_t”
36 IntelliSense: identifier “freenect_device”
37 IntelliSense: identifier “uint32_t”
38 IntelliSense: identifier “freenect_context”
39 IntelliSense: identifier “pthread_t”
40 IntelliSense: cannot open source file “sched.h”
41 IntelliSense: identifier “freenect_context”
42 IntelliSense: identifier “FREENECT_DEPTH_11BIT”
43 IntelliSense: identifier “FREENECT_VIDEO_RGB”
44 IntelliSense: identifier “uint8_t”
45 IntelliSense: identifier “rgb”
46 IntelliSense: “uint8_t”
47 IntelliSense: identifier “uint16_t”
48 IntelliSense: identifier “depth”
49 IntelliSense: “uint16_t”
50 IntelliSense: identifier “uint8_t”
51 IntelliSense: identifier “uint8_t”
52 IntelliSense: identifier “uint16_t”
very good tutorial that help me in the process of setup Kinect sensor on Windows. Also, I add a link to this tutorial into an article with many more tutorial related to Kinect sensor