Categories
code graphics opencv programming

Just a simple Laplacian pyramid blender using OpenCV [w/code]

I want to share a small piece of code to do Laplacian Blending using OpenCV. It’s one of the most basic and canonical methods of image blending, and is a must exercise for any computer graphics student.

Well basically it’s a matter of creating two Laplacian pyramids of both images, and a Gaussian pyramid of the mask.
Then we blend the pyramids into one, and collapse the resulting pyramid into the blended image.

#include "opencv2/opencv.hpp"
using namespace cv;
class LaplacianBlending {
private:
	Mat_<Vec3f> left;
	Mat_<Vec3f> right;
	Mat_<float> blendMask;
	vector<Mat_<Vec3f> > leftLapPyr,rightLapPyr,resultLapPyr;
	Mat leftSmallestLevel, rightSmallestLevel, resultSmallestLevel;
	vector<Mat_<Vec3f> > maskGaussianPyramid; //masks are 3-channels for easier multiplication with RGB
	int levels;
	void buildPyramids() {
		buildLaplacianPyramid(left,leftLapPyr,leftSmallestLevel);
		buildLaplacianPyramid(right,rightLapPyr,rightSmallestLevel);
		buildGaussianPyramid();
	}
	void buildGaussianPyramid() {
		assert(leftLapPyr.size()>0);
		maskGaussianPyramid.clear();
		Mat currentImg;
		cvtColor(blendMask, currentImg, CV_GRAY2BGR);
		maskGaussianPyramid.push_back(currentImg); //highest level
		currentImg = blendMask;
		for (int l=1; l<levels+1; l++) {
			Mat _down;
			if (leftLapPyr.size() > l) {
				pyrDown(currentImg, _down, leftLapPyr[l].size());
			} else {
				pyrDown(currentImg, _down, leftSmallestLevel.size()); //smallest level
			}
			Mat down;
			cvtColor(_down, down, CV_GRAY2BGR);
			maskGaussianPyramid.push_back(down);
			currentImg = _down;
		}
	}
	void buildLaplacianPyramid(const Mat& img, vector<Mat_<Vec3f> >& lapPyr, Mat& smallestLevel) {
		lapPyr.clear();
		Mat currentImg = img;
		for (int l=0; l<levels; l++) {
			Mat down,up;
			pyrDown(currentImg, down);
			pyrUp(down, up, currentImg.size());
			Mat lap = currentImg - up;
			lapPyr.push_back(lap);
			currentImg = down;
		}
		currentImg.copyTo(smallestLevel);
	}
	Mat_<Vec3f> reconstructImgFromLapPyramid() {
		Mat currentImg = resultSmallestLevel;
		for (int l=levels-1; l>=0; l--) {
			Mat up;
			pyrUp(currentImg, up, resultLapPyr[l].size());
			currentImg = up + resultLapPyr[l];
		}
		return currentImg;
	}
	void blendLapPyrs() {
		resultSmallestLevel = leftSmallestLevel.mul(maskGaussianPyramid.back()) +
									rightSmallestLevel.mul(Scalar(1.0,1.0,1.0) - maskGaussianPyramid.back());
		for (int l=0; l<levels; l++) {
			Mat A = leftLapPyr[l].mul(maskGaussianPyramid[l]);
			Mat antiMask = Scalar(1.0,1.0,1.0) - maskGaussianPyramid[l];
			Mat B = rightLapPyr[l].mul(antiMask);
			Mat_<Vec3f> blendedLevel = A + B;
			resultLapPyr.push_back(blendedLevel);
		}
	}
public:
	LaplacianBlending(const Mat_<Vec3f>& _left, const Mat_<Vec3f>& _right, const Mat_<float>& _blendMask, int _levels):
	left(_left),right(_right),blendMask(_blendMask),levels(_levels)
	{
		assert(_left.size() == _right.size());
		assert(_left.size() == _blendMask.size());
		buildPyramids();
		blendLapPyrs();
	};
	Mat_<Vec3f> blend() {
		return reconstructImgFromLapPyramid();
	}
};
Mat_<Vec3f> LaplacianBlend(const Mat_<Vec3f>& l, const Mat_<Vec3f>& r, const Mat_<float>& m) {
	LaplacianBlending lb(l,r,m,4);
	return lb.blend();
}
int main(int argc, char** argv) {
   Mat l8u = imread("left.png");
   Mat r8u = imread("right.png");
   Mat_<Vec3f> l; l8u.convertTo(l,CV_32F,1.0/255.0);
   Mat_<Vec3f> r; r8u.convertTo(r,CV_32F,1.0/255.0);
   Mat_<float> m(l.rows,l.cols,0.0);
   m(Range::all(),Range(0,m.cols/2)) = 1.0;
   Mat_<Vec3f> blend = LaplacianBlend(l, r, m);
   imshow("blended",blend);
   waitKey(0);
}

To use, simply call the function LaplacianBlend with your two images and your mask, and the result will be returned.
Here’s something I did with it:

Enjoy
Roy.

13 replies on “Just a simple Laplacian pyramid blender using OpenCV [w/code]”

Hi,
I tried using the code,
and i get an exception on this line:
cvtColor(blendMask, currentImg, CV_GRAY2BGR);
any idea what could cause the problem?
i’m kinda new to open cv

Hi,
I’m currently experimenting with your code and I have a few questions if you could answer:
does your class accept different kinds of images like CV_8UC3 and CV_32FC3;
and if not, what are the relevant portions in the code where the image being a certain type is essential?
I will be grateful if you can help me with this.

@Paulo Almeida
It actually only likes Mat_ which is CV_32FC3
You can make a simple conversion from CV_8UC3 to CV_32FC3 in the main function LaplacianBlend, or simply create another LaplacianBlend function that accepts Mat_ (CV_8UC3)
It would be nice if the whole thing will be templated, and then accept any kind of image without conversion

can you show me how I can call this in the main function? thanks.. I’am getting assertion failure.. So i was wondering, maybe there is a mistake on what I am doing.. Thanks and regards.

can you show how to use this in main, im getting assertion failure!
hope to hear something soon! thank you!

Thank you! when i run the program the command line window pops up when the code is at pyrDown(currentImg, down), i guess it just doesn’t do anything with it and just exits.. could that be because I am using opencv 2.2 ?

Even i’am facing the same problem as jack is facing,i’am also using open CV 2.2.
I would be highly thankful if you could help me out with it.

@k_dee @jack
To get a better idea of the problem I would need some debug info.. like an exception thrown, etc.
Anyway I recommend updating to OpenCV 2.3+ (2.4 is latest at the moment)
R.

Hi,
I’ve been using your code and it works perfectly. I have a doubt, when I have an image where the part of interest is sorrounded by black pixels I got a frame of strong Laplacian lines that I wouldn’t like to have (blending doesn’t look well). How do I manage this situation ?. Thanks for your help.

I get this error, can you tell me how to fix it?
here is the line of your code cause the error
Mat_ l; l8u.convertTo(l,CV_32F,1.0/255.0);
Here is the error message:
OpenCV Error: Assertion failed (!fixedType() || ((Mat*)obj)->type() == mtype) in unknown function.
PS: I am using Visual Studio 2012 and OpenCV 2.4.4

Leave a Reply

Your email address will not be published. Required fields are marked *