, , , , , , , , , , , , ,

Arkwood moaned, ‘Your virtual world is sooooo sterile.’

But it was true. My C++ Visual Studio application uses OpenGL graphics library and the Oculus SDK for Windows to render objects to the Oculus Rift virtual reality headset. But those objects are not of nature’s kind. They are cold objects such as plastic bottles and electronic alarm clocks.

So I grew some trees in Blender, the 3D creation suite:

The Blender Tutorial – How to make cute Cartoon Trees helped out.

Now just to place them in my virtual world, behind the TV set:

Arkwood said, ‘That’s better. But it would be better still if those virtual trees could reflect the real world around me.’

He is a demanding bastard. Still, I am not one for licking the plump lips of defeat. Here goes…

Now, you may have noticed that the TV set is displaying a stream of images from the webcam attached to my PC. What if I can detect lots of green colour in those images of the real world – maybe I can reflect that in my virtual world by making the trees more green.

I used OpenCV computer vision in my GreenDetector struct, to determine whether the amount of green in each webcam image was above a threshold:

struct GreenDetector
	bool IsThreadEnabled;
	bool IsGreen;
	const int Threshold;

	GreenDetector() : IsThreadEnabled(true), IsGreen(false), Threshold(15000) {}

	void DetectorThread() {

		while (IsThreadEnabled) {

			if (CapturedImage.empty()) continue;

			Mat hsv;
			cvtColor(CapturedImage, hsv, COLOR_BGR2HSV);

			Mat mask;
			inRange(hsv, Scalar(58, 50, 50), Scalar(78, 255, 255), mask);

			int colourCount = countNonZero(mask);

			if (colourCount > Threshold) {
				IsGreen = true;
			else {
				IsGreen = false;



The CapturedImage is our webcam image. We convert it to HSV and then determine how many pixels are within a green range. We count those pixels and check if it breaches the threshold. If so, we set our IsGreen flag to true.

We run our structs DetectorThread function in a thread, so not to block the main application from rendering to the Rift headset:

GreenDetector * greenDetector = new GreenDetector();
std::thread dt(&GreenDetector::DetectorThread, greenDetector);

Splendid. Now all we need do is tell the fragment shader whether the mesh we are about to render needs a boost of green, so as to reflect the real world:

if (MeshName.find("Leaves") != string::npos &&
	greenDetector->IsGreen) {

	glUniform1i(glGetUniformLocation(Fill->program, "uGreenBoost"), 1);

You’ll see that we are only interested in boosting ‘Leaves’ meshes (not the tree trunks), and only if our detector has found lots of green in the webcam images of the real world.

Finally, we add some logic to the fragment shader to boost the colour green:

"		if (uGreenBoost == true) {\n"
"			FragColor = oColor + vec4(0,0.3,0,0);\n"
"		} else {\n"
"			FragColor = oColor;\n"
"		}\n"

Okay, so let’s give it a whirl. I put a nice green plant in front of the webcam in the real world and watch as the trees in the virtual world turn a brighter green:

You can see the plant in the webcam images on the TV screen. And you can see that the virtual trees are now boosted to an almost fluorescent green!

Arkwood huffed. ‘That green is too bright. The trees are not a very natural colour.’

I rolled up my sleeve and punched him square in the testicles. His face turned a bright red, then puce with pain.