Tags

, , , , , , , , , , , , ,

In my last post, Augmented Reality using OpenCV, OpenGL and Blender, I was able to use a webcam to detect optical glyphs. What’s an optical glyph? Here’s one:

glyph_01

Once detected, I can project a 3D object from its surface. I projected a cone and a sphere.

In this post I will rotate a glyph in the webcam, thereby rotating the 3D object upon it. I will use a cube as my 3D object, with each side of the cube a different colour so we can easily see it turn.

Look again at the glyph above. You will see that the pattern changes if we rotate it 90 degrees, 180 degrees or 270 degrees. All we need do is match the pattern and rotate our 3D object accordingly. Easy!

Blender

Okay, let’s use Blender to create my cube. I assign a different material to each face of my cube, using Diffuse to set its colour. My cube’s Location and Scale settings are all 0.55. I export four versions of the cube in OBJ format, with the Y rotation at 0 degrees, 90 degrees, 180 degrees and 270 degrees.

OpenCV

Next, I need to update my glyphs.py file from the previous post, so as to yield the rotation value of the detected glyph. For example, if the glyph pattern spied in my webcam is at 0 degrees then the rotation value is 0. If the glyph has been turned on its side and therefore at 90 degrees then the rotation value is 1. If the glyph is upside down at 180 degrees then the rotation value is 2. Finally, if the glyph has been turned clockwise three times then the rotation value is 3.

I’ve amended the Python code to handle the rotation value from the glyph database:

glyph_found, glyph_rotation, glyph_name = match_glyph_pattern(glyph_pattern)

And once OpenCV computer vision has finished detecting glyphs, I ensure that the rotation value is included in the output:

glyphs.append([rvecs, tvecs, glyph_rotation, glyph_name])

OpenGL

We have four versions of our cube exported from Blender, each version matching a rotation value. Let’s load them into OpenGL using the import code from the previous post:

# assign cube
self.cube = [OBJ('cube_0.obj'), 
             OBJ('cube_1.obj'), 
             OBJ('cube_2.obj'), 
             OBJ('cube_3.obj')]

And we can update the OpenGL code to collect the rotation value yielded from OpenCV glyph detection:

rvecs, tvecs, glyph_rotation, _ = glyph

But how do we select the correct cube to draw, based on the rotation value? Turns out to be rather easy:

glCallList(self.cube[glyph_rotation].gl_list)

We simply select the version of the cube from our array using the rotation value! So if the rotation value of the glyph is 3 (270 degrees) then we select the last cube in the array, which is cube_3.obj.

Finally, I’ve updated the OpenGL code to include a glyphs cache. The idea being that if we fail to detect a glyph in our webcam then we render the previous cube instead. The glyphs cache will help reduce flickering during our augmented reality experience.

# manage glyphs cache
if glyphs:
    self.glyphs_cache = glyphs
elif self.glyphs_cache: 
    glyphs = self.glyphs_cache
    self.glyphs_cache = None
else:
    return

All the OpenGL graphics library code can be found at the foot of this post.

Demonstration

Phew! Enough of the code, let’s see cube rotation in action:

Hurray! Our augmented reality experience can now handle rotating objects.

I’m so happy, I will treat myself to a few pages of The Inscrutable Diaries Of Rodger Saltwash. Any questions, just leave a comment.

Ciao!

P.S.

I ran the code on my Windows 7 PC using Python Tools for Visual Studio.

Here’s my main OpenGL program:

from OpenGL.GL import *
from OpenGL.GLUT import *
from OpenGL.GLU import *
import cv2
from PIL import Image
import numpy as np
from webcam import Webcam
from glyphs import Glyphs
from objloader import *

class OpenGLGlyphs:
 
    # constants
    INVERSE_MATRIX = np.array([[ 1.0, 1.0, 1.0, 1.0],
                               [-1.0,-1.0,-1.0,-1.0],
                               [-1.0,-1.0,-1.0,-1.0],
                               [ 1.0, 1.0, 1.0, 1.0]])

    def __init__(self):
        # initialise webcam and start thread
        self.webcam = Webcam()
        self.webcam.start()

        # initialise glyphs
        self.glyphs = Glyphs()
        self.glyphs_cache = None

        # initialise cube
        self.cube = None 

        # initialise texture
        self.texture_background = None

    def _init_gl(self, Width, Height):
        glClearColor(0.0, 0.0, 0.0, 0.0)
        glClearDepth(1.0)
        glDepthFunc(GL_LESS)
        glEnable(GL_DEPTH_TEST)
        glShadeModel(GL_SMOOTH)
        glMatrixMode(GL_PROJECTION)
        glLoadIdentity()
        gluPerspective(33.7, 1.3, 0.1, 100.0)
        glMatrixMode(GL_MODELVIEW)
        
        # assign cube
        self.cube = [OBJ('cube_0.obj'), 
                     OBJ('cube_1.obj'), 
                     OBJ('cube_2.obj'), 
                     OBJ('cube_3.obj')]

        # assign texture
        glEnable(GL_TEXTURE_2D)
        self.texture_background = glGenTextures(1)

    def _draw_scene(self):
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        glLoadIdentity()

        # get image from webcam
        image = self.webcam.get_current_frame()

        # convert image to OpenGL texture format
        bg_image = cv2.flip(image, 0)
        bg_image = Image.fromarray(bg_image)     
        ix = bg_image.size[0]
        iy = bg_image.size[1]
        bg_image = bg_image.tostring("raw", "BGRX", 0, -1)
 
        # create background texture
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
        glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, bg_image)
        
        # draw background
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glPushMatrix()
        glTranslatef(0.0,0.0,-10.0)
        self._draw_background()
        glPopMatrix()

        # handle glyphs
        image = self._handle_glyphs(image)

        glutSwapBuffers()

    def _handle_glyphs(self, image):

        # attempt to detect glyphs
        glyphs = []

        try:
            glyphs = self.glyphs.detect(image)
        except Exception as ex: 
            print(ex)

        # manage glyphs cache
        if glyphs:
            self.glyphs_cache = glyphs
        elif self.glyphs_cache: 
            glyphs = self.glyphs_cache
            self.glyphs_cache = None
        else:
            return

        for glyph in glyphs:
            
            rvecs, tvecs, glyph_rotation, _ = glyph

            # build view matrix
            rmtx = cv2.Rodrigues(rvecs)[0]

            view_matrix = np.array([[rmtx[0][0],rmtx[0][1],rmtx[0][2],tvecs[0]],
                                    [rmtx[1][0],rmtx[1][1],rmtx[1][2],tvecs[1]],
                                    [rmtx[2][0],rmtx[2][1],rmtx[2][2],tvecs[2]],
                                    [0.0       ,0.0       ,0.0       ,1.0    ]])

            view_matrix = view_matrix * self.INVERSE_MATRIX

            view_matrix = np.transpose(view_matrix)

            # load view matrix and draw cube
            glPushMatrix()
            glLoadMatrixd(view_matrix)
            glCallList(self.cube[glyph_rotation].gl_list)
            glColor3f(1.0, 1.0, 1.0)
            glPopMatrix()

    def _draw_background(self):
        # draw background
        glBegin(GL_QUADS)
        glTexCoord2f(0.0, 1.0); glVertex3f(-4.0, -3.0, 0.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 4.0, -3.0, 0.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 4.0,  3.0, 0.0)
        glTexCoord2f(0.0, 0.0); glVertex3f(-4.0,  3.0, 0.0)
        glEnd( )

    def main(self):
        # setup and run OpenGL
        glutInit()
        glutInitDisplayMode(GLUT_RGBA | GLUT_DOUBLE | GLUT_DEPTH)
        glutInitWindowSize(640, 480)
        glutInitWindowPosition(800, 400)
        self.window_id = glutCreateWindow("OpenGL Glyphs")
        glutDisplayFunc(self._draw_scene)
        glutIdleFunc(self._draw_scene)
        self._init_gl(640, 480)
        glutMainLoop()
 
# run an instance of OpenGL Glyphs 
openGLGlyphs = OpenGLGlyphs()
openGLGlyphs.main()