Tags

, , , , , , , , , , , , ,

A few posts ago, I created some 3D Augmented Reality using OpenCV and Python. I was able to recognise a glyph and then project a cube from it:

It worked great, but OpenCV Computer Vision is not really geared to draw 3D graphics. Enter OpenGL Graphics Library. Let’s use OpenGL instead to project our cube, paving the way for more complex 3D objects.

Making some changes to the glyph recognition code

So what changes do we need to make to the previous post, to integrate it with OpenGL? Answer: not too much. We just need to yield our rotation and translation vectors, so that OpenGL can draw our cube.

To start with, we make a simple edit to our Effects class to pass back the vectors:

return image, rvecs, tvecs

Easy. Now we bubble these parameters up through our glyphfunctions.py:

return effects.render_cube(image, points)

Indeed, the only non-trivial task regards our main glyphs program – we need to turn it into a function (naming the file glyphs.py):

def detect_glyph(image):

We can put the function declaration in place of the while loop and drop support for Stage 1: Read an image from our webcam (as you can see, we are now passing the webcam image into the function).

The other amendments are an update to Stage 9: Add effects, to handle those bubbling parameters:

image, rvecs, tvecs = add_effects(image, approx.reshape(4, 2))
cv2.imshow('OpenCV Glyphs', image)
return rvecs, tvecs

And a change to Stage 10: Show augmented reality to pass back None if no glyph has been found:

return None, None

That’s it. For now, I will only attempt to detect a single glyph in the webcam image. Later, we can add back support for multiple glyphs.

Using OpenGL to draw the 3D object

Note: if you don’t have much experience with OpenGL, have a read through my post Augmented Reality using OpenCV and OpenGL (operations may be legacy-mode).

Now, the tricky bit is getting the OpenCV rotation and translation vectors into a format that OpenGL can understand. Djo1509’s response on the OpenCV Answers board was invaluable in explaining what needs to be done. Here’s my shot at it:

def _handle_glyph(self, image):

    # attempt to detect glyph
    rvecs = None
    tvecs = None

    try:
        rvecs, tvecs = detect_glyph(image)
    except Exception as ex: 
        print(ex)

    if rvecs == None or tvecs == None: 
        return

    # build view matrix
    rmtx = cv2.Rodrigues(rvecs)[0]

    view_matrix = np.array([[rmtx[0][0],rmtx[0][1],rmtx[0][2],tvecs[0]],
                            [rmtx[1][0],rmtx[1][1],rmtx[1][2],tvecs[1]],
                            [rmtx[2][0],rmtx[2][1],rmtx[2][2],tvecs[2]],
                            [0.0       ,0.0       ,0.0       ,1.0    ]])

    inverse_matrix = np.array([[ 1.0, 1.0, 1.0, 1.0],
                               [-1.0,-1.0,-1.0,-1.0],
                               [-1.0,-1.0,-1.0,-1.0],
                               [ 1.0, 1.0, 1.0, 1.0]])

    view_matrix = view_matrix * inverse_matrix

    view_matrix = np.transpose(view_matrix)

    # load view matrix and draw cube
    glBindTexture(GL_TEXTURE_2D, self.texture_cube)
    glPushMatrix()
    glLoadMatrixd(view_matrix)
    self._draw_cube()
    glPopMatrix()

First we attempt to get the rotation and translation vectors from OpenCV using the aforementioned detect_glyph function.

Next, we need to build the view matrix.

cv2.Rodrigues provides the rotation matrix, which slots into our view matrix alongside the translations.

We then apply an inverse matrix, as OpenGL’s Y and Z coordinates run in the opposite direction to OpenCV.

Finally we use np.transpose, as OpenCV’s matrixes are stored by row and OpenGL reads by column.

Now we can load our view matrix and draw our cube (the cube is set up to be in line with our OpenCV object points).

Another thing to look out for is the gluPerspective call:

gluPerspective(33.7, 1.3, 0.1, 100.0)

Programming Computer Vision with Python provides detail of how to use your OpenCV camera matrix to calculate the field of view and aspect parameters.

The full OpenGL code is provided at the foot of this post.

Demonstration

Time for a demo! The window in the top left is showing OpenCV projecting a cube from our glyph. The window in the bottom right is OpenGL doing the same thing:

I’m pleased with the results. There will no doubt be a bit of fine-tuning once I get more of a handle on OpenGL, but it’s a promising start.

Now to celebrate with a few pages of The Inscrutable Diaries Of Rodger Saltwash.

Ciao!

P.S.

The Stack Overflow contributions from ChronoTrigger and nkint were also a great help.

I ran the code on my Windows 7 PC using Python Tools for Visual Studio.

Here’s my OpenGL Glyphs class:

from OpenGL.GL import *
from OpenGL.GLUT import *
from OpenGL.GLU import *
import cv2
from PIL import Image
import numpy as np
from webcam import Webcam
from glyphs import *

class OpenGLGlyphs:

    # constants
    INVERSE_MATRIX = np.array([[ 1.0, 1.0, 1.0, 1.0],
                               [-1.0,-1.0,-1.0,-1.0],
                               [-1.0,-1.0,-1.0,-1.0],
                               [ 1.0, 1.0, 1.0, 1.0]])
 
    def __init__(self):
        # initialise webcam and start thread
        self.webcam = Webcam()
        self.webcam.start()

        # textures
        self.texture_background = None
        self.texture_cube = None

    def _init_gl(self, Width, Height):
        glClearColor(0.0, 0.0, 0.0, 0.0)
        glClearDepth(1.0)
        glDepthFunc(GL_LESS)
        glEnable(GL_DEPTH_TEST)
        glShadeModel(GL_SMOOTH)
        glMatrixMode(GL_PROJECTION)
        glLoadIdentity()
        gluPerspective(33.7, 1.3, 0.1, 100.0)
        glMatrixMode(GL_MODELVIEW)
        
        # enable textures
        glEnable(GL_TEXTURE_2D)
        self.texture_background = glGenTextures(1)
        self.texture_cube = glGenTextures(1)

        # create cube texture 
        image = Image.open("devil.jpg")
        ix = image.size[0]
        iy = image.size[1]
        image = image.tostring("raw", "RGBX", 0, -1)

        glBindTexture(GL_TEXTURE_2D, self.texture_cube)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
        glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, image)

    def _draw_scene(self):
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        glLoadIdentity()

        # get image from webcam
        image = self.webcam.get_current_frame()

        # convert image to OpenGL texture format
        bg_image = cv2.flip(image, 0)
        bg_image = Image.fromarray(bg_image)     
        ix = bg_image.size[0]
        iy = bg_image.size[1]
        bg_image = bg_image.tostring("raw", "BGRX", 0, -1)
 
        # create background texture
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
        glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, bg_image)
        
        # draw background
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glPushMatrix()
        glTranslatef(0.0,0.0,-10.0)
        self._draw_background()
        glPopMatrix()

        # handle glyph
        image = self._handle_glyph(image)

        glutSwapBuffers()

    def _handle_glyph(self, image):

        # attempt to detect glyph
        rvecs = None
        tvecs = None

        try:
            rvecs, tvecs = detect_glyph(image)
        except Exception as ex: 
            print(ex)

        if rvecs == None or tvecs == None: 
            return

        # build view matrix
        rmtx = cv2.Rodrigues(rvecs)[0]

        view_matrix = np.array([[rmtx[0][0],rmtx[0][1],rmtx[0][2],tvecs[0]],
                                [rmtx[1][0],rmtx[1][1],rmtx[1][2],tvecs[1]],
                                [rmtx[2][0],rmtx[2][1],rmtx[2][2],tvecs[2]],
                                [0.0       ,0.0       ,0.0       ,1.0    ]])

        view_matrix = view_matrix * self.INVERSE_MATRIX

        view_matrix = np.transpose(view_matrix)

        # load view matrix and draw cube
        glBindTexture(GL_TEXTURE_2D, self.texture_cube)
        glPushMatrix()
        glLoadMatrixd(view_matrix)
        self._draw_cube()
        glPopMatrix()

    def _draw_cube(self):
        # draw cube
        glBegin(GL_QUADS)

        glTexCoord2f(0.0, 0.0); glVertex3f( 0.0,  0.0,  0.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 1.0,  0.0,  0.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0,  0.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 0.0,  1.0,  0.0)

        glTexCoord2f(1.0, 0.0); glVertex3f( 0.0,  0.0, -1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 0.0,  1.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f( 1.0,  0.0, -1.0)

        glTexCoord2f(0.0, 1.0); glVertex3f( 0.0,  1.0, -1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f( 0.0,  1.0,  0.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 1.0,  1.0,  0.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)

        glTexCoord2f(1.0, 1.0); glVertex3f( 0.0,  0.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 1.0,  0.0, -1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f( 1.0,  0.0,  0.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 0.0,  0.0,  0.0)

        glTexCoord2f(1.0, 0.0); glVertex3f( 1.0,  0.0, -1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 1.0,  1.0,  0.0)
        glTexCoord2f(0.0, 0.0); glVertex3f( 1.0,  0.0,  0.0)

        glTexCoord2f(0.0, 0.0); glVertex3f( 0.0,  0.0, -1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 0.0,  0.0,  0.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 0.0,  1.0,  0.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 0.0,  1.0, -1.0)

        glEnd()

    def _draw_background(self):
        # draw background
        glBegin(GL_QUADS)
        glTexCoord2f(0.0, 1.0); glVertex3f(-4.0, -3.0, 0.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 4.0, -3.0, 0.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 4.0,  3.0, 0.0)
        glTexCoord2f(0.0, 0.0); glVertex3f(-4.0,  3.0, 0.0)
        glEnd( )

    def main(self):
        # setup and run OpenGL
        glutInit()
        glutInitDisplayMode(GLUT_RGBA | GLUT_DOUBLE | GLUT_DEPTH)
        glutInitWindowSize(640, 480)
        glutInitWindowPosition(800, 400)
        self.window_id = glutCreateWindow("OpenGL Glyphs")
        glutDisplayFunc(self._draw_scene)
        glutIdleFunc(self._draw_scene)
        self._init_gl(640, 480)
        glutMainLoop()
 
# run an instance of OpenGL Glyphs 
openGLGlyphs = OpenGLGlyphs()
openGLGlyphs.main()