Tags

, , , , , , , , , , , , ,

In my post Augmented Reality using OpenCV, OpenGL and Blender I made a deep and emotional promise to myself to:

  1. Create 3D objects using Blender
  2. Import 3D objects into OpenGL
  3. Detect 2D markers using OpenCV
  4. Draw 3D objects upon 2D markers using OpenGL

It all worked splendidly and I was emotionally enriched (for a while).

Sachin commented on the post, asking whether the code could be amended to do the following:

  1. Change the object on the marker when a key is pressed
  2. Do not rotate the object

Sensing an opportunity to be re-enriched, I updated the code from the previous post to meet these requirements.

We can use the GLUT glutKeyboardFunc function to detect when a key is pressed on the keyboard:

glutKeyboardFunc(self._key_pressed)

And introduce a new _key_pressed method to our OpenGLGlyphs class, to toggle a boolean isShapeSwitch class variable on keypress:

def _key_pressed (self, key, x, y):
    # toggle the shape switch
    self.isShapeSwitch = not self.isShapeSwitch

Now whenever a key is pressed, the class _handle_glyphs method will render a different shape on the marker:

if self.isShapeSwitch:
    glCallList(self.cone.gl_list)
else:
    glCallList(self.sphere.gl_list)

But what about the other requirement: Do not rotate the object

For this, we simply avoid updates on the view_matrix variable:

if(self.view_matrix.size == 0):
    self.view_matrix = np.array([[rmtx[0][0],rmtx[0][1],rmtx[0][2],tvecs[0]],
                                 [rmtx[1][0],rmtx[1][1],rmtx[1][2],tvecs[1]],
                                 [rmtx[2][0],rmtx[2][1],rmtx[2][2],tvecs[2]],
                                 [0.0       ,0.0       ,0.0       ,1.0     ]])
 
    self.view_matrix = self.view_matrix * self.INVERSE_MATRIX
 
    self.view_matrix = np.transpose(self.view_matrix)

Once our view_matrix variable has been rotated and translated on the first cycle of our app, we leave it well alone.

If we want to maintain translation, but not rotation, then just add an else clause:

else:
    self.view_matrix[3][0] = tvecs[0]
    self.view_matrix[3][1] = -(tvecs[1])
    self.view_matrix[3][2] = -(tvecs[2])

Okay, here’s the video to demonstrate that we can change the object on the marker when a key is pressed. And that the object does not rotate (or translate):

Superb. I am now emotionally enriched (for a while). If only Rodger was.

Ciao!

P.S.

Here’s the updated OpenGLGlyphs class (all the other code is available on previous post):

from OpenGL.GL import *
from OpenGL.GLUT import *
from OpenGL.GLU import *
import cv2
from PIL import Image
import numpy as np
from webcam import Webcam
from glyphs import Glyphs
from objloader import *
from constants import *
 
class OpenGLGlyphs:
  
    # constants
    INVERSE_MATRIX = np.array([[ 1.0, 1.0, 1.0, 1.0],
                               [-1.0,-1.0,-1.0,-1.0],
                               [-1.0,-1.0,-1.0,-1.0],
                               [ 1.0, 1.0, 1.0, 1.0]])
 
    def __init__(self):
        # initialise webcam and start thread
        self.webcam = Webcam()
        self.webcam.start()
 
        # initialise glyphs
        self.glyphs = Glyphs()
 
        # initialise shapes
        self.cone = None
        self.sphere = None
        self.isShapeSwitch = False;

        # initialise texture
        self.texture_background = None

        # initialise view matrix
        self.view_matrix = np.array([])

    def _init_gl(self, Width, Height):
        glClearColor(0.0, 0.0, 0.0, 0.0)
        glClearDepth(1.0)
        glDepthFunc(GL_LESS)
        glEnable(GL_DEPTH_TEST)
        glShadeModel(GL_SMOOTH)
        glMatrixMode(GL_PROJECTION)
        glLoadIdentity()
        gluPerspective(33.7, 1.3, 0.1, 100.0)
        glMatrixMode(GL_MODELVIEW)
         
        # assign shapes
        self.cone = OBJ('cone.obj')
        self.sphere = OBJ('sphere.obj')
 
        # assign texture
        glEnable(GL_TEXTURE_2D)
        self.texture_background = glGenTextures(1)
 
    def _draw_scene(self):
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        glLoadIdentity()
 
        # get image from webcam
        image = self.webcam.get_current_frame()
 
        # convert image to OpenGL texture format
        bg_image = cv2.flip(image, 0)
        bg_image = Image.fromarray(bg_image)     
        ix = bg_image.size[0]
        iy = bg_image.size[1]
        bg_image = bg_image.tobytes("raw", "BGRX", 0, -1)
  
        # create background texture
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
        glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, bg_image)
         
        # draw background
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glPushMatrix()
        glTranslatef(0.0,0.0,-10.0)
        self._draw_background()
        glPopMatrix()
 
        # handle glyphs
        image = self._handle_glyphs(image)
 
        glutSwapBuffers()
 
    def _handle_glyphs(self, image):

        # attempt to detect glyphs
        glyphs = []
 
        try:
            glyphs = self.glyphs.detect(image)
        except Exception as ex: 
            print(ex)
 
        if not glyphs: 
            return
 
        for glyph in glyphs:
             
            rvecs, tvecs, glyph_name = glyph
 
            # build view matrix
            rmtx = cv2.Rodrigues(rvecs)[0]
 
            if(self.view_matrix.size == 0):
                self.view_matrix = np.array([[rmtx[0][0],rmtx[0][1],rmtx[0][2],tvecs[0]],
                                             [rmtx[1][0],rmtx[1][1],rmtx[1][2],tvecs[1]],
                                             [rmtx[2][0],rmtx[2][1],rmtx[2][2],tvecs[2]],
                                             [0.0       ,0.0       ,0.0       ,1.0     ]])
 
                self.view_matrix = self.view_matrix * self.INVERSE_MATRIX
 
                self.view_matrix = np.transpose(self.view_matrix)
 
            # load view matrix and draw shape
            glPushMatrix()
            glLoadMatrixd(self.view_matrix)
 
            if self.isShapeSwitch:
                glCallList(self.cone.gl_list)
            else:
                glCallList(self.sphere.gl_list)
 
            glPopMatrix()
 
    def _draw_background(self):
        # draw background
        glBegin(GL_QUADS)
        glTexCoord2f(0.0, 1.0); glVertex3f(-4.0, -3.0, 0.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 4.0, -3.0, 0.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 4.0,  3.0, 0.0)
        glTexCoord2f(0.0, 0.0); glVertex3f(-4.0,  3.0, 0.0)
        glEnd( )

    def _key_pressed (self, key, x, y):
        # toggle the shape switch
        self.isShapeSwitch = not self.isShapeSwitch

    def main(self):
        # setup and run OpenGL
        glutInit()
        glutInitDisplayMode(GLUT_RGBA | GLUT_DOUBLE | GLUT_DEPTH)
        glutInitWindowSize(640, 480)
        glutInitWindowPosition(800, 400)
        self.window_id = glutCreateWindow("OpenGL Glyphs")
        glutDisplayFunc(self._draw_scene)
        glutIdleFunc(self._draw_scene)
        glutKeyboardFunc(self._key_pressed)
        self._init_gl(640, 480)
        glutMainLoop()
  
# run an instance of OpenGL Glyphs 
openGLGlyphs = OpenGLGlyphs()
openGLGlyphs.main()