Tags

, , , , , , , , , , , , ,

In my last post, Searching the web using Augmented Reality, I built some robots out of Python code.

The robots used the following technologies for an augmented reality experience:

  • Blender to create a 3D robot
  • OpenCV computer vision to detect a 2D marker in my webcam
  • OpenGL graphics library to render the robot upon the marker

I also used Speech To Text and Text To Speech technologies to ask the robots to search the web and listen to their results.

Today I will add animation to my robots!

Blender animation: create and export

We are already using Blender to create our 3D robots. But how do we add animation?

Well, it turns out to be a breeze if we want to do something very simple. Let’s move Rocky Robot’s head up and down by adding a few keyframes to the animation timeline:

BlenderAnimationInOpenGL_Timeline

We’ve only got a 10 frame animation, with a keyframe at position 0 and 10 to set the robot’s head to its normal position (on the robot’s shoulders, that is). The keyframe at position 5 will lift the head clear of the shoulders. When we run the animation, the robot’s head bobs.

There are lots of animation tutorials online to help get started. I found Sardi Pax’s Basics of Animation a good watch.

Great, next question: how do we export our animation from Blender so that we can use it in OpenGL?

Seems like there are a number of options. We could export as Collada – which is an XML format – and perhaps make use of Assimp to import the model for use in OpenGL. Or we could write our own file format, using a Blender Python script. For now, I will use Wavefront OBJ format as this is the format already in use in my OpenGL application.

But the OBJ format does not support animation, I hear you cry! And, of course, you are dead right. So the trick is to export each frame as a single OBJ file and then stitch them together within OpenGL. When you export from Blender, make sure the animation flag is ticked:

BlenderAnimationInOpenGL_ExportOption

So that you end up with an OBJ and supporting MTL file for each frame:

BlenderAnimationInOpenGL_ExportFiles

Cool. Now, for Sporty Robot I used 20 frames of animation to make his head bob (using a keyframe at position 10 to set the head’s highest position). Rocky Robot’s head will shake faster at 10 frames – but that’s okay, because Rocky Robot is a rock ‘n’ roll headbanger!

OpenGL animation: import and render

So how do we import our OBJ and supporting MTL files into OpenGL?

My post Augmented Reality using OpenCV, OpenGL and Blender introduced the OBJFileLoader, which I will now wrap into a new class so as to handle the animation:

import os, glob
from objloader import *

class Robot:

    def __init__(self):
        self.frames = []
        self.frames_length = 0
        self.frame_index = 0
        self.is_detected = False
                 
    # load frames from directory
    def load(self, directory):
        os.chdir(directory)
        
        for file in glob.glob("*.obj"):
            self.frames.append(OBJ(file))

        os.chdir('..')
        self.frames_length = len(self.frames)

    # get next frame
    def next_frame(self):
        self.frame_index += 1

        if self.frame_index >= self.frames_length:
            self.frame_index = 0

        return self.frames[self.frame_index].gl_list

My Robot class will be responsible for loading and serving all our frames of animation.

The load method will be called when the OpenGL application starts. First, it moves to the directory where all our OBJ and MTL files are located. Next, we loop every OBJ file in the directory and load it into our frames list. Finally, we move back out of the directory and take a note of the number of frames for later use.

The next_frame method will be called whenever we want to draw our robot to screen – this will be many times a second if the 2D marker for the robot is detected in the webcam. All we need do is increment the frame index and grab the next frame from our frames list. If we are at the end of our frames list then we set the index back to the start, so as to put our animation on a loop.

Now, a word on loading the OBJ files. It can be slow, especially once our animation becomes more advanced. And, of course, we have a separate file for every frame of the robot when all that is actually changing between the frames is its head position. Clearly, there are some savings to be made on preprocessing and selective loading if we decide to stick with OBJ format. That said, once loaded, each frame is stored in a compiled OpenGL display list for lightning-quick rendering.

glNewList(self.gl_list, GL_COMPILE)

Great. Let’s now amend our OpenGL main program _init_gl method from the previous post so as to load our animation:

# load robots
self.rocky_robot.load('rocky_robot')
self.sporty_robot.load('sporty_robot')

And the _handle_glyphs method, to render each frame of animation:

if glyph_name == ROCKY_ROBOT:
    self.rocky_robot.is_detected = True
    glCallList(self.rocky_robot.next_frame())
elif glyph_name == SPORTY_ROBOT:
    self.sporty_robot.is_detected = True
    glCallList(self.sporty_robot.next_frame())

I’ll put all the code from the OpenGL main program at the foot of post. But for now, let’s sit back and watch in wonder as the robots bob their heads whilst searching the web.

Okay, the animation is fairly shit. But now that the mechanism is in place, I can start to go to town. Maybe even move an arm? Or smoke an ivory pipe?

Talking of pipes, anyone for a bit of Saltwash?

Ciao!

P.S.

I ran the code on my Windows 7 PC using Python Tools for Visual Studio.

Here’s the OpenGL main program I promised:

from OpenGL.GL import *
from OpenGL.GLUT import *
from OpenGL.GLU import *
import cv2
from PIL import Image
import numpy as np
from robot import Robot
from webcam import Webcam
from glyphs import Glyphs
from browser import Browser
from constants import *

class OpenGLRobots:
 
    # constants
    INVERSE_MATRIX = np.array([[ 1.0, 1.0, 1.0, 1.0],
                               [-1.0,-1.0,-1.0,-1.0],
                               [-1.0,-1.0,-1.0,-1.0],
                               [ 1.0, 1.0, 1.0, 1.0]])

    def __init__(self):
        # initialise robots
        self.rocky_robot = Robot()
        self.sporty_robot = Robot()

        # initialise webcam
        self.webcam = Webcam()

        # initialise glyphs
        self.glyphs = Glyphs()
        self.glyphs_cache = None

        # initialise browser
        self.browser = Browser()

        # initialise texture
        self.texture_background = None

    def _init_gl(self, Width, Height):
        glClearColor(0.0, 0.0, 0.0, 0.0)
        glClearDepth(1.0)
        glDepthFunc(GL_LESS)
        glEnable(GL_DEPTH_TEST)
        glShadeModel(GL_SMOOTH)
        glMatrixMode(GL_PROJECTION)
        glLoadIdentity()
        gluPerspective(33.7, 1.3, 0.1, 100.0)
        glMatrixMode(GL_MODELVIEW)
        
        # load robots
        self.rocky_robot.load('rocky_robot')
        self.sporty_robot.load('sporty_robot')
        
        # start threads
        self.webcam.start()
        self.browser.start()

        # assign texture
        glEnable(GL_TEXTURE_2D)
        self.texture_background = glGenTextures(1)

    def _draw_scene(self):
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        glLoadIdentity()

        # get image from webcam
        image = self.webcam.get_current_frame()

        # convert image to OpenGL texture format
        bg_image = cv2.flip(image, 0)
        bg_image = Image.fromarray(bg_image)     
        ix = bg_image.size[0]
        iy = bg_image.size[1]
        bg_image = bg_image.tostring("raw", "BGRX", 0, -1)
 
        # create background texture
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
        glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, bg_image)
        
        # draw background
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glPushMatrix()
        glTranslatef(0.0,0.0,-10.0)
        self._draw_background()
        glPopMatrix()

        # handle glyphs
        self.rocky_robot.is_detected = False
        self.sporty_robot.is_detected = False
        self._handle_glyphs(image)
       
        # handle browser
        if self.rocky_robot.is_detected:
            self.browser.load(ROCK)
        elif self.sporty_robot.is_detected:
            self.browser.load(SPORT)
        else:
            self.browser.halt()

        glutSwapBuffers()

    def _handle_glyphs(self, image):

        # attempt to detect glyphs
        glyphs = []

        try:
            glyphs = self.glyphs.detect(image)
        except Exception as ex: 
            print(ex)

        # manage glyphs cache
        if glyphs:
            self.glyphs_cache = glyphs
        elif self.glyphs_cache: 
            glyphs = self.glyphs_cache
            self.glyphs_cache = None
        else:
            return

        for glyph in glyphs:
            
            rvecs, tvecs, _, glyph_name = glyph

            # build view matrix
            rmtx = cv2.Rodrigues(rvecs)[0]

            view_matrix = np.array([[rmtx[0][0],rmtx[0][1],rmtx[0][2],tvecs[0]],
                                    [rmtx[1][0],rmtx[1][1],rmtx[1][2],tvecs[1]],
                                    [rmtx[2][0],rmtx[2][1],rmtx[2][2],tvecs[2]],
                                    [0.0       ,0.0       ,0.0       ,1.0    ]])

            view_matrix = view_matrix * self.INVERSE_MATRIX

            view_matrix = np.transpose(view_matrix)

            # load view matrix and draw cube
            glPushMatrix()
            glLoadMatrixd(view_matrix)

            if glyph_name == ROCKY_ROBOT:
                self.rocky_robot.is_detected = True
                glCallList(self.rocky_robot.next_frame())
            elif glyph_name == SPORTY_ROBOT:
                self.sporty_robot.is_detected = True
                glCallList(self.sporty_robot.next_frame())
            
            glColor3f(1.0, 1.0, 1.0)
            glPopMatrix()

    def _draw_background(self):
        # draw background
        glBegin(GL_QUADS)
        glTexCoord2f(0.0, 1.0); glVertex3f(-4.0, -3.0, 0.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 4.0, -3.0, 0.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 4.0,  3.0, 0.0)
        glTexCoord2f(0.0, 0.0); glVertex3f(-4.0,  3.0, 0.0)
        glEnd( )

    def main(self):
        # setup and run OpenGL
        glutInit()
        glutInitDisplayMode(GLUT_RGBA | GLUT_DOUBLE | GLUT_DEPTH)
        glutInitWindowSize(640, 480)
        glutInitWindowPosition(800, 400)
        self.window_id = glutCreateWindow("OpenGL Robots")
        glutDisplayFunc(self._draw_scene)
        glutIdleFunc(self._draw_scene)
        self._init_gl(640, 480)
        glutMainLoop()
 
# run an instance of OpenGL Robots
openGLRobots = OpenGLRobots()
openGLRobots.main()
Advertisements