3D Graphics, Animation, Augmented Reality, Blender, Computer Vision, Object Detection, OpenCV, OpenGL, Pose Estimation, PyOpenGL, Python, Python Tools for Visual Studio, Speech To Text, Text To Speech
In my last post, Searching the web using Augmented Reality, I built some robots out of Python code.
The robots used the following technologies for an augmented reality experience:
- Blender to create a 3D robot
- OpenCV computer vision to detect a 2D marker in my webcam
- OpenGL graphics library to render the robot upon the marker
I also used Speech To Text and Text To Speech technologies to ask the robots to search the web and listen to their results.
Today I will add animation to my robots!
Blender animation: create and export
We are already using Blender to create our 3D robots. But how do we add animation?
Well, it turns out to be a breeze if we want to do something very simple. Let’s move Rocky Robot’s head up and down by adding a few keyframes to the animation timeline:
We’ve only got a 10 frame animation, with a keyframe at position 0 and 10 to set the robot’s head to its normal position (on the robot’s shoulders, that is). The keyframe at position 5 will lift the head clear of the shoulders. When we run the animation, the robot’s head bobs.
There are lots of animation tutorials online to help get started. I found Sardi Pax’s Basics of Animation a good watch.
Great, next question: how do we export our animation from Blender so that we can use it in OpenGL?
Seems like there are a number of options. We could export as Collada – which is an XML format – and perhaps make use of Assimp to import the model for use in OpenGL. Or we could write our own file format, using a Blender Python script. For now, I will use Wavefront OBJ format as this is the format already in use in my OpenGL application.
But the OBJ format does not support animation, I hear you cry! And, of course, you are dead right. So the trick is to export each frame as a single OBJ file and then stitch them together within OpenGL. When you export from Blender, make sure the animation flag is ticked:
So that you end up with an OBJ and supporting MTL file for each frame:
Cool. Now, for Sporty Robot I used 20 frames of animation to make his head bob (using a keyframe at position 10 to set the head’s highest position). Rocky Robot’s head will shake faster at 10 frames – but that’s okay, because Rocky Robot is a rock ‘n’ roll headbanger!
OpenGL animation: import and render
So how do we import our OBJ and supporting MTL files into OpenGL?
My post Augmented Reality using OpenCV, OpenGL and Blender introduced the OBJFileLoader, which I will now wrap into a new class so as to handle the animation:
import os, glob from objloader import * class Robot: def __init__(self): self.frames =  self.frames_length = 0 self.frame_index = 0 self.is_detected = False # load frames from directory def load(self, directory): os.chdir(directory) for file in glob.glob("*.obj"): self.frames.append(OBJ(file)) os.chdir('..') self.frames_length = len(self.frames) # get next frame def next_frame(self): self.frame_index += 1 if self.frame_index >= self.frames_length: self.frame_index = 0 return self.frames[self.frame_index].gl_list
My Robot class will be responsible for loading and serving all our frames of animation.
The load method will be called when the OpenGL application starts. First, it moves to the directory where all our OBJ and MTL files are located. Next, we loop every OBJ file in the directory and load it into our frames list. Finally, we move back out of the directory and take a note of the number of frames for later use.
The next_frame method will be called whenever we want to draw our robot to screen – this will be many times a second if the 2D marker for the robot is detected in the webcam. All we need do is increment the frame index and grab the next frame from our frames list. If we are at the end of our frames list then we set the index back to the start, so as to put our animation on a loop.
Now, a word on loading the OBJ files. It can be slow, especially once our animation becomes more advanced. And, of course, we have a separate file for every frame of the robot when all that is actually changing between the frames is its head position. Clearly, there are some savings to be made on preprocessing and selective loading if we decide to stick with OBJ format. That said, once loaded, each frame is stored in a compiled OpenGL display list for lightning-quick rendering.
Great. Let’s now amend our OpenGL main program _init_gl method from the previous post so as to load our animation:
# load robots self.rocky_robot.load('rocky_robot') self.sporty_robot.load('sporty_robot')
And the _handle_glyphs method, to render each frame of animation:
if glyph_name == ROCKY_ROBOT: self.rocky_robot.is_detected = True glCallList(self.rocky_robot.next_frame()) elif glyph_name == SPORTY_ROBOT: self.sporty_robot.is_detected = True glCallList(self.sporty_robot.next_frame())
I’ll put all the code from the OpenGL main program at the foot of post. But for now, let’s sit back and watch in wonder as the robots bob their heads whilst searching the web.
Okay, the animation is fairly shit. But now that the mechanism is in place, I can start to go to town. Maybe even move an arm? Or smoke an ivory pipe?
Talking of pipes, anyone for a bit of Saltwash?
I ran the code on my Windows 7 PC using Python Tools for Visual Studio.
Here’s the OpenGL main program I promised:
from OpenGL.GL import * from OpenGL.GLUT import * from OpenGL.GLU import * import cv2 from PIL import Image import numpy as np from robot import Robot from webcam import Webcam from glyphs import Glyphs from browser import Browser from constants import * class OpenGLRobots: # constants INVERSE_MATRIX = np.array([[ 1.0, 1.0, 1.0, 1.0], [-1.0,-1.0,-1.0,-1.0], [-1.0,-1.0,-1.0,-1.0], [ 1.0, 1.0, 1.0, 1.0]]) def __init__(self): # initialise robots self.rocky_robot = Robot() self.sporty_robot = Robot() # initialise webcam self.webcam = Webcam() # initialise glyphs self.glyphs = Glyphs() self.glyphs_cache = None # initialise browser self.browser = Browser() # initialise texture self.texture_background = None def _init_gl(self, Width, Height): glClearColor(0.0, 0.0, 0.0, 0.0) glClearDepth(1.0) glDepthFunc(GL_LESS) glEnable(GL_DEPTH_TEST) glShadeModel(GL_SMOOTH) glMatrixMode(GL_PROJECTION) glLoadIdentity() gluPerspective(33.7, 1.3, 0.1, 100.0) glMatrixMode(GL_MODELVIEW) # load robots self.rocky_robot.load('rocky_robot') self.sporty_robot.load('sporty_robot') # start threads self.webcam.start() self.browser.start() # assign texture glEnable(GL_TEXTURE_2D) self.texture_background = glGenTextures(1) def _draw_scene(self): glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT) glLoadIdentity() # get image from webcam image = self.webcam.get_current_frame() # convert image to OpenGL texture format bg_image = cv2.flip(image, 0) bg_image = Image.fromarray(bg_image) ix = bg_image.size iy = bg_image.size bg_image = bg_image.tostring("raw", "BGRX", 0, -1) # create background texture glBindTexture(GL_TEXTURE_2D, self.texture_background) glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST) glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST) glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, bg_image) # draw background glBindTexture(GL_TEXTURE_2D, self.texture_background) glPushMatrix() glTranslatef(0.0,0.0,-10.0) self._draw_background() glPopMatrix() # handle glyphs self.rocky_robot.is_detected = False self.sporty_robot.is_detected = False self._handle_glyphs(image) # handle browser if self.rocky_robot.is_detected: self.browser.load(ROCK) elif self.sporty_robot.is_detected: self.browser.load(SPORT) else: self.browser.halt() glutSwapBuffers() def _handle_glyphs(self, image): # attempt to detect glyphs glyphs =  try: glyphs = self.glyphs.detect(image) except Exception as ex: print(ex) # manage glyphs cache if glyphs: self.glyphs_cache = glyphs elif self.glyphs_cache: glyphs = self.glyphs_cache self.glyphs_cache = None else: return for glyph in glyphs: rvecs, tvecs, _, glyph_name = glyph # build view matrix rmtx = cv2.Rodrigues(rvecs) view_matrix = np.array([[rmtx,rmtx,rmtx,tvecs], [rmtx,rmtx,rmtx,tvecs], [rmtx,rmtx,rmtx,tvecs], [0.0 ,0.0 ,0.0 ,1.0 ]]) view_matrix = view_matrix * self.INVERSE_MATRIX view_matrix = np.transpose(view_matrix) # load view matrix and draw cube glPushMatrix() glLoadMatrixd(view_matrix) if glyph_name == ROCKY_ROBOT: self.rocky_robot.is_detected = True glCallList(self.rocky_robot.next_frame()) elif glyph_name == SPORTY_ROBOT: self.sporty_robot.is_detected = True glCallList(self.sporty_robot.next_frame()) glColor3f(1.0, 1.0, 1.0) glPopMatrix() def _draw_background(self): # draw background glBegin(GL_QUADS) glTexCoord2f(0.0, 1.0); glVertex3f(-4.0, -3.0, 0.0) glTexCoord2f(1.0, 1.0); glVertex3f( 4.0, -3.0, 0.0) glTexCoord2f(1.0, 0.0); glVertex3f( 4.0, 3.0, 0.0) glTexCoord2f(0.0, 0.0); glVertex3f(-4.0, 3.0, 0.0) glEnd( ) def main(self): # setup and run OpenGL glutInit() glutInitDisplayMode(GLUT_RGBA | GLUT_DOUBLE | GLUT_DEPTH) glutInitWindowSize(640, 480) glutInitWindowPosition(800, 400) self.window_id = glutCreateWindow("OpenGL Robots") glutDisplayFunc(self._draw_scene) glutIdleFunc(self._draw_scene) self._init_gl(640, 480) glutMainLoop() # run an instance of OpenGL Robots openGLRobots = OpenGLRobots() openGLRobots.main()