Tags

, , , , , , , , , , , , ,

Whac-A-Mole is an arcade game, whereby the player has to hit plastic moles with a mallet as they randomly pop out of holes. How about we create the game on our PC with nothing more than some Python code and a smattering of augmented reality? And a dangerous steel hammer.

Detecting the hammer

So how are we going to detect our use of a hammer?

Well, our webcam can take snaps of our violent use of the tool. We can then use OpenCV motion detection to determine which part of the snap we are wielding the hammer in.

Let’s divide the snap into six cells as thus:

whacamole_cells

And here’s OpenCV detecting motion as we swing the hammer:

whacamole_threshold

All we need do is determine which of the six cells has the most motion in it. That will be the cell our hammer is hitting.

Let’s walk through the code of our Detection class:

import cv2
import numpy as np

class Detection(object):

    THRESHOLD = 1500

    def __init__(self, image):
        self.previous_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    def get_active_cell(self, image):
        # obtain motion between previous and current image
        current_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        delta = cv2.absdiff(self.previous_gray, current_gray)
        threshold_image = cv2.threshold(delta, 25, 255, cv2.THRESH_BINARY)[1]

        # debug
        cv2.imshow('OpenCV Threshold', threshold_image)

        # store current image
        self.previous_gray = current_gray

        # set cell height and width
        height, width = threshold_image.shape[:2]
        cell_height = height/2
        cell_width = width/3

        # store motion level for each cell
        cells = np.array([0, 0, 0, 0, 0, 0])
        cells[0] = cv2.countNonZero(threshold_image[0:cell_height, 0:cell_width])
        cells[1] = cv2.countNonZero(threshold_image[0:cell_height, cell_width:cell_width*2])
        cells[2] = cv2.countNonZero(threshold_image[0:cell_height, cell_width*2:width])
        cells[3] = cv2.countNonZero(threshold_image[cell_height:height, 0:cell_width])
        cells[4] = cv2.countNonZero(threshold_image[cell_height:height, cell_width:cell_width*2])
        cells[5] = cv2.countNonZero(threshold_image[cell_height:height, cell_width*2:width])

        # obtain the most active cell
        top_cell =  np.argmax(cells)

        # return the most active cell, if threshold met
        if(cells[top_cell] >= self.THRESHOLD):
            return top_cell
        else:
            return None

We feed the current image from our webcam into the get_active_cell method.

The first job of the method is to determine the differences between our current and previous webcam images, yielding a black and white threshold image (the white pixels in the threshold image are where motion is detected).

Next, we need to store the current image, so that it can be used as the previous image the next time we try to detect motion.

Now we get to the task of splitting our threshold image into six cells, and checking which of the cells has the most motion in it. We do this by counting all the white pixels in each cell.

Once we have identified the most active cell, we return it – but only if it meets a certain threshold value (after all, a few specks of white does not constitute a hammer!).

Whacking the mole

So we have used OpenCV motion detection to determine which part of the webcam snap the hammer is in. Let’s now use OpenGL to draw a mole and play the game.

The OpenGL Whac-A-Mole class is quite a size, so I will split it down into methods. First up, the __init__ method:

from OpenGL.GL import *
from OpenGL.GLUT import *
from OpenGL.GLU import *
import cv2
from PIL import Image
import random
from webcam import Webcam
from detection import Detection
 
class WhacAMole:
 
    def __init__(self):
        # initialise webcam and start thread
        self.webcam = Webcam()
        self.webcam.start()

        # initialise detection with first webcam frame
        self.detection = Detection(self.webcam.get_current_frame()) 

        # initialise game settings
        self.time_limit = 40
        self.time_left = self.time_limit
        self.score = 0
        self.hammer_cell = None
        self.mole_cell = None
        self.mole_coords = [(-3,2),(0,2),(3,2),(-3,-2),(0,-2),(3,-2)]

        # initialise axis
        self.x_axis = 0.0
        self.z_axis = 0.0

        # initialise textures
        self.texture_background = None
        self.texture_mole = None

We initialize our Webcam class, starting a thread for fetching its current image (the Webcam class can be found at the foot of this post).

We initialize our Detection class with the first webcam image.

Next, we initialize some game settings. We need a time limit in which to whack the mole (the time limit is also used between moles appearing). We need to keep track of time left, which will be a countdown from the time limit. Obviously, we need to keep a tally of our score! Notice how we also track the cell in the image where our hammer and mole reside. Finally, we have mole coordinates, which are needed to draw our 3D mole.

The last thing we do is initialize the axis that our mole will spin around, as well as the textures for rendering the mole and background.

def _init_gl(self, Width, Height):
    glClearColor(0.0, 0.0, 0.0, 0.0)
    glClearDepth(1.0)
    glDepthFunc(GL_LESS)
    glEnable(GL_DEPTH_TEST)
    glShadeModel(GL_SMOOTH)
    glMatrixMode(GL_PROJECTION)
    glLoadIdentity()
    gluPerspective(45.0, float(Width)/float(Height), 0.1, 100.0)
    glMatrixMode(GL_MODELVIEW)
        
    # enable textures
    glEnable(GL_TEXTURE_2D)
    self.texture_background = glGenTextures(1)
    self.texture_mole = glGenTextures(1)

    # create mole texture 
    image = Image.open("mole.jpg")
    ix = image.size[0]
    iy = image.size[1]
    image = image.tostring("raw", "RGBX", 0, -1)

    glBindTexture(GL_TEXTURE_2D, self.texture_mole)
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
    glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, image)

Our _init_gl method is executed only once, at the start of our game. Alongside some initial OpenGL settings, we enable our textures – loading and binding an image of a mole to the texture_mole texture.

def _draw_scene(self):
    # handle any hammer whack
    self._handle_hammer()
 
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
    glLoadIdentity()

    # draw background
    glBindTexture(GL_TEXTURE_2D, self.texture_background)
    glPushMatrix()
    glTranslatef(0.0,0.0,-11.2)
    self._draw_background()
    glPopMatrix()

    # draw mole
    if self.mole_cell != None:
        glColor4f(1.0, 1.0, 1.0, 0.8)
        glBlendFunc(GL_SRC_ALPHA, GL_ONE)
        glEnable(GL_BLEND)
        glDisable(GL_DEPTH_TEST)

        glBindTexture(GL_TEXTURE_2D, self.texture_mole)
        mole_coord = self.mole_coords[self.mole_cell]

        glPushMatrix()
        glTranslatef(mole_coord[0],mole_coord[1],-9.0)
        glRotatef(self.x_axis,1.0,0.0,0.0)
        glRotatef(0.0,0.0,1.0,0.0)
        glRotatef(self.z_axis,0.0,0.0,1.0)
        self._draw_mole()
        glPopMatrix()

        glDisable(GL_BLEND)
        glEnable(GL_DEPTH_TEST)

        # update axis
        self.x_axis = self.x_axis - 2
        self.z_axis = self.z_axis - 2

    # determine game state
    if self.mole_cell == None and self.time_left == 0:
            
        # ready to show new mole!
        self.mole_cell = random.randrange(6)
        self.time_left = self.time_limit
        
    elif self.mole_cell != None:
            
        if self.time_left == 0:
                
            # mole not whacked in time! score drops to 0 
            self.score = 0
            self.mole_cell = None
            self.time_left = self.time_limit

        elif self.hammer_cell == self.mole_cell:
                
            # mole whacked! score increased by 1
            self.score += 1
            self.mole_cell = None
            self.time_left = self.time_limit

    # decrease game time
    self.time_left -= 1

    glutSwapBuffers()

Right. The _draw_scene method is where all the shit happens. Good shit, that is. It is executed every time the gaming window is redrawn.

First we handle any use of a hammer – namely, working out which cell of the image our steel tool is in. More on the _handle_hammer method later.

Next we draw the background to our window, which is the current snap from our webcam. After all, if we want an augmented reality experience then our mole needs to be sitting in the real world!

Now we draw our mole – but only if it has been given a cell to pop up in. We use OpenGL to blend our mole into the current image from our webcam. Notice how we use the mole’s current cell to fetch the coordinates needed to position it. We rotate our mole. Then we draw it. The axis are updated so that our mole rotates further, the next time the window is drawn.

Okay, so now we have got to the juicy bit – figuring out whether we have whacked our mole with a hammer! The first thing we determine is whether we need to show a new mole – if so, we assign it a random cell and reset the time limit. However, if the mole is already in a cell then we check whether the time has run out – bad news if it has, cos we drop the score to zero. But what instead if the hammer cell is the same as the mole cell? Hurray! That means that we have indeed whacked our furry little rodent and our score increases by one.

Whatever the outcome of our game state, we need to decrease the time left. The game never stops, after all.

def _handle_hammer(self):
    # get image from webcam 
    image = self.webcam.get_current_frame()
        
    # draw current score on image
    cv2.putText(image, "Score: {}".format(self.score), (25,25), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,255,0), 2)

    # detect any hammer whack in image
    self.hammer_cell = self.detection.get_active_cell(image)

    # convert image to OpenGL texture format
    image = cv2.flip(image, 0)
    gl_image = Image.fromarray(image)     
    ix = gl_image.size[0]
    iy = gl_image.size[1]
    gl_image = gl_image.tostring("raw", "BGRX", 0, -1)
 
    # create background texture
    glBindTexture(GL_TEXTURE_2D, self.texture_background)
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
    glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, gl_image)

As promised, here’s the _handle_hammer method.

We fetch an image from the webcam and draw the current score on it.

We feed the image to our Detection class, to try to locate the hammer in one of the image cells.

Finally, we convert the webcam image from OpenCV to OpenGL format and bind it to our texture_background texture.

def _draw_background(self):
    # draw background
    glBegin(GL_QUADS)
    glTexCoord2f(0.0, 1.0); glVertex3f(-4.0, -3.0, 4.0)
    glTexCoord2f(1.0, 1.0); glVertex3f( 4.0, -3.0, 4.0)
    glTexCoord2f(1.0, 0.0); glVertex3f( 4.0,  3.0, 4.0)
    glTexCoord2f(0.0, 0.0); glVertex3f(-4.0,  3.0, 4.0)
    glEnd( )

def _draw_mole(self):
    # draw mole
    glBegin(GL_QUADS)
    glTexCoord2f(0.0, 0.0); glVertex3f(-1.0, -1.0,  1.0)
    glTexCoord2f(1.0, 0.0); glVertex3f( 1.0, -1.0,  1.0)
    glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0,  1.0)
    glTexCoord2f(0.0, 1.0); glVertex3f(-1.0,  1.0,  1.0)
    glTexCoord2f(1.0, 0.0); glVertex3f(-1.0, -1.0, -1.0)
    glTexCoord2f(1.0, 1.0); glVertex3f(-1.0,  1.0, -1.0)
    glTexCoord2f(0.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
    glTexCoord2f(0.0, 0.0); glVertex3f( 1.0, -1.0, -1.0)
    glTexCoord2f(0.0, 1.0); glVertex3f(-1.0,  1.0, -1.0)
    glTexCoord2f(0.0, 0.0); glVertex3f(-1.0,  1.0,  1.0)
    glTexCoord2f(1.0, 0.0); glVertex3f( 1.0,  1.0,  1.0)
    glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
    glTexCoord2f(1.0, 1.0); glVertex3f(-1.0, -1.0, -1.0)
    glTexCoord2f(0.0, 1.0); glVertex3f( 1.0, -1.0, -1.0)
    glTexCoord2f(0.0, 0.0); glVertex3f( 1.0, -1.0,  1.0)
    glTexCoord2f(1.0, 0.0); glVertex3f(-1.0, -1.0,  1.0)
    glTexCoord2f(1.0, 0.0); glVertex3f( 1.0, -1.0, -1.0)
    glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
    glTexCoord2f(0.0, 1.0); glVertex3f( 1.0,  1.0,  1.0)
    glTexCoord2f(0.0, 0.0); glVertex3f( 1.0, -1.0,  1.0)
    glTexCoord2f(0.0, 0.0); glVertex3f(-1.0, -1.0, -1.0)
    glTexCoord2f(1.0, 0.0); glVertex3f(-1.0, -1.0,  1.0)
    glTexCoord2f(1.0, 1.0); glVertex3f(-1.0,  1.0,  1.0)
    glTexCoord2f(0.0, 1.0); glVertex3f(-1.0,  1.0, -1.0)
    glEnd()

The _draw_background and _draw_mole methods are fairly self-explanatory. Our background is a square with the current webcam image as a texture. Our mole texture is applied to a cube shape (yes, moles aren’t really cube-shaped, but this is a first stab).

    def main(self):
        # setup and run OpenGL
        glutInit()
        glutInitDisplayMode(GLUT_RGBA | GLUT_DOUBLE | GLUT_DEPTH)
        glutInitWindowSize(640, 480)
        glutInitWindowPosition(800, 400)
        glutCreateWindow("OpenGL Whac-A-Mole")
        glutDisplayFunc(self._draw_scene)
        glutIdleFunc(self._draw_scene)
        self._init_gl(640, 480)
        glutMainLoop()
 
# run an instance of Whac-A-Mole 
whacAMole = WhacAMole()
whacAMole.main()

main is our entry point for all our code. We create our gaming window. We assign our _draw_scene method to each sketch of the window. We execute our aforementioned _init_gl method. Then we “Get Ready” for a game of Whac-A-Mole!

Indeed, we only need two lines of code outside of our Whac-A-Mole class. The first line creates an instance of the class. The second line executes its main method.

Demonstration

That’s the code. Time for a demo:

Fantastic! I’m really squashing those moles with my steel hammer. Don’t tell the RSPCA.

No doubt the game could do with a few licks of paint. We could award a better score for quicker whacks. And a high score table. Still, playing Whac-A-Mole with augmented reality is fun. Just hope no one peers into my house and spies me smashing a hammer into thin air. I don’t want the police round again.

Ciao!

P.S.

The OpenCV Python tutorials are a great place to start learning about computer vision.

The NeHe tutorials are a great help in understanding OpenGL (though operations may be legacy-mode).

I ran the code on my Windows 7 PC using Python Tools for Visual Studio.

Here’s the Webcam class:

import cv2
from threading import Thread
  
class Webcam:
  
    def __init__(self):
        self.video_capture = cv2.VideoCapture(0)
        self.current_frame = self.video_capture.read()[1]
          
    # create thread for capturing images
    def start(self):
        Thread(target=self._update_frame, args=()).start()
  
    def _update_frame(self):
        while(True):
            self.current_frame = self.video_capture.read()[1]
                  
    # get the current frame
    def get_current_frame(self):
        return self.current_frame

So why are we using a thread to capture frames from our webcam? Two reasons:

  1. To prevent frame lag, where the image served up by the webcam is a few seconds out of date
  2. To prevent the webcam from slowing down the rendering of the gaming graphics
Advertisements