Tags

, , , , , , , , , , , , ,

Toad! Smelly toad!

Never mind that. In my last post, Whac-A-Mole with Augmented Reality (Mark II), I recreated the classic arcade game Whac-A-Mole with some Python code and a smattering of augmented reality. And a dangerous steel hammer:

The following technologies were harnessed:

  • OpenCV Computer Vision to detect the wielding of a hammer in the webcam
  • OpenGL Graphics Library to render a 3D mole on the screen and play the game

It had a nice high score table, with built-in voice recognition. Sweetie.

Let’s add some more features…

Title screen

Every game needs a title screen. Here’s the code:

from OpenGL.GL import *

class TitleScreen(object):

    def __init__(self, fonts):
        self.fonts = fonts

    def _draw_mole(self):
        # draw mole
        glBegin(GL_QUADS)
        glTexCoord2f(1.0, 0.0); glVertex3f(-1.0, -1.0, -1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f(-1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f( 1.0, -1.0, -1.0)
        glEnd()

    def render(self, texture_titlescreen, texture_mole):
        # create start screen content to render
        content = ["*** Whac-A-Mole ***"]
        content.append("Hit any key (or Esc to exit)...")

        # render start screen content using fonts
        self.fonts.render(texture_titlescreen, content)

        # render moles
        glBindTexture(GL_TEXTURE_2D, texture_mole)
        glPushMatrix()
        
        glTranslatef(-3.0,0.0,-9.0); self._draw_mole()
        glTranslatef(3.0,0.0,0.0); self._draw_mole()
        glTranslatef(3.0,0.0,0.0); self._draw_mole()

        glPopMatrix()

Our render method takes care of displaying the title screen.

It uses the Fonts class to write some text to the screen i.e. the game name “*** Whac-A-Mole ***” and instruction for how to start playing.

Note: we are now sharing the Fonts class between our title screen and high score table.

It also draws three moles, side by side underneath the text.

Music

I fuckin’ hate silence. It unnerves me. So the game needs a signature tune.

I have used Pygame to play the guitar track to my song Horseshit:

# initialise music
pygame.mixer.init()
pygame.mixer.Sound(GAME_MUSIC).play(-1).set_volume(0.5)

The music in on a permanent loop.

Note: the music is paused when speech recognition is employed. Otherwise we will not hear the player’s voice over the fantastic electric riffing!

Flipping the webcam image

The webcam image is used as a background during gameplay, to provide an augmented reality experience. And now we are flipping this image:

# get image from webcam 
image = cv2.flip(self.webcam.get_current_frame(), 1)

But why the hell do we need to flip the image vertically? Well, previously during gameplay, if the mole I wanted to whack appeared on the right-hand side of the screen then I had to move my hammer to the left-hand side. Very unnatural. Very frustrating. Very shit.

Now the game is far more natural to play. If the mole appears on the right-hand side then that’s where my hammer goes.

Game state

Now that the game has a title screen and high score table, we need to navigate it with the press of a key:

def _key_pressed(self, *args):
    # exit game if Escape key pressed
    if args[0] == ESCAPE_KEY:
        self.fonts.delete()
        pygame.mixer.stop()
        glutDestroyWindow(self.window_id)

    # update game state
    if self.game_state == GAMESTATE_TITLESCREEN:
        self.game_state = GAMESTATE_PLAY
            
    elif self.game_state == GAMESTATE_HIGHSCORE:
        self.game_state = GAMESTATE_TITLESCREEN

If the Escape key is detected then we clear up any resources and destroy the window.

Otherwise we navigate through the game. If we are currently on the title screen then we start playing the game. If we are currently on the high score table then we go to the title screen.

Notice how every time we draw the window, we check the state of the game:

# check game state
if self.game_state != GAMESTATE_PLAY:
            
    if self.game_state == GAMESTATE_TITLESCREEN:
        self.titlescreen.render(self.texture_titlescreen, self.texture_mole)
            
    elif self.game_state == GAMESTATE_HIGHSCORE:
        self.highscore.render(self.texture_highscore)
            
    glutSwapBuffers()
    return

If we are not currently engaged in a tussle with some moles, then we simply draw the title screen or high score table.

Miscellaneous

Every new version of a game has some miscellaneous bullshit that the author feels obliged to recount, if only as a way of self-documenting. Indulge me.

I am now drawing the score on the background image after it has been used to detect a hammer. This makes sense, as we don’t want to pollute the image before conducting motion detection.

# draw current score on image
bg_image = image.copy()
cv2.putText(bg_image, "Score: {}".format(self.score), (25,25), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,255,0), 2)

The game has been sped up. The rotation of the mole is now twice as fast. And the time limit in which to whack it is 25% less. We need to separate the men from the boys. And woman from the girls.

# update axis
self.x_axis = self.x_axis - 4
self.z_axis = self.z_axis - 4
self.time_limit = 30

Demonstration

Enough yackin’. Here’s the new and improved game. This time I am using my hand instead of a hammer to whack the moles. It’s not right for a grown man to be hitting thin air with a dangerous steel tool:

Lovely. We have music. Speech recognition. A high score table. And a title screen to entice you into another game. Who needs a PlayStation 4.

Well, that’s all folks. I’m off to calm down and read a few pages of The Inscrutable Diaries Of Rodger Saltwash.

Ciao!

P.S.

I ran the code on my Windows 7 PC using Python Tools for Visual Studio.

Here’s all the code. First up, our main Whac-A-Mole class:

from OpenGL.GL import *
from OpenGL.GLUT import *
from OpenGL.GLU import *
import cv2
from PIL import Image
import random
import pygame
from constants import *
from webcam import Webcam
from detection import Detection
from fonts import Fonts
from titlescreen import TitleScreen
from highscore import HighScore

class WhacAMole:
 
    def __init__(self):
        # initialise webcam and start thread
        self.webcam = Webcam()
        self.webcam.start()

        # initialise detection with first webcam frame
        image = cv2.flip(self.webcam.get_current_frame(), 1)
        self.detection = Detection(image) 

        # title screen and high score table
        self.fonts = None
        self.titlescreen = None
        self.highscore = None

        # game settings
        self.game_state = GAMESTATE_TITLESCREEN
        self.time_limit = 30
        self.time_left = self.time_limit
        self.score = 0
        self.hammer_cell = None
        self.mole_cell = None
        self.mole_coords = [(-3,2),(0,2),(3,2),(-3,-2),(0,-2),(3,-2)]

        # axis
        self.x_axis = 0.0
        self.z_axis = 0.0

        # textures
        self.texture_background = None
        self.texture_mole = None
        self.texture_titlescreen = None
        self.texture_highscore = None

        # window id
        self.window_id = None

    def _init_gl(self, Width, Height):
        glClearColor(0.0, 0.0, 0.0, 0.0)
        glClearDepth(1.0)
        glDepthFunc(GL_LESS)
        glEnable(GL_DEPTH_TEST)
        glShadeModel(GL_SMOOTH)
        glMatrixMode(GL_PROJECTION)
        glLoadIdentity()
        gluPerspective(45.0, float(Width)/float(Height), 0.1, 100.0)
        glMatrixMode(GL_MODELVIEW)
        
        # enable textures
        glEnable(GL_TEXTURE_2D)
        self.texture_background = glGenTextures(1)
        self.texture_mole = glGenTextures(1)
        self.texture_titlescreen = glGenTextures(1)
        self.texture_highscore = glGenTextures(1)

        # create mole texture 
        image = Image.open("mole.jpg")
        ix = image.size[0]
        iy = image.size[1]
        image = image.tostring("raw", "RGBX", 0, -1)

        glBindTexture(GL_TEXTURE_2D, self.texture_mole)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
        glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, image)
 
        # initialise title screen and high score table
        self.fonts = Fonts()
        self.titlescreen = TitleScreen(self.fonts)
        self.highscore = HighScore(self.fonts)

        # initialise music
        pygame.mixer.init()
        pygame.mixer.Sound(GAME_MUSIC).play(-1).set_volume(0.5)

    def _key_pressed(self, *args):
        # exit game if Escape key pressed
        if args[0] == ESCAPE_KEY:
            self.fonts.delete()
            pygame.mixer.stop()
            glutDestroyWindow(self.window_id)

        # update game state
        if self.game_state == GAMESTATE_TITLESCREEN:
            self.game_state = GAMESTATE_PLAY
            
        elif self.game_state == GAMESTATE_HIGHSCORE:
            self.game_state = GAMESTATE_TITLESCREEN

    def _draw_scene(self):
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        glLoadIdentity()
                
        # check game state
        if self.game_state != GAMESTATE_PLAY:
            
            if self.game_state == GAMESTATE_TITLESCREEN:
                self.titlescreen.render(self.texture_titlescreen, self.texture_mole)
            
            elif self.game_state == GAMESTATE_HIGHSCORE:
                self.highscore.render(self.texture_highscore)
            
            glutSwapBuffers()
            return

        # handle any hammer whack
        self._handle_hammer()
        
        # draw background
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glPushMatrix()
        glTranslatef(0.0,0.0,-11.2)
        self._draw_background()
        glPopMatrix()
        
        # draw mole
        if self.mole_cell != None:
            glColor4f(1.0, 1.0, 1.0, 0.8)
            glBlendFunc(GL_SRC_ALPHA, GL_ONE)
            glEnable(GL_BLEND)
            glDisable(GL_DEPTH_TEST)

            glBindTexture(GL_TEXTURE_2D, self.texture_mole)
            mole_coord = self.mole_coords[self.mole_cell]

            glPushMatrix()
            glTranslatef(mole_coord[0],mole_coord[1],-9.0)
            glRotatef(self.x_axis,1.0,0.0,0.0)
            glRotatef(0.0,0.0,1.0,0.0)
            glRotatef(self.z_axis,0.0,0.0,1.0)
            self._draw_mole()
            glPopMatrix()

            glDisable(GL_BLEND)
            glEnable(GL_DEPTH_TEST)

            # update axis
            self.x_axis = self.x_axis - 4
            self.z_axis = self.z_axis - 4

        # game logic
        if self.mole_cell == None and self.time_left == 0:
            
            # ready to show new mole!
            self.mole_cell = random.randrange(6)
            self.time_left = self.time_limit
        
        elif self.mole_cell != None:
            
            if self.time_left == 0:
                
                # mole not whacked in time! score drops to 0 
                self.highscore.record(self.score)
                self.score = 0
                self.mole_cell = None
                self.time_left = self.time_limit
                self.game_state = GAMESTATE_HIGHSCORE

            elif self.hammer_cell == self.mole_cell:
                
                # mole whacked! score increased by 1
                self.score += 1
                self.mole_cell = None
                self.time_left = self.time_limit

        # decrease game time
        self.time_left -= 1

        glutSwapBuffers()
                   
    def _handle_hammer(self):
        # get image from webcam 
        image = cv2.flip(self.webcam.get_current_frame(), 1)

        # detect any hammer whack in image
        self.hammer_cell = self.detection.get_active_cell(image)

        # draw current score on image
        bg_image = image.copy()
        cv2.putText(bg_image, "Score: {}".format(self.score), (25,25), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,255,0), 2)

        # convert image to OpenGL texture format
        bg_image = cv2.flip(bg_image, 0)
        bg_image = Image.fromarray(bg_image)     
        ix = bg_image.size[0]
        iy = bg_image.size[1]
        bg_image = bg_image.tostring("raw", "BGRX", 0, -1)
 
        # create background texture
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
        glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, bg_image)

    def _draw_background(self):
        # draw background
        glBegin(GL_QUADS)
        glTexCoord2f(0.0, 1.0); glVertex3f(-4.0, -3.0, 4.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 4.0, -3.0, 4.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 4.0,  3.0, 4.0)
        glTexCoord2f(0.0, 0.0); glVertex3f(-4.0,  3.0, 4.0)
        glEnd( )

    def _draw_mole(self):
        # draw mole
        glBegin(GL_QUADS)
        glTexCoord2f(0.0, 0.0); glVertex3f(-1.0, -1.0,  1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 1.0, -1.0,  1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0,  1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f(-1.0,  1.0,  1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f(-1.0, -1.0, -1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f(-1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f( 1.0, -1.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f(-1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f(-1.0,  1.0,  1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 1.0,  1.0,  1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f(-1.0, -1.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 1.0, -1.0, -1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f( 1.0, -1.0,  1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f(-1.0, -1.0,  1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 1.0, -1.0, -1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 1.0,  1.0,  1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f( 1.0, -1.0,  1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f(-1.0, -1.0, -1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f(-1.0, -1.0,  1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f(-1.0,  1.0,  1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f(-1.0,  1.0, -1.0)
        glEnd()

    def main(self):
        # setup and run OpenGL
        glutInit()
        glutInitDisplayMode(GLUT_RGBA | GLUT_DOUBLE | GLUT_DEPTH)
        glutInitWindowSize(640, 480)
        glutInitWindowPosition(800, 400)
        self.window_id = glutCreateWindow("OpenGL Whac-A-Mole")
        glutDisplayFunc(self._draw_scene)
        glutIdleFunc(self._draw_scene)
        glutKeyboardFunc(self._key_pressed)
        self._init_gl(640, 480)
        glutMainLoop()
 
# run an instance of Whac-A-Mole 
whacAMole = WhacAMole()
whacAMole.main()

Our High Score class:

import pygame
from speech import *

class HighScore(object):
  
    BLANK_PLAYER_NAME = "____"

    def __init__(self, fonts):
        self.fonts = fonts
        self.speech = Speech()
        self.table = [(45,"Paul"), (31,"Johnny"), (24,"Steve"), (16,"Sid"), (10,self.BLANK_PLAYER_NAME)]

    def record(self, score):

        # ensure score qualifies as a high score
        if score < self.table[-1][0]: return

        # pause music
        pygame.mixer.pause()

        # use Text To Speech to request player's name
        self.speech.text_to_speech("Please state your name")

        # use Speech to Text to obtain player's name
        player_name = self.speech.speech_to_text()
        if player_name == None: 
            player_name = self.BLANK_PLAYER_NAME

        # unpause music
        pygame.mixer.unpause()

        # add player's name and score to the high score table
        for idx, val in enumerate(self.table):
            if score >= val[0]:
                self.table.insert(idx, (score, player_name))
                break

        # ditch lowest score
        self.table.pop()

    def render(self, texture):
        # create high score content to render
        content = ["*** HIGHSCORE TABLE ***"]

        for val in self.table:
            content.append("{} : {}".format(val[0], val[1]))

        content.append("Hit any key...")

        # render high score content using fonts
        self.fonts.render(texture, content)

Our Fonts class:

from OpenGL.GL import *
from OpenGL.GLUT import *
from OpenGL.WGL import *
import win32ui
import struct

class Fonts:

    def __init__(self):
        self.base = None
        self._build()

    # build fonts
    def _build(self):
        wgldc = wglGetCurrentDC()
        
        if hasattr(wgldc,'value'): 
            wgldc = wgldc.value
        
        hDC = win32ui.CreateDCFromHandle(struct.unpack('i',struct.pack('I', wgldc & 0xffffffff))[0])
        self.base = glGenLists(32+96);

        font_properties = { "name" : "Courier New",
						    "width" : 0 ,
						    "height" : -24,
						    "weight" : 800 }
        
        font = win32ui.CreateFont(font_properties)
        oldfont = hDC.SelectObject(font)
        wglUseFontBitmapsA(wgldc, 32, 96, self.base+32)
        hDC.SelectObject(oldfont)

    # delete fonts
    def delete(self):
        glDeleteLists(self.base, 32+96)

    # print text
    def _print_text(self, text):
        glPushAttrib(GL_LIST_BIT);
        
        try:
            glListBase(self.base);
            glCallLists(text)
        finally:
            glPopAttrib();

    # render fonts
    def render(self, texture, content):
        glBindTexture(GL_TEXTURE_2D, texture)
        glColor3f (0.0, 0.3, 0.7)        
        glPushMatrix()
        glTranslatef(0.0, 0.0, -1.0)

        rasterpos_y = 0.32
        
        for line in content:
            glRasterPos2f(-0.45, rasterpos_y)
            self._print_text(line)
            rasterpos_y -= 0.1

        glPopMatrix()
        glColor3f (1.0, 1.0, 1.0)

The constants:

# game state constants
GAMESTATE_TITLESCREEN = 0
GAMESTATE_PLAY = 1
GAMESTATE_HIGHSCORE = 2

# key constants
ESCAPE_KEY = '\x1b'

# music constants
GAME_MUSIC = "song/horseshit_guitar.wav"

The Title Screen class can be found in the main body of this post. The Speech, Detection and Webcam classes can be found in the previous post.