Tags

, , , , , , , , , , , , ,

Is technology good or evil? Neither. It is how one wields the hammer. Technology grants our species a horizon – subdues our chaotic flailing arms and busies our hands, as we mesh together a natural and synthetic world. But is it right to wield that hammer upon a poor mole, if that rodent is only pixels? The judge whacked down his gavel. ‘Sir, there is intent in your eyes to harm! You wretched soul!’

Never mind that. In my last post, Whac-A-Mole with Augmented Reality, I recreated the classic arcade game Whac-A-Mole with some Python code and a smattering of augmented reality. And a dangerous steel hammer:

Playing the game is great fun. Let’s look to extend it so as to provide a high score table.

High score table

So, to recap: Whac-A-Mole with Augmented Reality uses the following technologies:

  • OpenCV Computer Vision to detect the wielding of a hammer in the webcam
  • OpenGL Graphics Library to render a 3D mole on the screen and play the game

The game ends when the player fails to whack the mole with a hammer. At this point I want to record the player’s score to a high score table (if the score is good enough, of course) and then display the high score table. Let’s look at the High Score class:

from speech import *
from fonts import *

class HighScore(object):
  
    BLANK_PLAYER_NAME = "____"

    def __init__(self):
        self.speech = Speech()
        self.fonts = Fonts()
        self.table = []

    def build(self):
        # build high score fonts and table
        self.fonts.build()
        self.table = [(45,"Paul"), (31,"Johnny"), (24,"Steve"), (16,"Sid"), (0,self.BLANK_PLAYER_NAME)]

    def record(self, score):
        # ensure high score table has been built
        if not self.table: return

        # ensure score qualifies as a high score
        if score < self.table[-1][0]: return

        # use Text To Speech to request player's name
        self.speech.text_to_speech("Please state your name")

        # use Speech to Text to obtain player's name
        player_name = self.speech.speech_to_text()
        if player_name == None: 
            player_name = self.BLANK_PLAYER_NAME

        # add player's name and score to the high score table
        for idx, val in enumerate(self.table):
            if score >= val[0]:
                self.table.insert(idx, (score, player_name))
                break

        # ditch lowest score
        self.table.pop()

    def render(self, texture):
        # ensure fonts have been built
        if not self.fonts.is_built(): return
        
        # create high score content to render
        content = ["*** HIGHSCORE TABLE ***"]

        for val in self.table:
            content.append("{} : {}".format(val[0], val[1]))

        content.append("Press any key to continue...")

        # render high score content using fonts
        self.fonts.render(texture, content)

The build method does two things. It builds the fonts we will be rendering to screen (more on that later). And it builds our high score table with some dummy entries.

The record method first ensures that our high score table has been built, and that the provided score qualifies as an entry (there is only space for 5 top scores).

Next, we use the Speech class’s text_to_speech method to ask our player, via computer speakers, what their name is.

The Speech class’s speech_to_text method listens for the player’s response, via a computer microphone, and converts their voice into text.

Finally, the player’s name is added to the high score table (and the lowest score is discarded).

The render method uses our Fonts class to render the high score table to the screen (assuming the fonts have been built).

Speech recognition

So how the hell did we manage to ask the player their name, and then convert the vocal response to text? Easy. I used my Speech class:

import speech_recognition as sr
import pyttsx

class Speech(object):
  
    def __init__(self):
        self.stt = sr.Recognizer()
        self.tts = pyttsx.init()

    # speech to text (using Google Speech Recognition)
    def speech_to_text(self):

        with sr.Microphone() as source:
            print "listening..."
            audio = self.stt.listen(source)
        
        try:
            return self.stt.recognize_google(audio)
        except sr.UnknownValueError:
            print("Google Speech Recognition could not understand audio")
        except sr.RequestError:
            print("Could not request results from Google Speech Recognition service")

        return None
  
    # text to speech (using pyttsx)
    def text_to_speech(self, text):

        if not text:
            print("No text provided to pyttsx")
            return

        self.tts.say(text)
        self.tts.runAndWait()

The speech_to_text method uses a Python Speech Recognition library to convert speech to text. I’ve configured the library to target Google Speech Recognition.

The text_to_speech method uses pyttsx to convert text to speech.

Displaying fonts

So we have a high score table. And we can talk to the player, to get their name. But how do we actually display the high score table on the screen? Here’s the Fonts class:

from OpenGL.GL import *
from OpenGL.GLUT import *
from OpenGL.WGL import *
import win32ui
import struct

class Fonts:

    def __init__(self):
        self.base = None

    # build fonts
    def build(self):
        wgldc = wglGetCurrentDC()
        
        if hasattr(wgldc,'value'): 
            wgldc = wgldc.value
        
        hDC = win32ui.CreateDCFromHandle(struct.unpack('i',struct.pack('I', wgldc & 0xffffffff))[0])
        self.base = glGenLists(32+96);
        
        font_properties = { "name" : "Courier New",
						    "width" : 0 ,
						    "height" : -24,
						    "weight" : 800 }
        
        font = win32ui.CreateFont(font_properties)
        oldfont = hDC.SelectObject(font)
        wglUseFontBitmapsA(wgldc, 32, 96, self.base+32)
        hDC.SelectObject(oldfont)

    # check if fonts are built
    def is_built(self):
        return self.base != None

    # print text
    def _print_text(self, text):
        glPushAttrib(GL_LIST_BIT);
        
        try:
            glListBase(self.base);
            glCallLists(text)
        finally:
            glPopAttrib();

    # render fonts
    def render(self, texture, content):
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        glLoadIdentity()
        glBindTexture(GL_TEXTURE_2D, texture)
        glColor3f (0.0, 0.3, 0.7)
        glTranslatef(0.0, 0.0, -1.0)

        rasterpos_y = 0.32
        
        for line in content:
            glRasterPos2f(-0.45, rasterpos_y)
            self._print_text(line)
            rasterpos_y -= 0.1

        glColor3f (0.0, 0.0, 0.0)
        glutSwapBuffers()

We are in OpenGL land. Rather than me poorly explaining the specifics of each line of code, take a read through the NeHe article on Bitmap Fonts (operations may be legacy-mode).

The build method selects the font properties we wish to use (Courier New, as it happens) and stores our fonts for later use.

The is_built method simply informs any calling method whether the fonts are built and ready to use.

Our render method is where all the fun happens (promise). It is provided with content to display on the screen. Notice how it uses a rasterpos_y variable to ensure each line of content is positioned underneath the previous line. We call the _print_text private method to get the characters to render.

Demonstration

All that’s left to do is hook our high score table into our Whac-A-Mole game and take it for a spin. Here’s me playing Whac-A-Mole and making a high score:

Granted, the high score table is not going to win any awards for stunning graphics. But it is a first step in rendering fonts to screen, as well as using speech recognition to obtain the player’s name.

Ciao!

P.S.

I ran the code on my Windows 7 PC using Python Tools for Visual Studio.

Here’s the Whac-A-Mole class, amended to incorporate the high score table:

from OpenGL.GL import *
from OpenGL.GLUT import *
from OpenGL.GLU import *
import cv2
from PIL import Image
import random
from webcam import Webcam
from detection import Detection
from highscore import HighScore

class WhacAMole:
 
    def __init__(self):
        # initialise webcam and start thread
        self.webcam = Webcam()
        self.webcam.start()

        # initialise detection with first webcam frame
        self.detection = Detection(self.webcam.get_current_frame()) 

        # initialise high score table
        self.highscore = HighScore()

        # initialise game settings
        self.game_active = True
        self.time_limit = 40
        self.time_left = self.time_limit
        self.score = 0
        self.hammer_cell = None
        self.mole_cell = None
        self.mole_coords = [(-3,2),(0,2),(3,2),(-3,-2),(0,-2),(3,-2)]

        # initialise axis
        self.x_axis = 0.0
        self.z_axis = 0.0

        # initialise textures
        self.texture_background = None
        self.texture_mole = None
        self.texture_highscore = None

    def _init_gl(self, Width, Height):
        glClearColor(0.0, 0.0, 0.0, 0.0)
        glClearDepth(1.0)
        glDepthFunc(GL_LESS)
        glEnable(GL_DEPTH_TEST)
        glShadeModel(GL_SMOOTH)
        glMatrixMode(GL_PROJECTION)
        glLoadIdentity()
        gluPerspective(45.0, float(Width)/float(Height), 0.1, 100.0)
        glMatrixMode(GL_MODELVIEW)
        
        # enable textures
        glEnable(GL_TEXTURE_2D)
        self.texture_background = glGenTextures(1)
        self.texture_mole = glGenTextures(1)
        self.texture_highscore = glGenTextures(1)

        # create mole texture 
        image = Image.open("mole.jpg")
        ix = image.size[0]
        iy = image.size[1]
        image = image.tostring("raw", "RGBX", 0, -1)

        glBindTexture(GL_TEXTURE_2D, self.texture_mole)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
        glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, image)
 
        # build high score table
        self.highscore.build()

    def _key_pressed(self, *args):
        # on key press, set game to active
        self.game_active = True

    def _draw_scene(self):
        
        # check whether game is active
        if not self.game_active:
            self.highscore.render(self.texture_highscore)
            return

        # handle any hammer whack
        self._handle_hammer()
 
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        glLoadIdentity()

        # draw background
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glPushMatrix()
        glTranslatef(0.0,0.0,-11.2)
        self._draw_background()
        glPopMatrix()

        # draw mole
        if self.mole_cell != None:
            glColor4f(1.0, 1.0, 1.0, 0.8)
            glBlendFunc(GL_SRC_ALPHA, GL_ONE)
            glEnable(GL_BLEND)
            glDisable(GL_DEPTH_TEST)

            glBindTexture(GL_TEXTURE_2D, self.texture_mole)
            mole_coord = self.mole_coords[self.mole_cell]

            glPushMatrix()
            glTranslatef(mole_coord[0],mole_coord[1],-9.0)
            glRotatef(self.x_axis,1.0,0.0,0.0)
            glRotatef(0.0,0.0,1.0,0.0)
            glRotatef(self.z_axis,0.0,0.0,1.0)
            self._draw_mole()
            glPopMatrix()

            glDisable(GL_BLEND)
            glEnable(GL_DEPTH_TEST)

            # update axis
            self.x_axis = self.x_axis - 2
            self.z_axis = self.z_axis - 2

        # determine game state
        if self.mole_cell == None and self.time_left == 0:
            
            # ready to show new mole!
            self.mole_cell = random.randrange(6)
            self.time_left = self.time_limit
        
        elif self.mole_cell != None:
            
            if self.time_left == 0:
                
                # mole not whacked in time! score drops to 0 
                self.highscore.record(self.score)
                self.score = 0
                self.mole_cell = None
                self.time_left = self.time_limit
                self.game_active = False

            elif self.hammer_cell == self.mole_cell:
                
                # mole whacked! score increased by 1
                self.score += 1
                self.mole_cell = None
                self.time_left = self.time_limit

        # decrease game time
        self.time_left -= 1

        glutSwapBuffers()
                   
    def _handle_hammer(self):
        # get image from webcam 
        image = self.webcam.get_current_frame()
        
        # draw current score on image
        cv2.putText(image, "Score: {}".format(self.score), (25,25), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,255,0), 2)

        # detect any hammer whack in image
        self.hammer_cell = self.detection.get_active_cell(image)

        # convert image to OpenGL texture format
        image = cv2.flip(image, 0)
        gl_image = Image.fromarray(image)     
        ix = gl_image.size[0]
        iy = gl_image.size[1]
        gl_image = gl_image.tostring("raw", "BGRX", 0, -1)
 
        # create background texture
        glBindTexture(GL_TEXTURE_2D, self.texture_background)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
        glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, gl_image)

    def _draw_background(self):
        # draw background
        glBegin(GL_QUADS)
        glTexCoord2f(0.0, 1.0); glVertex3f(-4.0, -3.0, 4.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 4.0, -3.0, 4.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 4.0,  3.0, 4.0)
        glTexCoord2f(0.0, 0.0); glVertex3f(-4.0,  3.0, 4.0)
        glEnd( )

    def _draw_mole(self):
        # draw mole
        glBegin(GL_QUADS)
        glTexCoord2f(0.0, 0.0); glVertex3f(-1.0, -1.0,  1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 1.0, -1.0,  1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0,  1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f(-1.0,  1.0,  1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f(-1.0, -1.0, -1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f(-1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f( 1.0, -1.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f(-1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f(-1.0,  1.0,  1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 1.0,  1.0,  1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f(-1.0, -1.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 1.0, -1.0, -1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f( 1.0, -1.0,  1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f(-1.0, -1.0,  1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f( 1.0, -1.0, -1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f( 1.0,  1.0, -1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f( 1.0,  1.0,  1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f( 1.0, -1.0,  1.0)
        glTexCoord2f(0.0, 0.0); glVertex3f(-1.0, -1.0, -1.0)
        glTexCoord2f(1.0, 0.0); glVertex3f(-1.0, -1.0,  1.0)
        glTexCoord2f(1.0, 1.0); glVertex3f(-1.0,  1.0,  1.0)
        glTexCoord2f(0.0, 1.0); glVertex3f(-1.0,  1.0, -1.0)
        glEnd()

    def main(self):
        # setup and run OpenGL
        glutInit()
        glutInitDisplayMode(GLUT_RGBA | GLUT_DOUBLE | GLUT_DEPTH)
        glutInitWindowSize(640, 480)
        glutInitWindowPosition(800, 400)
        glutCreateWindow("OpenGL Whac-A-Mole")
        glutDisplayFunc(self._draw_scene)
        glutIdleFunc(self._draw_scene)
        glutKeyboardFunc(self._key_pressed)
        self._init_gl(640, 480)
        glutMainLoop()
 
# run an instance of Whac-A-Mole 
whacAMole = WhacAMole()
whacAMole.main()

The main amendments are:

  • Building the high score table, in the _init_gl method
  • Recording our score when we fail to whack a mole, in the _draw_scene method
  • Rendering the high score table when the game is no longer active, in the _draw_scene method
  • Handling key press for a new game, in the _key_pressed method

The Webcam and Detection classes can be found in my previous post.

Advertisements