, , , , , , , , , , , , ,

Arkwood, my debased Belgian buddy, is gunning for a world record at retro computer game Pac-Man. In my previous post, Camshift on Raspberry Pi, I wrote some Python code to help him track the green ghost during gameplay. ‘I will be able to offer you real-time strategy, now that I know the exact coordinates of the enemies,’ I bragged. The only problem was, the tiny Raspberry Pi computer’s way too slow to process screenshots from the game. The lag was simply unbearable.

Not to be undone, I have ported all the code onto the Windows 7 PC that Arkwood will be using to play Pac-Man. Now I can be sure that each screenshot will be processed immediately, yielding a top strategy.

So let’s look at the main program:

import cv2
import numpy as np
from PIL import ImageGrab
import winsound
from time import sleep

# constants
X,Y,W,H = 600,400,480,100

# initialise global variables
frame = None
roi_hist = None
track_window = X,Y,W,H
image_index = 0

# initialise state

while True:
    # grab next frame
    # do camshift
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
    dst = cv2.calcBackProject([hsv],[0],roi_hist,[0,180],1)
    prev_track_window = track_window
    ret, track_window = cv2.CamShift(dst, track_window, TERM_CRIT)

    # check for tracking of object
    if _is_tracking(prev_track_window):
        track_window = X,Y,W,H
    # save frame at interval
    image_index += 1
    if image_index % SAVE_INTERVAL == 0:

    # throttle frame capture
    # sleep(0.8)

Now, the main program clearly has a number of supporting functions which I will cover at the foot of this post. But for now, let’s just concentrate on the key events, which build upon the OpenCV tutorial on Meanshift and Camshift.

First up, we initialise some constants, variables and state, before dropping into a while loop in order to track our green ghost.

We grab a frame from the game – a screenshot of Arkwood furiously eating yellow dots whilst avoiding the pack of ghosts.

The OpenCV CamShift method will then attempt to track the green ghost, using the provided frame against its ‘region of interest’.

Once CamShift is done, we check for tracking, using a beep sound to alert Arkwood if the green ghost is on the move. If the green ghost is not being monitored then we reset our tracking window.

To complete the while loop, we save our frame to disk and throttle our frame capture rate (this is handy, because if we take screenshots too quickly the code may think that the ghost is not moving).

Here’s some screenshots of the green ghost being tracked, causing a high-pitched beep sound to play through the speakers:







If you look closely, you will see the tracking window drawn as a blue box around the green ghost.

When Pac-Man eats an energizer, all the ghosts turn blue. The green ghost is not green anymore, so is no longer tracked:



Also, when Pac-Man is killed, the green ghost stops moving so is no longer tracked:



Every time tracking falters, the tracking window returns to the centre of the screen, waiting for the green ghost to reappear in its den:



‘Hurray,’ I said to Arkwood, ‘Now a high-pitched beep sound will let you know when the green ghost is a threat during live gameplay. What do you think?’

‘Sorry?’ my chum replied, rubbing his ears. ‘I can’t hear you. I think that the beeping sound has given me tinnitus.’

Hmm. Perhaps the real-time strategy could be administered using a more soothing sound? Certainly once I have the code tracking all the ghosts in a more intelligent way, the beep could prove too primitive. But wait…

I dashed through to my bedroom and returned with a tattered book on Morse code. ‘Get reading,’ I instructed Arkwood, ‘I have just had a brainwave!’


Here’s the functions I promised, called from the main program. First up, the code to grab a screenshot of Arkwood playing Pac-Man:

# grab frame from screen
def _grab_frame():
    global frame

    screenshot = ImageGrab.grab(bbox=(0,50,1680,900))
    frame = cv2.cvtColor(np.array(screenshot), cv2.COLOR_RGB2BGR)

Next, the function to set our ‘region of interest’ for CamShift:

# set up region of interest
def _initialise_roi():
    global roi_hist

    roi_hsv =  cv2.cvtColor(frame[Y:Y+H, X:X+W], cv2.COLOR_BGR2HSV)
    mask = cv2.inRange(roi_hsv, np.array([40,100,100]), np.array([60,255,255]))
    roi_hist = cv2.calcHist([roi_hsv],[0],mask,[180],[0,180])

The key thing to note here is that the mask is taking an array range for the colour green, as we only want to track the green ghost.

Now the function to check if our green ghost is being tracked:

# check for tracking of object
def _is_tracking(prev_track_window):

    # make sure object has coordinates
    if track_window == (0,0,0,0):
        return False

    # make sure object is actually moving
    if prev_track_window == track_window:
        return False
    # make sure object is appropriate size
    tx,ty,tw,th = track_window
    if tw > W or th > H:
        return False
    return True

Note that there are a few conditions to overcome before we can be sure that the green ghost is on the move.

Finally, the function to save the frame to disk, so we can review our ghost tracking retrospectively:

# save frame to disk
def _save_frame():
    tx,ty,tw,th = track_window

I used Python Tools for Visual Studio to run the code on the Windows 7 PC.

I used VICE emulator to play the Commodore 64 version of Pac-Man on the Windows 7 PC.