, , , , , , , , ,

Arkwood has been, by hook or by crook, trying to get the world record at a retro computer game. I built a co-pilot to help him play Donkey Kong. I used optical flow to alert him to traffic on Frogger. But now he is gunning for the biggest game of them all: Pac-Man.

‘I don’t care much for your cheating,’ I sniped, ‘But, nevertheless, I will write some Python code to help you keep track of the ghosts on Pac-Man.’

The OpenCV tutorial on Meanshift and Camshift will provide the basis for tracking the ghosts. But first, I need to take screenshots of Arkwood playing Pac-Man on his Windows 7 laptop, saving the snaps to my Raspberry Pi for real-time analysis. The following code will do the trick:

from time import sleep
from PIL import ImageGrab

image_index = 0

while True:
    screenshot = ImageGrab.grab(bbox=(0,50,1366,600))
    screenshot.save("\\\\RASPBERRYPI\MyPython\shift\screenshots\{}.jpg".format(image_index), "JPEG")

    image_index += 1

Lovely. Here’s an example of a screenshot of Pac-Man:


Now that my Raspberry Pi is being provided with screenshots of Arkwood playing Pac-Man, it can analyse each image in turn and attempt to track the ghosts. To start with we will just attempt to track the green ghost.

Here’s the main program:

import numpy as np
import cv2
from time import sleep
# load first frame
image_index = 0
frame = _load_frame()

# set track window and roi history
x,y,w,h = 470,260,420,100

track_window = x,y,w,h

hsv_roi =  cv2.cvtColor(frame[y:y+h, x:x+w], cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv_roi, np.array([50,100,100]), np.array([70,255,255]))
roi_hist = cv2.calcHist([hsv_roi],[0],mask,[180],[0,180])
term_crit = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT,10,1)

while True:
    # load next frame 
    image_index += 1
    frame = _load_frame()

    # do camshift
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
    dst = cv2.calcBackProject([hsv],[0],roi_hist,[0,180],1)

    prev_track_window = track_window
    ret, track_window = cv2.CamShift(dst, track_window, term_crit)
    # check for tracking of object
    if not _is_tracking(prev_track_window):
        track_window = x,y,w,h


First up, we grab the initial screenshot of Arkwood playing Pac-Man.

Next we set up our ‘region of interest’, which is the part of the image where we expect to find our ghost when the game starts i.e. the ghost’s den. Note that for our mask we are providing array values in the range of the colour green (as it is the green ghost we care to track).

Finally, we enter a while loop. We grab the next screenshot and let the CamShift method handle the tracking of our green ghost. If tracking is not detected, we reset our tracking window back to the ghost’s den. Before the while loop completes, the tracking is displayed in a window and saved to disk.

Here’s a screenshot of the ghost’s den, where we start and reset our tracking:


Hurray! So let’s see the tracking in action.

Here’s our green ghost being tracked in its den:


Now, as the ghost leaves its den, we can see that it is being successfully tracked around the screen:







Great. Now let’s see what happens when our Pac-Man perishes:






When Pac-Man gets brutalised by the orange ghost, the green ghost stops moving. This lack of movement causes our tracking window to reset to the centre of the screen, waiting to pick up the green ghost as soon as it reappears in the den…






Fantastic. The green ghost is being tracked again, as the game restarts.

‘So how’s tracking a ghost gonna help me get a top score?’ my filthy Belgian buddy asked me.

It was a fair question. Perhaps the Python code could play a sound, to let Arkwood know when all ghosts have left the den? Or maybe we could check the coordinates of each ghost, to tell if any are in the tunnel?

I told him, ‘The only thing holding me back is the performance of the Raspberry Pi. It’s a bit slow to process the screenshots, which leads to a real-time lag.’

Arkwood crinkled his brow. Biting down on a cheese and pickle sandwich, he retorted, ‘Well, it’s a crock of shit then, isn’t it.’

A red-hot anger surged within me and I pummelled his testicles with my fist, forcing pieces of sandwich out of his mouth and across the room.



Here’s the remaining code, which assists the main program in tracking the green ghost. First, the methods for loading screenshots and saving tracking images:

# load frame from disk
def _load_frame():
    while True:
        frame = cv2.imread('screenshots/{}.jpg'.format(image_index),1)

        if frame != None:
            return frame


# save frame to disk
def _save_frame():

Next, the method to display the tracking in a window:

# display window
def _display_window():
    tx,ty,tw,th = track_window
    cv2.imshow('display window',frame)

And finally the method which lets us know whether the green ghost has stopped moving (or whether we are tracking an ever-expanding part of the screen which does not contain the ghost):

# check for tracking of object
def _is_tracking(prev_track_window):

    # first, make sure object is actually moving
    if prev_track_window == track_window:
        return False

    # next, make sure object is appropriate size
    tx,ty,tw,th = track_window
    if tw > w or th > h:
        return False

    return True