Using Camshift for Pac-Man


, , , , , , , , , , , , ,

It is common knowledge amongst the townsfolk that Arkwood is gunning for a world record at the classic arcade game Pac-Man. Decorative lights and bunting have been strung along the high street. Community halls are awash with hysteria and repeated use of the phrase, ‘He’s putting Bo’ness on the map.’ Indeed, such is the buzz that the village elders have even pardoned my buddy’s crime of indecent exposure. But still he weeps.

‘Help me!’ Arkwood bleated, gripping tight my shirt sleeve, ‘I have been practicing Pac-Man for eight days solid but still do not have a top score.’

In my last post, Camshift on Windows 7, I used OpenCV Camshift in order to track the green ghost in Pac-Man. The Python code took regular screenshots of my chum playing the game on his Windows 7 PC, making a beep sound through a set of speakers if the ghost was a threat.

‘Can’t you make it a bit smarter, so that all the ghosts are tracked,’ he pleaded.

‘Okay, leave it with me. But I have to say, if the good people of Bo’ness find out that you have been cheating to get a high score, you will be strung up on a lamppost aside the bunting.’

Let’s take a look at the main program, which now tracks both the green and orange ghost:

from ObjectTracker import ObjectTracker
from PIL import ImageGrab
import numpy as np
import cv2
import winsound

# grab frame from screen
def _grab_frame():
    screenshot = ImageGrab.grab(bbox=(0,50,1680,900))
    return cv2.cvtColor(np.array(screenshot), cv2.COLOR_RGB2BGR)

# initialise object trackers
frame = _grab_frame()
greenGhost = ObjectTracker("GreenGhost",np.array([40,100,100]),np.array([60,255,255]), frame)
orangeGhost = ObjectTracker("OrangeGhost",np.array([10,100,100]),np.array([30,255,255]), frame)

while True:
    # grab next frame
    frame = _grab_frame()

    # track ghosts

    # alert Arkwood if ghosts out of den
    if(greenGhost.is_tracking_outwith_init_coord() and orangeGhost.is_tracking_outwith_init_coord()):

We grab an initial frame – a screenshot of Arkwood playing Pac-Man. Our _grab_frame function does the heavy lifting.

Next we initialise our green and orange ghosts, creating an ObjectTracker class for each. The parameters for ObjectTracker allow us to grant the ghost a name, set its colour range through two numpy arrays, and provide the initial frame.

Finally, we enter a while loop. We grab a frame of live gameplay and pass it to each of our ghost objects. The is_tracking_outwith_init_coord method is then invoked for each ghost, to determine whether both ghosts are being tracked outwith their den. If so, we play a beep sound to alert Arkwood of the danger.

Now, before we take a peek at the inner workings of the ObjectTracker class, let’s have a demo…

Here’s the game starting, with both ghosts being tracked inside their den (note the tracking window drawn as a blue box around the green and orange ghosts):



With only the green ghost out of its den, we do not alert Arkwood via the beep sound:



But now both ghosts are out of den, and the beep sound comes through the speakers:



If one of the ghosts drifts back into the den area, the beeping stops:



‘That’s great!’ Arkwood exclaimed, ‘Can you get it working with all the ghosts?’

Okay, let’s try adding the pink ghost into the mix:

pinkGhost = ObjectTracker("PinkGhost",np.array([0,100,100]),np.array([10,255,255]), frame)


if(greenGhost.is_tracking_outwith_init_coord() and orangeGhost.is_tracking_outwith_init_coord() and pinkGhost.is_tracking_outwith_init_coord()):

Problem is, the pink ghost and the red ghost are similar in colour. The object tracker for the pink ghost would sometimes switch to the red ghost. Nevertheless, we did have some success tracking three ghosts at the same time:



I told Arkwood that I would need more time to get the tracking system working for all four ghosts. Once accomplished, I would be able to offer up sophisticated real-time strategies to help him get the world record at Pac-Man. ‘I can check each of the ghosts’ coordinates and provide instruction of what you should do.’

He replied, ‘Well hurry. The mayor has commissioned a statue of me for the town square. If I fail to get the world record I will be burned at the stake.’



Here’s the ObjectTracker class:

import cv2
import numpy as np

class ObjectTracker(object):

    # constants
    X,Y,W,H = 550,350,580,200
    # initialise the object tracker
    def __init__(self, object_name, lower_colour, upper_colour, frame):
        self.object_name = object_name
        self.lower_colour = lower_colour
        self.upper_colour = upper_colour
        self.frame = frame
        self.roi_hist = self._initialise_roi()
        self.track_window = self.X,self.Y,self.W,self.H
        self.is_tracking = False
        self.image_index = 0

    # set up region of interest
    def _initialise_roi(self):
        roi_hsv =  cv2.cvtColor(self.frame[self.Y:self.Y+self.H, self.X:self.X+self.W], cv2.COLOR_BGR2HSV)
        mask = cv2.inRange(roi_hsv, self.lower_colour, self.upper_colour)
        roi_hist = cv2.calcHist([roi_hsv],[0],mask,[180],[0,180])
        return roi_hist

    # check for tracking of object
    def _check_tracking(self, prev_track_window):

        # make sure object has coordinates
        if self.track_window == (0,0,0,0):
            self.roi_hist = self._initialise_roi()
            return False

        # make sure object is actually moving
        if prev_track_window == self.track_window:
            return False
        # make sure object is appropriate size
        tx,ty,tw,th = self.track_window
        if tw > self.W or th > self.H:
            return False
        return True

    # save frame to disk
    def _save_frame(self):
        tx,ty,tw,th = self.track_window

    # track object on supplied frame
    def track(self, frame):
        self.frame = frame

        # do camshift
        hsv = cv2.cvtColor(self.frame, cv2.COLOR_BGR2HSV)
        dst = cv2.calcBackProject([hsv],[0],self.roi_hist,[0,180],1)
        prev_track_window = self.track_window
        ret, self.track_window = cv2.CamShift(dst, self.track_window, self.TERM_CRIT)

        # check for tracking of object
        if self._check_tracking(prev_track_window):
            self.is_tracking = True
            self.track_window = self.X,self.Y,self.W,self.H
            self.is_tracking = False
        # save frame at interval
        self.image_index += 1
        if self.image_index % self.SAVE_INTERVAL == 0:

    # is object being tracked outwith its initial coordinates
    def is_tracking_outwith_init_coord(self):
        # make sure object is being tracked
        if not self.is_tracking:
            return False

        # make sure object is outwith its initial coordinates
        if (self.track_window[0] >= self.X and self.track_window[0] <= (self.X + self.W)) and \
            (self.track_window[1] >= self.Y and self.track_window[1] <= (self.Y + self.H)):
            return False

        return True

The __init__ method takes care of setting up the object, with the name, colour and initial frame provided from the main program.

We have a private method _initialise_roi to start tracking at the centre of the frame, where the ghosts’ den is.

Our _check_tracking private method compares our current tracking window against the previous one, to determine if the ghost is actually moving.

We also have a private method _save_frame which takes care of drawing the tracking window on the current frame and saving it to disk.

The public method track is called by our main program. It utilises the OpenCV CamShift method to track the ghost – we use the the tracking window it returns to check if the ghost is moving. At a set interval, we save the current frame to disk for later inspection.

The public method is_tracking_outwith_init_coord is also called by our main program. It returns True or False, depending on whether the ghost is currently being tracked outside of its den.

A couple of things to note regarding tracking…

If a ghost goes through the tunnel in Pac-Man, we lose tracking. Our code needs to be updated to deal with this scenario.

Once the Pac-Man energizer wears off and the blue ghosts return to their normal colours, we lose tracking. Our code needs to be updated to deal with this scenario.

If a ghost drifts too far between frames, we lose tracking. We must ensure that the code is optimised on each iteration, or that the playing speed of the game is reduced.

If a ghost does not drift enough between frames, we lose tracking. We can introduce a throttle to our code, as per my previous post, to slow each iteration of the code.

A couple of other things to note…

The OpenCV Changing Colorspaces tutorial has a section entitled How to find HSV values to track? which can help to determine appropriate colour ranges for each ghost.

If you want to avoid drawing all the tracking windows on the same frame, simply make a copy of the the frame as thus:


I used Python Tools for Visual Studio to run the code on the Windows 7 PC.

I used VICE emulator to play the Commodore 64 version of Pac-Man on the Windows 7 PC.


Get every new post delivered to your Inbox.

Join 67 other followers