, , , , , , , , , , , , ,

Arkwood, my slovenly Belgian buddy, is still gunning for a world record at the classic arcade game Pac-Man. He is quite prepared to exercise unscrupulous means.

In my post, Using Camshift for Pac-Man, I unwittingly agreed to help him. I wrote some Python code that took regular screenshots of Arkwood playing Pac-Man on his Windows 7 PC. The code then used OpenCV Camshift to track the green and orange ghosts around the screen, playing a beep sound though a set of speakers if the ghosts were a threat.

‘It works great,’ my chum informed, ‘But I can’t get that beep sound out of my ears. I think you have given me tinnitus.’

Luckily, I have a Plan B. I told him, ‘You need to think like a ninja, if you are to achieve the top score in Pac-Man.’ I will use OpenCV Background Subtraction instead, to enlighten Arkwood as to the ways of the ghost. Here’s the plan…

  1. Take a screenshot of Arkwood playing Pac-Man
  2. Remove the background from the screenshot
  3. Save the amended screenshot to disk
  4. Show the amended screenshot to Arkwood, whilst he plays Pac-Man

If we can repeat these steps quickly enough, Arkwood will have a real-time display of the ghosts’ movement around the screen, without all the background clutter. He will learn of the ghosts’ Yin and Yang. He will become One with the spirit world.

Let’s look at the Python code that will run in the background on Arkwood’s Windows 7 PC:

import cv2
import ImageGrab
import numpy

# grab frame from screen
def _grab_frame():
    screenshot = ImageGrab.grab(bbox=(0,50,1300,900))
    return cv2.cvtColor(numpy.array(screenshot),cv2.COLOR_RGB2BGR)

# save frame to disk
def _save_frame(frame):

# constants

# set up background subtraction
fgbg = cv2.BackgroundSubtractorMOG()

# apply background subtraction
while True:
    frame = _grab_frame()
    fgmask = fgbg.apply(frame, learningRate=1.0/HISTORY)

Once OpenCV Background Subtraction has been set up, we drop into a while loop.

We use the _grab_frame function to obtain a screenshot of Arkwood playing Pac-Man.

The apply method of Background Subtraction is provided with said screenshot, returning the image with its background removed.

Finally, the _save_frame function commits the amended screenshot to disk.

Here’s an example of the screenshot before the background is removed:


And after the background is removed:


Great. Now all we need is a way to show each amended screenshot to Arkwood, whilst he is playing Pac-Man. For this, we will use a browser. Here’s the web page code:

<script type="text/JavaScript">
function timedRefresh(timeoutPeriod)
    canvas = document.getElementById("canvas");
    context = canvas.getContext("2d");
    img = new Image();
    img.src = "pacman.jpg?t=" + new Date().getTime();

    img.onload = function() {
        context.drawImage(img, 0, 0, canvas.width, canvas.height);

    img.onerror = function() {
<title>Game Panel</title>
<body onload="JavaScript:timedRefresh(100);">
<canvas id="canvas" width="280" height="180" />

The timedRefresh JavaScript function is called when the page loads.

timedRefresh draws the amended screenshot of Pac-Man onto a canvas, utilising the image onload event. Notice how the timedRefresh is then re-called via setTimeout, so as to pick up the next screenshot.

If there is an error loading the image, onerror ensures setTimeout is invoked so that our display does not halt.

We also need to add a unique parameter to our image source – in this case, the current date and time – to avoid the ‘pacman.jpg’ being cached by the browser.

Hats off to the contributors on a stackoverflow post for the HTML and JavaScript.

Time for a demo. Here’s the browser window sitting alongside the Pac-Man window, showing a succession of amended screenshots to Arkwood as he plays the game…




Arkwood is now able to ‘feel’ the flow of the ghosts at a glance. Granted, there is a fraction of a second delay between gameplay and the amended screenshot.

‘So, what do you think?’ I asked my pal, as he tackled Pac-Man.

‘It’s superb,’ he enthused, ‘I am well on my way to a top score. All the pretty girls will want me when I am the champ!’

It’s true. Arkwood’s success with the ladies has been woeful, but things are on the up. He’ll soon be able to pack away his gramophone and the sad tales of George Formby. Hurray!


Note that OpenCV BackgroundSubtractorMOG can be swapped out for the alternative BackgroundSubtractorMOG2 if desired:

fgbg = cv2.BackgroundSubtractorMOG2()

To ensure objects are dispensed with once they have left the screen (or are no longer active), set the learningRate parameter of the apply method. Stackoverflow provided the detail.

I used Python Tools for Visual Studio to run the Python code on the Windows 7 PC.

I used VICE emulator to play the Commodore 64 version of Pac-Man on the Windows 7 PC.

I used Google Chome version ‘39.0.2171.95 m’ to run the web page code on the Windows 7 PC.