Tags

, , , , , , , , , , ,

In my last post, Detecting objects in Pac-Man, I used OpenCV techniques such as Background Subtraction, Contours and colour matching to plot the coordinates of characters in the retro arcade game Pac-Man.

Here’s an image showing the coordinates of Pac-Man and the four ghosts:

pacman_objectdetection_marktwo_132

And here’s that same image displayed in a side panel, whilst my buddy Arkwood plays Pac-Man:

pacman_objectdetection_marktwo_screenshot

But why the devil would you do such a thing? I hear you ask. Well, Arkwood is trying to get a world record at Pac-Man. So I wrote some Python code that took a screenshot of him playing the game, applied the OpenCV techniques, and then updated the side panel so as to provide him with tactics and warnings.

But what’s the use in displaying coordinates? I hear you scream. True, Arkwood is hardly going to have time to check out the latest coordinates of the ghosts whilst he’s immersed in gameplay. So let’s update our code to show green bars:

pacman_objectdetection_marktwo_nw_705

What the hell is that? you growl, searching your house for a crowbar with which to assault me. Put the weapon down and allow me to explain. The green bars are indicating that, on average, the ghosts are moving north-westerly i.e. towards the top-left side of the screen. Now let’s consider the following image:

pacman_objectdetection_marktwo_se_671

This time the ghosts, on average, are moving south-easterly. All my grubby friend need do is glance at the side panel, to check the flow of the ghosts. Surely this is a promising first step in providing him with tactics and warnings, so as to put his name in the record books.

Okay, so let me show you how I have updated the code from my previous post in order to calculate the average flow of the ghosts in Pac-Man.

# set coordinates
def set_coordinates(self, coordinates):
    self.previous_coord = self.current_coord
    self.current_coord = coordinates

# get direction
def get_direction(self):
    x_direction = self.current_coord[0] - self.previous_coord[0]
    y_direction = self.current_coord[1] - self.previous_coord[1]

    return (x_direction, y_direction)

I’ve added two new methods to the Character class, the class which represents Pac-Man and the four ghosts.

The set_coordinates method allows each character to maintain a record of their current and previous coordinates.

The get_direction method uses the current and previous coordinates in order to calculate the direction that the character is moving in.

# get flow of characters
def get_flow(flow, character):
    if character.name == pacman.name:
        return flow

    direction = character.get_direction()
    return (flow[0] + direction[0], flow[1] + direction[1])

I’ve added a get_flow function. It adds together a global flow variable with the current character’s direction (excluding the Pac-Man character, as we only want to monitor our ghosts for now).

# draw flow of characters
def draw_flow(flow, image):
    negative_threshold = -20
    positive_threshold = 20

    # draw East or West bar
    if(flow[0] < negative_threshold): 
        cv2.rectangle(image,(0,0),(50,850),(0,255,0),-1)
    elif(flow[0] > positive_threshold):   
        cv2.rectangle(image,(1250,0),(1300,850),(0,255,0),-1)
        
    # draw North or South bar
    if(flow[1] < negative_threshold):             
        cv2.rectangle(image,(0,0),(1300,50),(0,255,0),-1)
    elif(flow[1] > positive_threshold):
        cv2.rectangle(image,(0,800),(1300,850),(0,255,0),-1)

    # print flow
    cv2.putText(image, "x {} : y {}".format(flow[0], flow[1]), (100, 100), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0))

I’ve added a draw_flow function, which is executed once all of the ghosts have updated the global flow variable with their direction. If the x-coordinate of the flow is a negative number, then, on average, the ghosts are moving west (to the left of the screen). If the flow’s y-coordinate is negative then, on average, the ghosts are moving north (to the top of the screen). For positive numbers, the ghosts are moving east and south. We draw green bars on the image – dependant on a threshold – to represent the flow of the ghosts. We also print the flow coordinates on the image.

OpenCV provides functions for drawing rectanges and other such shapes.

Let’s look at a series of images, where the ghosts are moving in a north-easterly direction:

pacman_objectdetection_marktwo_series_ne1_765

pacman_objectdetection_marktwo_series_ne2_766

pacman_objectdetection_marktwo_series_ne3_767

The green bar at the top of the screen indicates that the ghosts are moving north, hence the negative y-coordinate value.

The green bar at the right of the screen indicates that the ghosts are moving east, hence the positive x-coordinate value.

What about this series of images?

pacman_objectdetection_marktwo_series_sw1_560

pacman_objectdetection_marktwo_series_sw2_561

pacman_objectdetection_marktwo_series_sw3_562

The green bar at the bottom of the screen indicates that the ghosts are moving south, hence the positive y-coordinate value. In the last shot the y-coordinate slips below the threshold and the green bar disappears.

The green bar at the left of the screen indicates that the ghosts are moving east, hence the negative x-coordinate value. In the last shot the x-coordinate rises above the threshold and the green bar appears.

And that is that!

‘How are you finding the green bars?’ I asked Arkwood, who was taking a comfort break from playing Pac-Man.

‘It’s top-drawer bro. I’ve got the ghouls under my paw for sure, dude.’

I had no idea what he was talking about, so left him sitting on the toilet with his trousers around his ankles and tissue paper in his hand. Of course, I’d rather talk to him over a cup of tea and some digestive biscuits, but he’s never off that bloody Pac-Man game. It’s the way of the true gamer, he tells me.

P.S.

Here’s the updated code. First up, the main program:

import cv2
import ImageGrab
import numpy
from character import Character

# grab screenshot
def grab_screenshot():
    screenshot = ImageGrab.grab(bbox=(0,50,1300,900))
    return cv2.cvtColor(numpy.array(screenshot), cv2.COLOR_RGB2BGR)

# get contours from image
def get_contours(image):
    edges = cv2.Canny(image, 100, 200)
    contours, hierarchy = cv2.findContours(edges, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    return contours

# get contour centroid
def get_contour_centroid(contour):
    try:
        M = cv2.moments(contour)
        cx = int(M['m10']/M['m00'])
        cy = int(M['m01']/M['m00'])
        return (cx, cy)
    except:
        return (0, 0)

# draw contour detail on image
def draw_contour(contour, image, coord, character):
    cv2.drawContours(image, [contour], -1, (0, 255, 0), 3)
    cv2.putText(image, "{} {}".format(character.name, coord), (coord[0] + TEXT_OFFSET, coord[1]), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0))

# get flow of characters
def get_flow(flow, character):
    if character.name == pacman.name:
        return flow

    direction = character.get_direction()
    return (flow[0] + direction[0], flow[1] + direction[1])

# draw flow of characters
def draw_flow(flow, image):
    negative_threshold = -20
    positive_threshold = 20

    # draw East or West bar
    if(flow[0] < negative_threshold): 
        cv2.rectangle(image,(0,0),(50,850),(0,255,0),-1)
    elif(flow[0] > positive_threshold):   
        cv2.rectangle(image,(1250,0),(1300,850),(0,255,0),-1)
        
    # draw North or South bar
    if(flow[1] < negative_threshold):             
        cv2.rectangle(image,(0,0),(1300,50),(0,255,0),-1)
    elif(flow[1] > positive_threshold):
        cv2.rectangle(image,(0,800),(1300,850),(0,255,0),-1)

    # print flow
    cv2.putText(image, "x {} : y {}".format(flow[0], flow[1]), (100, 100), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0))

# constants
HISTORY = 12
SAMPLE_RATE = 1
PIXEL_OFFSET = 10
TEXT_OFFSET = 50

# set up characters
pacman = Character("Pac-Man", [114, 213, 205], [134, 233, 225], True)
red_ghost = Character("Blinky", [54, 63, 136], [74, 83, 156], True)
pink_ghost = Character("Pinky", [113, 117, 188], [133, 137, 208], True)
green_ghost = Character("Inky", [66, 169, 102], [86, 189, 122], True)
orange_ghost = Character("Clyde", [32, 95, 145], [52, 115, 165], True)

characters = [pacman, red_ghost, pink_ghost, green_ghost, orange_ghost]

# set up background subtraction
fgbg = cv2.BackgroundSubtractorMOG()

# set variables
flow = (0,0)
sample_counter = 0 

while True:

    # apply background subtraction
    frame = grab_screenshot()
    fgmask = fgbg.apply(frame, learningRate=1.0/HISTORY)
    fgoutput = cv2.cvtColor(fgmask, cv2.COLOR_GRAY2RGB)

    # detect objects at set interval
    if sample_counter % SAMPLE_RATE == 0:
        
        # get contours for objects in foreground
        contours = get_contours(fgmask)
        contours = sorted(contours, key=cv2.contourArea, reverse=True)[:10]

        # find characters in foreground objects
        for contour in contours:

            # get centre coordinates of object
            coord = get_contour_centroid(contour)
            
            # extract colour from object
            colour = frame[coord[1] - PIXEL_OFFSET, coord[0]]

            # loop all characters and attempt to match colour
            for character in characters:
                if character.enabled and character.is_colour_match(colour):
                    character.set_coordinates(coord)
                    flow = get_flow(flow, character)
                    draw_contour(contour, fgoutput, coord, character)
                    character.enabled = False
                    break

        # draw flow
        draw_flow(flow, fgoutput)

    # save image to disk
    cv2.imwrite('Images/pacman.jpg', fgoutput)

    # re-enable characters
    for character in characters:
        character.enabled = True

    # set variables
    flow = (0,0)
    sample_counter += 1

And here’s the Character class:

class Character(object):
          
    # initialise the character
    def __init__(self, name, lower_colour, upper_colour, enabled):
        self.name = name
        self.lower_colour = lower_colour
        self.upper_colour = upper_colour
        self.enabled = enabled
        self.current_coord = (0,0)
        self.previous_coord = (0,0)

    # check for colour match
    def is_colour_match(self, colour):
        for i in range(3):
            if colour[i] < self.lower_colour[i] or colour[i] > self.upper_colour[i]:
                return False
        
        return True

    # set coordinates
    def set_coordinates(self, coordinates):
        self.previous_coord = self.current_coord
        self.current_coord = coordinates

    # get direction
    def get_direction(self):
        x_direction = self.current_coord[0] - self.previous_coord[0]
        y_direction = self.current_coord[1] - self.previous_coord[1]

        return (x_direction, y_direction)

Note: I’ve amended the sample rate from the previous post, setting it at ‘1’. We could just get rid of the sample rate, but I find it a handy value to tweak. For example, if we find that the code is taking too long to execute, and thus having a negative effect on background subtraction, we can simply adjust the sample rate.

Due to the ghosts’ random nature, the green bars do fluctuate a fair bit. The thresholds associated with the green bars are useful, therefore, in only reporting significant flow.

Advertisements