, , , , , , , , , , , , ,

Peters, my grotesque Dutch lodger, has won every card game against me. But I have a smart plan to win all my money back.

‘Peters. I challenge you to a game of the fruit machine!’ I announced.

Those little piggy eyes of his did emit a spark or two.

‘You’re on! But, please, let us play to the international standards of the slot machine.’


I added a new Fruit Machine feature to SaltwashAR, the Python Augmented Reality application, so that a robot croupier can guide us through the game.

Without further to do, let’s take a gander at the augmented reality fruit machine:


Rocky Robot, our croupier, asks us to say the word Start. Upon which, he spins the fruits.

If we get the same fruit on all three reels of the fruit machine then we win 50 coins and Rocky’s head bobs with happiness. Otherwise we lose 10 coins and Rocky’s head slumps in despair.

The fruits can spin at different speeds, for different degrees of rotation, and we can even use motion detection to HOLD up to two fruits!

‘Wait a minute!’ Peters exclaimed, placing his Ginsters pasty on the arm of the sofa. ‘For all I know, you could have programmed the slot machine to work in your favour.’

I did not take kindly to his accusation, and told him so – by way of punching him in the scrotum. Once he got his breath back, Peters smacked me around the head with a full 1.5 litre bottle of Coke. I yanked the flowery hood off the table lamp and administered an electric shock upon his rotund frame, which threw him into a spasm against the living room wall. The game was abandoned.

Building the Fruit Machine feature

So how the devil did I build the SaltwashAR Fruit Machine? Turns out to be rather as any other SaltwashAR feature – let’s take a peek at the Python code:

from features.base import *
from detection import Detection
import numpy as np
from reels import *
from random import randint
import time
from constants import *

class FruitMachine(Feature, Speaking, Emotion):

    def __init__(self, text_to_speech, speech_to_text):
        Speaking.__init__(self, text_to_speech)
        self.speech_to_text = speech_to_text
        self.background_image = np.array([])
        self.detection_image = np.array([])
        self.detection = Detection()
        self.reels = [None, None, None]
        self.holds = [None, None]
        self.coins = 100

The FruitMachine class inherits from the Feature base class, which provides threading (all features run in threads so as not to block the main application process from rendering to screen). We also inherit from the Speaking base class, to let the robot’s mouth move when he speaks. And the Emotion base class, to let the robot be happy or sad.

The class __init__ method is passed a Text To Speech parameter, so that the robot can talk to us, and a Speech To Text parameter, so that we can talk to the robot. We have class instance variables to store and inspect images sent from the webcam, as well as to keep track of the fruit machine reels, HOLD selections, and the amount of coins we have.

# start thread
def start(self, args=None):
    Feature.start(self, args)
    self.background_image = args
    self.detection_image = args.copy()

    # draw holds
    self.background_image = draw_holds(self.holds, self.background_image)

    # rotate and draw reels
    self.reels = rotate_reels(self.reels)

# stop thread
def stop(self):
    self.background_image = np.array([])

Next, we have the start method, which takes care of kicking off a thread for our feature. It’s also a handy place to store our latest webcam image, draw the HOLD selections we have made, and to spin and draw our fruit machine reels.

We also have a stop method, to clear the background image if the robot is no longer in front of the webcam.

# run thread
def _thread(self, args):

    # check player has coins
    if self.coins == 0:
        self._text_to_speech("Sorry dude, you're all out of cash")

    # on occasion, allow player to hold reels
    if (not None in self.reels) and (randint(0,2) == 0):

        # croupier tells player that one or two reels can be held
        self._text_to_speech("If you want to hold one or two fruits, press them now")

        # player selects holds

        for i, hold in enumerate(self.holds):
            timeout = time.time() + 5

            while True:
                active_cell = self.detection.get_active_cell(self.detection_image)

                if (active_cell != None) and (active_cell not in self.holds):
                    self.holds[i] = active_cell

                if time.time() > timeout:

            if self.holds[i] == None: break

    # croupier asks player if ready to spin reels
    self._text_to_speech("Just say the word Start, and I'll spin the fruits")

    # wait until player says "start"
    while self.speech_to_text.convert() != "start": continue

    # refresh reels
    self.reels = refresh_reels(self.reels, self.holds)

    # wait while reels rotate
    while is_reels_rotating(self.reels):

    # clear any holds
    self.holds = [None, None]

    # determine if player has won or lost
    if is_reels_win(self.reels):
        self.coins += 50
        self._text_to_speech("Wow, you won! You now have {} coins".format(self.coins))
        self.coins -= 10
        self._text_to_speech("Damn, you lost! You now have {} coins".format(self.coins))

Okay, here’s the feature’s _thread method, where all the shit happens (as a toilet attendant would say)…

First we check whether the player has any coins left – if not, Rocky Robot uses Text To Speech through the computer speakers to tell us “Sorry dude, you’re all out of cash” and won’t let us play (the swine!).

Next, we have a 1 in 3 chance of being allowed to HOLD up to two reels of the fruit machine. Rocky will announce “If you want to hold one or two fruits, press them now”, and then OpenCV motion detection is used to determine whether our hand is groping for a fruit. We have a time limit in which to select two fruits.

Note: don’t fret, I’ll go into the motion detection stuff later in the post.

Right, now the robot says “Just say the word Start, and I’ll spin the fruits”. So I employ Speech To Text through my computer microphone to utter the word “start”.

We refresh the reels of our fruit machine (assuming that they have not been selected as HOLD). But what does refreshing a reel mean? Well, we assign it a new pattern, for example:

{'current_rotation': 0, 'total_rotation': 450, 'speed': 3 }

This reel pattern will rotate 450 degrees (which is in effect a full rotation plus a quarter). It will rotate at a speed of 3 degrees each cycle. Note that when we assign a reel a new pattern, we have to pass across values from the previous pattern (otherwise the reel will lose its current rotation position).

With the reels refreshed, we simply wait until the aforementioned start method has finished rotating our reels on their new patterns.

Finally, we can check whether the reels are all displaying the same fruit.

If so, we add 50 coins to our purse and our robot croupier announces our tidy profit. Rocky Robot bobs his head in joy (not sure why he is so happy, handing over his employer’s money?)

If the fruits are not all the same then we are deducted 10 coins, and Rocky Robot tells us the sad news with a tear in his eye (crocodile tears, if you ask me).

And that is that! We can now gamble night and day until we have lost our house, car, wife and children and are lying in some gutter with a brown paper bag bottle of gin in our scabby, frostbitten hands.

Adding motion detection

OpenCV computer vision provides the motion detection of our hand groping for a reel on the fruit machine.

Here’s the image from the webcam, with my hand reaching for a fruit:


Where is the fruit? I hear you cry. Well, we don’t want to draw the fruit on the image we are using for motion detection – instead, the fruit is drawn on a separate background image.

And here’s the motion detection of my paw:


I have added green lines to the motion detection image to show the three cells where our fruit machine reels reside. As you can see, my hand has been detected in the third cell (bottom right) – so the third reel will be selected to HOLD.

Here’s the Detection class:

import cv2
import numpy as np

class Detection(object):
    THRESHOLD = 1500
    def __init__(self):
        self.previous_gray = None
    def set_previous_image(self, image):
        self.previous_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    def get_active_cell(self, image):
        # obtain motion between previous and current image
        current_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        delta = cv2.absdiff(self.previous_gray, current_gray)
        threshold_image = cv2.threshold(delta, 25, 255, cv2.THRESH_BINARY)[1]

        # set cell height and width
        height, width = threshold_image.shape[:2]
        cell_height = height/2
        cell_width = width/3
        # store motion level for each cell
        cells = np.array([0, 0, 0])
        cells[0] = cv2.countNonZero(threshold_image[cell_height:height, 0:cell_width])
        cells[1] = cv2.countNonZero(threshold_image[cell_height:height, cell_width:cell_width*2])
        cells[2] = cv2.countNonZero(threshold_image[cell_height:height, cell_width*2:width])
        # obtain the most active cell
        top_cell =  np.argmax(cells)
        # return the most active cell, if threshold met
        if(cells[top_cell] >= self.THRESHOLD):
            return top_cell
            return None

Once we have a threshold image – which is the difference between the previous and current webcam image – we can inspect each of the three cells at the base of the image for the most amount of motion. If the motion has stirred a certain number of pixels (in this case, 1500 pixels) then we return the wining cell (otherwise, we return None).

Where do I get this code?

All the code for the Fruit Machine feature is at SaltwashAR GitHub, including the code to control and draw the reels.

The SaltwashAR Wiki has detail on how to install and help develop the SaltwashAR Python Augmented Reality application.

Any questions, feel free to ask.

But for now, ciao!