, , , , , , , , , , , , ,

Arkwood screamed, ‘You left the toilet seat up again!’ Since when did he care?

Anyway, I told him I would pay more attention the next time I concluded my extraction of liquid toxins. I gave him the Okay gesture with my hand, to let him know that I sympathised with his grievance. But when his back was turned, I gave him the Vicky sign.

Why am I telling you all this? Well, it is a convenient bridge into a discussion about SaltwashAR, my Python Augmented Reality application. I will add a new feature to the app, so that the robots can detect hand gestures and respond accordingly.

For example, if I give the robot an Okay hand gesture then she will reply “Hi there, buddy”.


But if I give the robot a Vicky hand gesture, she will reply “Well, there is no need to be so rude!”.


How amusing it will be. Let’s go!

Building a feature for SaltwashAR

It’s easy to build a new feature for SaltwashAR. All we need is a Python class that implements a thread. Here’s the basic template:

class MyFeature:
    def __init__(self):
        ### Put initialization here ###

    def start(self):
        ### Start thread here ###

    def stop(self):
        ### Stop thread here ###

    def _thread(self):
        ### Put thread logic here ###

Let’s fill in the class, so as to implement the hand gesture feature.

First, the initialization code:

from threading import Thread
import cv2
from texttospeech import TextToSpeech

class HandGesture:
    def __init__(self):
        self.thread = None
        self.is_stop = False
        self.is_speaking = False
        self.text_to_speech = TextToSpeech()

We initialize class variables for our thread, for a Stop and Speaking flag, and also Text To Speech capability.

Next, the start method:

def start(self, image):
    self.is_stop = False

    if self.thread and self.thread.is_alive(): return

    self.thread = Thread(target=self._thread, args=(image,))

First, we set the Stop flag to False. Next, we need to check if the thread is already running, by way of its is_alive method. Finally we start our thread.

But why are we using a thread? Because our Augmented Reality application is drawing to the screen many times a second – if we wait for our logic to finish, the screen will freeze.

def stop(self):
    self.is_stop = True

The stop method is simple. It stops the thread.

Right – here’s where the fun happens. Let’s implement the thread logic, so that we can communicate with the robot via hand gestures:

def _thread(self, image):
    # detect hand gesture in image
    is_okay = self._is_item_detected_in_image('classifiers/haarcascade_okaygesture.xml', image.copy())
    is_vicky = self._is_item_detected_in_image('classifiers/haarcascade_vickygesture.xml', image.copy())

    # check whether to stop thread
    if self.is_stop: return

    # respond to hand gesture
    if is_okay:
        self._text_to_speech("Hi there, buddy")
    elif is_vicky:
        self._text_to_speech("Well, there is no need to be so rude!")

First, we attempt to detect the Okay and Vicky hand gestures in our latest webcam image, using OpenCV Haar Feature-based Cascade Classifiers (more on that below).

Next, we check if the Stop flag has been set by the stop method. We need to check at convenient points in our thread whether a request has been made to halt the logic.

Finally, we use Text To Speech so that the robot can respond to our hand gesture. If we make the Okay hand gesture then the robot will respond “Hi there, buddy”. But if we make the vile Vicky hand gesture then the robot responds “Well, there is no need to be so rude!”.

Splendid. And that is that! We have just implemented a new feature for SaltwashAR.

The Hand Gesture class has two extra private methods, to support the thread logic. Here’s the _is_item_detected_in_image method:

def _is_item_detected_in_image(self, classifier_path, image):

    classifier = cv2.CascadeClassifier(classifier_path)
    gray_image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
    items = classifier.detectMultiScale(gray_image, scaleFactor=1.1, minNeighbors=4, minSize=(200, 260))

    return len(items) > 0

My post Hand gestures with OpenCV and OpenGL explains how to create and use OpenCV Haar Feature-based Cascade Classifiers to detect hand gestures.

And the _text_to_speech method:

def _text_to_speech(self, text):
    self.is_speaking = True
    self.is_speaking = False

Notice that we are wrapping the Text To Speech call around the Speaking flag. The flag will let our AR app know when the robot is talking, so we can animate its mouth!

Plumbing the feature into SaltwashAR

Now that we have our new hand gesture feature for SaltwashAR, let’s plumb it in and see it work.

First, we will add a new entry into the appsettings.ini file for hand gesture:


And put a new property into our configprovider.py script:

def hand_gesture(self):
    return self.config.getboolean("Features", "HandGesture")

This way, we will only initialize our Hand Gesture class if it is enabled in config:

# initialise hand gesture
self.hand_gesture = None
if self.config_provider.hand_gesture:
    self.hand_gesture = HandGesture()

Why does the new feature need to be enabled via a settings file? Well, the plan is to have hundreds of features for SaltwashAR – clearly they can not all be enabled at the same time, so we will use the appsettings file to switch them on and off.

The initialization code is added to SaltwashAR class __init__ method in the main.py script. Be sure to add the import statement as well:

from handgesture import HandGesture

We also need to add a new method call to the _draw_scene method:

# handle hand gesture

And then implement that new method:

def _handle_hand_gesture(self, image):

    # check hand gesture instantiated
    if not self.hand_gesture: return

    # handle hand gesture
    if self.rocky_robot.is_facing or self.sporty_robot.is_facing:

We will start a thread for hand gesture detection if one of our robots is facing the webcam. Otherwise, we will stop the thread.

Finally, when we come to draw our robot to screen in the _handle_glyphs method, we need to check the Speaking flag on our Hand Gesture class. If it is True, we will animate the robot’s mouth!

is_speaking = (self.browser and self.browser.is_speaking) \
                or (self.hand_gesture and self.hand_gesture.is_speaking)

Using the feature in SaltwashAR

What a triumph! We have just built and plumbed in a new feature for SaltwashAR. Let’s have a look at it in action:

Rocky Robot is not best pleased when we show her the Vicky sign.

And if you were wondering how our OpenCV Classifiers got on detecting our hand gestures:



Yep. My debug code – to draw a rectangle around detected items and save the image to disk – clearly shows our hand signs being matched.


Building new features for SaltwashAR is easy, as the application already has core Augmented Reality technology for detecting 2D markers and rendering 3D robots.

The sky is the limit for what these new features can do. We can plumb in computer vision such as our hand gesture feature above. We can use Speech To Text and Text To Speech to ask our robots to fetch web content and read it out to us, as per our existing Browser feature. We can add Artificial Intelligence to the robots. Let them detect changes in light and temperature. Maybe play musical notes of our choosing.

Do you want to help develop SaltwashAR into a varied and immersive AR experience? Just drop me a line.

You can build a new feature simply by letting me know your idea (I’ll code it up and add it to SaltwashAR, if feasible). Or you can write the new feature using the Python template above, and I will test it out and plumb it into SaltwashAR. Or you can even fork SaltwashAR on Github and submit a pull request. Whatever your preference, feel free to get involved.

As for me, I’m off to see the great man himself.