Tags

, , , , , , , , , , , , ,

SaltwashAR is a Python Augmented Reality application. Out of the box, it lets us render 3D robots upon 2D markers:

Github_SaltwashAR_OutOfTheBox

Rocky Robot and Sporty Robot in all their splendor!

But it is SaltwashAR Features that let us interact with the robots. For example, we can ask the robots to search the web, translate phrases or play card games. The robots can respond to our hand gestures, read printed words or watch TV.

All these features – and more – are already available once you install SaltwashAR on your computer. For details of how to install and use SaltwashAR, check out the SaltwashAR Wiki.

It is easy to add your own feature to SaltwashAR. So let’s do it!

Plumbing in a new feature

Okay, first we will add the basic shell of a feature to SaltwashAR.

appsettings.ini

Add a new entry into the appsettings.ini config file, so that the feature can be switched on and off:

Copycat=True

We will call our new feature Copycat. But you will have to wait until the next section to find out why. No peeking!

scripts/configprovider.py

Add a new entry into the configprovider.py class, so as to make our new setting available to the application:

@property 
def copycat(self):
    return self.config.getboolean("Features", "Copycat")

scripts/features/copycat

Okay, here’s where we get to create our new feature.

First, create a new folder in the SaltwashAR application called ‘copycat’ (under scripts/features).

Add two new Python files called ‘__init__.py’ and ‘copycat.py’.

Add the following code to __init__.py:

from copycat import Copycat

Add the following code to copycat.py:

from features.base import Feature

class Copycat(Feature):

    def __init__(self):
        Feature.__init__(self)
    
    def _thread(self, args):
        print "Copycat thread is running..."

Our Copycat class is inheriting from a Feature base class, which will provide all the threading functionality. Features need to run in threads so as not to block the main application process from rendering to screen.

All our Copycat class is doing is printing to a console window the words “Copycat thread is running…” whenever the thread is run. That’s fine for now – we can add more functionality later, once we have tested that the new feature is working.

scripts/features/features.py

The last thing we need to do is add some entries into the features.py file, which is responsible for orchestrating requests and responses between features and the main application.

Add the following code block to the foot of the __init__ method:

self.copycat = None
if config_provider.copycat:
    from copycat import Copycat
    self.copycat = Copycat()

As you can see, we only initialize our Copycat class if it is enabled in the appsettings.ini config file.

Next, add a new _handle_copycat method:

# handle copycat
def _handle_copycat(self, rocky_robot, sporty_robot):
    if not self.copycat: return

    if rocky_robot.is_facing or sporty_robot.is_facing:
        self.copycat.start()
    else:
        self.copycat.stop()

We bail out of the method if our Copycat class is disabled in the appsettings.ini config file. Only if Rocky Robot or Sporty Robot is facing the webcam, do we start our Copycat thread – otherwise, we stop the thread.

We need to call this new method from the foot of the handle method:

self._handle_copycat(rocky_robot, sporty_robot)

Great, we have now plumbed a new feature into SaltwashAR!

Let’s run the application and watch our thread print to the console window whenever one of our robots is facing the webcam:

SaltwashAR_Copycat_Shell

Implementing the new feature

In the previous section we added the basic shell of our new Copycat feature to SaltwashAR. But printing words to a console window does not constitute interacting with a robot. So let’s add some more code!

scripts/features/copycat/copycat.py

Update the copycat.py file to the following code:

from features.base import Feature, Speaking

class Copycat(Feature, Speaking):

    def __init__(self, text_to_speech, speech_to_text):
        Feature.__init__(self)
        Speaking.__init__(self, text_to_speech)
        self.speech_to_text = speech_to_text
    
    def _thread(self, args):
        # user speaks
        text = self.speech_to_text.convert()
        if not text: return

        # check whether to stop thread
        if self.is_stop: return

        # copycat speaks
        self._text_to_speech(text)

So what’s changed?

We are now inheriting from a Speaking base class (as well as the Feature base class). The Speaking class will ensure that the robot’s mouth moves when it speaks to us.

The __init__ method is being passed parameters for Text To Speech and Speech To Text.

The _thread method is dead simple. We make use of the built-in Speech To Text functionality of SaltwashAR to speak into our computer microphone. For example, we say the words “I like tea and toast” when prompted by the application console window (the console window will prompt with the text “listening…”).

If our spoken words are picked up correctly, the robot will make use of the built-in Text To Speech functionality of SaltwashAR to speak the same words back to us: “I like tea and toast”.

And we now have a copycat robot!

P.S. notice we also check whether to stop the thread. If the robot is no longer in front of the webcam once our speech has been converted to text, then there is no use in making the robot reply to us. Instead, we bail out of the thread.

scripts/features/features.py

A couple of tweaks are required to the features.py file. First, the __init__ method:

self.copycat = None
if config_provider.copycat:
    from copycat import Copycat
    self.copycat = Copycat(text_to_speech, speech_to_text)

As you can see, we are now passing the Text To Speech and Speech To Text parameters to our Copycat class.

We also need to amend the Text To Speech and Speech To Text sections of the __init__ method, adding a clause to their if statements for our new Copycat feature:

config_provider.copycat

Note: since we are making use of the Text To Speech and Speech To Text functionality in our Copycat feature, be sure to install the feature dependencies pyttsx 1.1 and SpeechRecognition 3.1.3

Also, the is_speaking method needs a similar amendment to its return statement:

self.copycat and self.copycat.is_speaking

Nice. With these amendments we are able to speak to our robot and the robot is able to speak to us (and move its mouth to boot!).

Demonstration

Let’s see the robot in action, copying our words:

Wow! The cheeky robot is repeating our phrase: “I like tea and toast”.

Summary

Hopefully you can now plumb in and implement a new feature for SaltwashAR. Of course, drop me a line if you have any difficulties.

The sky is the limit on what you can do with SaltwashAR Features. You can use any Python package you like to make the robots smart, whether that be via online data, artificial intelligence, computer vision, audio or so on. The robots even have some basic emotions that you can trigger, such as happy or angry.

If you do add a new feature to SaltwashAR, be sure to let me know. SaltwashAR is an open-source Python Augmented Reality application released under the GNU General Public Licence Version 3. I want as many cool features available to everyone!

Ciao!