, , , , , , , , , , , , ,

Arkwood has decided to become an accountant.

‘They get paid tons and do little,’ he said, matter-of-fact.

I told him that I was impressed by his get-up-and-go.

‘One problem though,’ I continued, ‘You can’t actually do basic maths.’

It’s true, Arkwood is unable to add 2 and 2 together. And that’s bad. He started to wail like a baby, tears streaming down his flush cheeks.

Not wishing to see my Belgian buddy suffer another moment of hysteria, I set about adding a new feature to SaltwashAR – the Python Augmented Reality application. I will build a Robot Calculator which will teach Arkwood elementary mathematics, namely: Add, Subtract, Multiply, Divide.

Here’s the new class for my Calculator feature:

from features.base import Feature, Speaking
class Calculator(Feature, Speaking):
    ADD = ['+', 'plus', 'added']
    SUBTRACT = ['-', 'minus', 'subtracted']
    MULTIPLY = ['x', 'times', 'multiplied']
    DIVIDE = ['/', 'divided']

    def __init__(self, text_to_speech, speech_to_text):
        Speaking.__init__(self, text_to_speech)
        self.speech_to_text = speech_to_text
    def _thread(self, args):
        # user provides expression
        expression = self.speech_to_text.convert()
        if not expression: return

        # check whether to stop thread
        if self.is_stop: return

        # calculate answer from expression
        expression = expression.lower().split()
        if len(expression) < 3: return       

        expression_left = expression[0]
        expression_middle = expression[1]
        expression_right = expression[-1]

        answer = None

            if expression_middle in self.ADD:
                answer = int(expression_left) + int(expression_right)
            elif expression_middle in self.SUBTRACT:
                answer = int(expression_left) - int(expression_right)
            if expression_middle in self.MULTIPLY:
                answer = int(expression_left) * int(expression_right)
            if expression_middle in self.DIVIDE:
                answer = int(expression_left) / float(expression_right)
            print "Unable to calculate an answer"

        # calculator provides answer
        if answer:
            self._text_to_speech("The answer is {}".format(answer))

Our Calculator class is inheriting from the Feature base class, which provides the threading functionality (all features need to run in threads, so as not to block the main application process from rendering to screen).

The Calculator class also inherits from the Speaking base class, which ensures that the robot’s mouth moves whenever it talks.

Next, we define some constant lists for Add, Subtract, Multiply and Divide – these lists will allow us to match the words we speak through the computer microphone to a mathematical calculation.

The __init__ method is passed parameters for Text To Speech and Speech To Text, allowing us to have a conversation with the robot.

Okay, next up is the _thread method, which is where all the shit happens, as a professor of calculus would say.

We use Speech To Text to convert the words we utter into the computer mic to text.

Then we check whether the thread should be stopped. Why? Well, if the robot is no longer in front of the webcam once our words are converted to text, there is no point in doing the calculation (as there will be no robot to provide the answer).

Now to the actual maths. We split our spoken words into a Python list and – assuming we have at least three words – we organise the first, second and last word into left, middle and right parts of an expression.

For example, if we said the words “nine subtracted by five” into the computer microphone, then the expression variables would be:

expression_left = ‘9’

expression_middle = ‘subtracted’

expression_right = ‘5’

Hang on! I hear you cry. How come you said the word “nine” and the expression_left variable has the string ‘9’? Well, I am using Google Speech Recognition which handily converts my spoken word “nine” to ‘9’, ready for use in a calculation. If this was not the case, I’d have done the conversion myself.

Great. Now that I have the parts of my expression at hand, I can simply check which of my constant lists contain the expression_middle value.

Continuing with the example above, the expression_middle value of ‘subtracted’ can be found in the SUBTRACT constant list. Therefore, we drop into the calculation expression_left – expression_right (which is 9 – 5) and obtain an answer of ‘4’.

Finally, we use Text To Speech to let the robot speak the answer “4” through the computer speakers.

Time for a demo – here’s Rocky Robot giving us the answers to our mathematical conundrums:

Fantastic – a talking calculator! Arkwood will soon learn the times table and get that high-paid job as an accountant.

Please check out the SaltwashAR Wiki for details on how to install and help develop the SaltwashAR Python Augmented Reality application.

SaltwashAR is an open-source Python Augmented Reality application released under the GNU General Public Licence Version 3.