Tags

, , , , , , , , , , ,

Arkwood has twisted his ankle, and as such is stretched out at home on the sofa. ‘I need a remote control,’ he told me, ‘so that I can request toast and beer from my mum in the kitchen.’ I told him that I could write some Python code on the Raspberry Pi to handle his food orders. But first off, let’s put together the actual remote control…

I will use the RasWIK Wireless Inventors Kit as the remote control. It comprises of an Arduino board, with jump wires to a small breadboard which will house the buttons and bulb of our remote control. It can be placed up to 100m away from our Raspberry Pi, communicating with a radio component slotted onto the Pi. Here’s a photo of the remote control set up with two buttons for requesting beer and toast, plus a bulb which will light up to confirm that Arkwood’s mother has received his order:

pi_remotecontrol_raswiksetup

Great. Now we have the remote control for Arkwood to use in the living room, all we need is to write a program on the Raspberry Pi in the kitchen (so that his mum can receive his orders):

from raswik import Raswik
from webcam import Webcam
from speech import Speech
from time import sleep

raswik = Raswik()
webcam = Webcam()
speech = Speech()

#loop until hole in ozone
while True:
    toast = False
    beer = False
    
    # detect button press on remote control
    if(raswik.is_button_pressed('a--D10READ--')): 
        print("Toast button pressed")
        toast = True
    elif(raswik.is_button_pressed('a--D12READ--')): 
        print("Beer button pressed")
        beer = True
    else:
        continue
    
    # wait until motion detected in kitchen
    webcam.detect_motion()

    # ask mum for toast or beer
    if(toast):
        speech.text_to_speech("I want toast with lashings of butter")
    elif(beer):
        speech.text_to_speech("I want beer and make it cold")

    # get mum's reply
    reply = speech.speech_to_text('/home/pi/PiAUISuite/VoiceCommand/speech-recog.sh')

    # send success to remote control
    if(reply == "yes"):
        if(raswik.switch_bulb('a--D05HIGH--')):
            print("Success bulb switched on")
            sleep(10)
            raswik.switch_bulb('a--D05LOW---')

Let’s step through this code. Once in a loop, we first check whether Arkwood has pressed either of the RasWIK buttons on the remote control in the living room. If not, we just keep looping until he has. Once a toast or beer order is received, we launch our webcam and wait until we detect his mum moving about in the kitchen. When we spot her, we use Google’s Text To Speech service to announce through speakers attached to the Raspberry Pi that we want either toast or beer, and then use Google’s Speech To Text service to capture through a microphone her reply. If her reply is ‘yes’ then we send a signal from the Pi in the kitchen back to the RasWIK remote control in the living room, lighting up a green bulb to inform Arkwood that his order is on its way. If his mum says anything else (or perhaps says nothing at all) then the bulb will not light up and Arkwood will either have to resend his request or take to his crutches.

Here’s the Raswik class I have written, to take care of the wireless communication between the RasWIK remote control and the Raspberry Pi:

import serial
from time import sleep
  
# class to handle communication to
# raswik wireless inventors kit
class Raswik(object):
  
    # com port and speed
    PORT = "/dev/ttyAMA0"
    BAUD = 9600
  
    #check if a button has been pressed
    def is_button_pressed(self, pin):
   
        #check pin param
        if(pin == ""):
            return False
  
        reply = ""
        try:
            # establish a serial connection
            ser = serial.Serial(port= self.PORT, baudrate= self.BAUD)          

            # wait a mo...
            sleep(0.2)
  
            # write to pin e.g. a--D10READ--
            ser.write(pin)
      
            # wait some more...
            sleep(0.2)
      
            # now read from pin
            reply = ser.read(12)
  
            # close serial connection
            ser.close()
        except:
            print ("Error detecting raswik button press")
            return False
  
        # grab the bit of the reply we care about
        message = reply[6:].strip('-')
  
        # return whether button is currently pressed down
        if(message == "LOW"):
            return True
        else:
            return False

    # switch a bulb on or off
    def switch_bulb(self, pin):
        
        #check pin param
        if(pin == ""):
            return False
        
        try:
            # establish a serial connection
            ser = serial.Serial(port= self.PORT, baudrate= self.BAUD)          
  
            # wait a mo...
            sleep(0.2)
  
            # write to pin e.g. a--D05HIGH-- or a--D05LOW---
            ser.write(pin)

            # wait some more...
            sleep(0.2)

            # close serial connection
            ser.close()
        except:
            print ("Error detecting raswik bulb")
            return False

        return True

Here’s the Webcam class, which detects motion:

import cv2
from datetime import datetime

class Webcam(object):

    WINDOW_NAME = "Arkwood's Surveillance System"

    # constructor
    def __init__(self):
        self.webcam = cv2.VideoCapture(0)
        
    # save image to disk
    def _save_image(self, path, image):
        filename = datetime.now().strftime('%Y%m%d_%Hh%Mm%Ss%f') + '.jpg'
        cv2.imwrite(path + filename, image)

    # obtain changes between images
    def _delta(self, t0, t1, t2):
        d1 = cv2.absdiff(t2, t1)
        d2 = cv2.absdiff(t1, t0)
        return cv2.bitwise_and(d1, d2)
    
    # wait until motion is detected 
    def detect_motion(self):

        # set motion threshold
        threshold = 170000

        # hold three b/w images at any one time
        t_minus = cv2.cvtColor(self.webcam.read()[1], cv2.COLOR_RGB2GRAY)
        t = cv2.cvtColor(self.webcam.read()[1], cv2.COLOR_RGB2GRAY)
        t_plus = cv2.cvtColor(self.webcam.read()[1], cv2.COLOR_RGB2GRAY)

        # now let's loop until we detect some motion
        while True:
  
          # obtain the changes between our three images 
          delta = self._delta(t_minus, t, t_plus)
  
          # display changes in surveillance window
          cv2.imshow(self.WINDOW_NAME, delta)
          cv2.waitKey(10)

          # obtain white pixel count i.e. where motion detected
          count = cv2.countNonZero(delta)

          # debug
          print (count)

          # if the threshold has been breached, save some snaps to disk
          # and get the hell out of function...
          if (count > threshold):

              self._save_image('WebCam/Motion/', delta)
              self._save_image('WebCam/Photograph/', self.webcam.read()[1])

              cv2.destroyAllWindows()
              return True

          # ...otherise, let's handle a new snap
          t_minus = t
          t = t_plus
          t_plus = cv2.cvtColor(self.webcam.read()[1], cv2.COLOR_RGB2GRAY)

And finally the Speech class, which handle Google’s Text To Speech and Speech To Text services:

from subprocess import Popen, PIPE, call
import urllib
 
class Speech(object):
 
    # converts speech to text
    def speech_to_text(self, filepath):
        try:
            # utilise PiAUISuite to turn speech into text
            text = Popen(['sudo', filepath], stdout=PIPE).communicate()[0]
 
            # tidy up text
            text = text.replace('"', '').strip()
 
            # debug
            print(text)

            return text
        except:
            print ("Error translating speech")
 
    # converts text to speech
    def text_to_speech(self, text):
        try:
            # truncate text as google only allows 100 chars
            text = text[:100]
 
            # encode the text
            query = urllib.quote_plus(text)
 
            # build endpoint
            endpoint = "http://translate.google.com/translate_tts?tl=en&q=" + query
 
            # debug
            print(endpoint)
 
            # get google to translate and mplayer to play
            call(["mplayer", endpoint], shell=False, stdout=PIPE, stderr=PIPE)
        except:
            print ("Error translating text")

As always, I write all my code using the free Python Tools for Visual Studio, before porting it from my PC to the Raspberry Pi.

Now for the moment of truth – can Arkwood get his mum to fix up some toast and beer? Here’s a screenshot of the program in action:

pi_remotecontrol

Arkwood’s toast request from the living room is received by the Raspberry Pi in the kitchen. The webcam then outputs some motion detection values until a threshold is reached and his mum has been detected (our black and white surveillance window shows motion being detected). We can see Google’s Text To Speech service asking his mum for toast, and Google’s Speech To Text services capturing her reply (which is ‘yes’). Then the bulb on Arkwood’s remote control is switched on – toast is on its way!

pi_remotecontrol_raswik

Unfortunately, Arkwood’s request for a nice cold beer falls on deaf ears. But he’s got his toast.

‘So, that should sort you out for now, until your ankle is better,’ I informed him.

‘Hm,’ my sordid little Belgian friend muttered, ‘Can you put a third button on this remote control, so that I can request prostitutes?’

I snatched the remote control out of his sticky hands and told him to start using his crutches.