Arkwood has twisted his ankle, and as such is stretched out at home on the sofa. ‘I need a remote control,’ he told me, ‘so that I can request toast and beer from my mum in the kitchen.’ I told him that I could write some Python code on the Raspberry Pi to handle his food orders. But first off, let’s put together the actual remote control…
I will use the RasWIK Wireless Inventors Kit as the remote control. It comprises of an Arduino board, with jump wires to a small breadboard which will house the buttons and bulb of our remote control. It can be placed up to 100m away from our Raspberry Pi, communicating with a radio component slotted onto the Pi. Here’s a photo of the remote control set up with two buttons for requesting beer and toast, plus a bulb which will light up to confirm that Arkwood’s mother has received his order:
Great. Now we have the remote control for Arkwood to use in the living room, all we need is to write a program on the Raspberry Pi in the kitchen (so that his mum can receive his orders):
from raswik import Raswik from webcam import Webcam from speech import Speech from time import sleep raswik = Raswik() webcam = Webcam() speech = Speech() #loop until hole in ozone while True: toast = False beer = False # detect button press on remote control if(raswik.is_button_pressed('a--D10READ--')): print("Toast button pressed") toast = True elif(raswik.is_button_pressed('a--D12READ--')): print("Beer button pressed") beer = True else: continue # wait until motion detected in kitchen webcam.detect_motion() # ask mum for toast or beer if(toast): speech.text_to_speech("I want toast with lashings of butter") elif(beer): speech.text_to_speech("I want beer and make it cold") # get mum's reply reply = speech.speech_to_text('/home/pi/PiAUISuite/VoiceCommand/speech-recog.sh') # send success to remote control if(reply == "yes"): if(raswik.switch_bulb('a--D05HIGH--')): print("Success bulb switched on") sleep(10) raswik.switch_bulb('a--D05LOW---')
Let’s step through this code. Once in a loop, we first check whether Arkwood has pressed either of the RasWIK buttons on the remote control in the living room. If not, we just keep looping until he has. Once a toast or beer order is received, we launch our webcam and wait until we detect his mum moving about in the kitchen. When we spot her, we use Google’s Text To Speech service to announce through speakers attached to the Raspberry Pi that we want either toast or beer, and then use Google’s Speech To Text service to capture through a microphone her reply. If her reply is ‘yes’ then we send a signal from the Pi in the kitchen back to the RasWIK remote control in the living room, lighting up a green bulb to inform Arkwood that his order is on its way. If his mum says anything else (or perhaps says nothing at all) then the bulb will not light up and Arkwood will either have to resend his request or take to his crutches.
Here’s the Raswik class I have written, to take care of the wireless communication between the RasWIK remote control and the Raspberry Pi:
import serial from time import sleep # class to handle communication to # raswik wireless inventors kit class Raswik(object): # com port and speed PORT = "/dev/ttyAMA0" BAUD = 9600 #check if a button has been pressed def is_button_pressed(self, pin): #check pin param if(pin == ""): return False reply = "" try: # establish a serial connection ser = serial.Serial(port= self.PORT, baudrate= self.BAUD) # wait a mo... sleep(0.2) # write to pin e.g. a--D10READ-- ser.write(pin) # wait some more... sleep(0.2) # now read from pin reply = ser.read(12) # close serial connection ser.close() except: print ("Error detecting raswik button press") return False # grab the bit of the reply we care about message = reply[6:].strip('-') # return whether button is currently pressed down if(message == "LOW"): return True else: return False # switch a bulb on or off def switch_bulb(self, pin): #check pin param if(pin == ""): return False try: # establish a serial connection ser = serial.Serial(port= self.PORT, baudrate= self.BAUD) # wait a mo... sleep(0.2) # write to pin e.g. a--D05HIGH-- or a--D05LOW--- ser.write(pin) # wait some more... sleep(0.2) # close serial connection ser.close() except: print ("Error detecting raswik bulb") return False return True
Here’s the Webcam class, which detects motion:
import cv2 from datetime import datetime class Webcam(object): WINDOW_NAME = "Arkwood's Surveillance System" # constructor def __init__(self): self.webcam = cv2.VideoCapture(0) # save image to disk def _save_image(self, path, image): filename = datetime.now().strftime('%Y%m%d_%Hh%Mm%Ss%f') + '.jpg' cv2.imwrite(path + filename, image) # obtain changes between images def _delta(self, t0, t1, t2): d1 = cv2.absdiff(t2, t1) d2 = cv2.absdiff(t1, t0) return cv2.bitwise_and(d1, d2) # wait until motion is detected def detect_motion(self): # set motion threshold threshold = 170000 # hold three b/w images at any one time t_minus = cv2.cvtColor(self.webcam.read(), cv2.COLOR_RGB2GRAY) t = cv2.cvtColor(self.webcam.read(), cv2.COLOR_RGB2GRAY) t_plus = cv2.cvtColor(self.webcam.read(), cv2.COLOR_RGB2GRAY) # now let's loop until we detect some motion while True: # obtain the changes between our three images delta = self._delta(t_minus, t, t_plus) # display changes in surveillance window cv2.imshow(self.WINDOW_NAME, delta) cv2.waitKey(10) # obtain white pixel count i.e. where motion detected count = cv2.countNonZero(delta) # debug print (count) # if the threshold has been breached, save some snaps to disk # and get the hell out of function... if (count > threshold): self._save_image('WebCam/Motion/', delta) self._save_image('WebCam/Photograph/', self.webcam.read()) cv2.destroyAllWindows() return True # ...otherise, let's handle a new snap t_minus = t t = t_plus t_plus = cv2.cvtColor(self.webcam.read(), cv2.COLOR_RGB2GRAY)
And finally the Speech class, which handle Google’s Text To Speech and Speech To Text services:
from subprocess import Popen, PIPE, call import urllib class Speech(object): # converts speech to text def speech_to_text(self, filepath): try: # utilise PiAUISuite to turn speech into text text = Popen(['sudo', filepath], stdout=PIPE).communicate() # tidy up text text = text.replace('"', '').strip() # debug print(text) return text except: print ("Error translating speech") # converts text to speech def text_to_speech(self, text): try: # truncate text as google only allows 100 chars text = text[:100] # encode the text query = urllib.quote_plus(text) # build endpoint endpoint = "http://translate.google.com/translate_tts?tl=en&q=" + query # debug print(endpoint) # get google to translate and mplayer to play call(["mplayer", endpoint], shell=False, stdout=PIPE, stderr=PIPE) except: print ("Error translating text")
As always, I write all my code using the free Python Tools for Visual Studio, before porting it from my PC to the Raspberry Pi.
Now for the moment of truth – can Arkwood get his mum to fix up some toast and beer? Here’s a screenshot of the program in action:
Arkwood’s toast request from the living room is received by the Raspberry Pi in the kitchen. The webcam then outputs some motion detection values until a threshold is reached and his mum has been detected (our black and white surveillance window shows motion being detected). We can see Google’s Text To Speech service asking his mum for toast, and Google’s Speech To Text services capturing her reply (which is ‘yes’). Then the bulb on Arkwood’s remote control is switched on – toast is on its way!
Unfortunately, Arkwood’s request for a nice cold beer falls on deaf ears. But he’s got his toast.
‘So, that should sort you out for now, until your ankle is better,’ I informed him.
‘Hm,’ my sordid little Belgian friend muttered, ‘Can you put a third button on this remote control, so that I can request prostitutes?’
I snatched the remote control out of his sticky hands and told him to start using his crutches.