Tags

, , , , , , , , , , , , ,

‘Why so sad, still?’ I asked Arkwood. He told me that things between the virtual girlfriend and himself had turned sour. There’s just no spice in the bedroom, he lamented. ‘Right,’ I announce, rolling up my sleeves, ‘let’s make a list this time, of all your filthy, sordid desires’.

My job would be to code up his wishes using the Python programming language, so that he is able to get steamy with her on the Raspberry Pi.

After a few cups of coffee (and a whisky or two to steady the nerves) we jotted down his carnal needs:

  1. To easily record and store real-world conversations, so that they can be replayed when Arkwood wants to sweet-talk his sweetheart
  2. To be able to present himself to her on webcam, thus initiating some intercourse (vocal, or otherwise)
  3. To give the girlfriend a name, and display a suitable photograph of her on-screen
  4. To speak to her, and for her to speak back to him

There were a few ‘additional’ requests from my scrawny Belgian friend, but I made it clear that those types of interaction would certainly have to be in a later phase of development. Who’s going to wipe down the equipment each time? I snapped.

So, without further to do, here’s how I set about turning Arkwood’s wish list into a reality…

Storage

1. To easily record and store real-world conversations, so that they can be replayed when Arkwood wants to sweet-talk his sweetheart

I created a Storage class, which has two functions: one for writing conversations to a config file, and one for reading them back out again.

class Storage(object):

    SEPARATOR = '='

    # write to config file
    def write_config(self, filename, target_key, target_value):
        try:
            with open(filename, "a") as conf_file:
                conf_file.write(target_key.strip() + self.SEPARATOR + target_value.strip() + '\n')
        except:
            print ("Error writing config")
            
    # read from config file
    def read_config(self, filename, target_key):
        try:
            with open(filename) as conf_file:
                for line in conf_file:
                    key, value = line.partition(self.SEPARATOR)[::2]

                    if (key.strip() == target_key.strip()):
                        return value.strip()
        except:
            print ("Error reading config")

Nothing too complicated here. Now we just have to use the class to record real-world conversations for later use.

from speech import Speech
from storage import Storage
from time import sleep

speech = Speech()
storage = Storage()

# loop until armageddon
while True:

    # get question
    question = speech.speech_to_text('/home/pi/PiAUISuite/VoiceCommand/speech-recog.sh')
    sleep(5)

    # get answer
    answer = speech.speech_to_text('/home/pi/PiAUISuite/VoiceCommand/speech-recog.sh')
    sleep(5)

    # store chat, if available
    if(question and answer):
        storage.write_config("chat.conf", question, answer)

Great. We’ll cover how the speech to text works later in the post – for now, note that we can speak into a microphone attached to the Raspberry Pi to record dialog in a question / answer format. The exciting thing is we can obtain this dialog from anywhere we can stick a mic. One such possibility was the TV set, permitting us to record broadcasts of Hollyoaks and BBC Parliament.

BBC Parliament:

you risk making=going insane
real world example is that=nc report should have been having his Department
does a rule that was a potato=call to ask about the way spending cuts with
pictures of Frisco sons of the fallen=pica
son seduces a spectacular tea good for=on
impacts on income=find scotchy the City 2
and then Alabama=Scotland independence
toonces in Scotland=find it difficult to access
alone=does infringe opening 4
will save on 100=pretty Appetit sings save
so it's coin=speak with science background Chris on high ground
Colisee say=epic on the challenge
Holeman and we should do this=consistency instability around
maple Bloomfield basis printed=Westminster that means to be a much richer
because I was inspired=someone who just someone
signs=what is once in whole self destruction
shooting corporation=I'm on Saturday in hand
work on the Bloodhound SSC=couple of hours at Big Bend WI Court recent
Carissa this we need to be built=people who run in Idaho universe

It really depends on what subjects Arkwood wants to converse with his virtual girlfriend on. If he is feeling highbrow then BBC Parliament; if not, Hollyoaks. And if he really wants to get dirty, then he can hook up the microphone to a sex channel, or talk into it himself whilst arousing sensual fantasies. Whatever.

Arkwood’s fantasies:

I love you=I love you too tarzan
kiss me=do you want a French kiss
touch me=place my hand where you want it
unpeel me=you are my little banana

‘Maybe I’ll take the microphone on public transport,’ my buddy said, licking his lips, ‘and record conversations between commuters’. I told him that this was probably not legal.

Webcam

2. To be able to present himself to her on webcam, thus initiating some intercourse (vocal, or otherwise)

Using OpenCV technology, we can easily detect motion in front of the webcam, and even make sure that a face has been detected. Ideally, we would recognise Arkwood’s face from some other sad loner – but for now we simply have face detection, not recognition.

Our Webcam class has two core functions, detect_motion and detect_faces:

import cv2
from datetime import datetime

class Webcam(object):

    WINDOW_NAME = "Arkwood's Surveillance System"

    # constructor
    def __init__(self):
        self.webcam = cv2.VideoCapture(0)
        
    # save image to disk
    def _save_image(self, path, image):
        filename = datetime.now().strftime('%Y%m%d_%Hh%Mm%Ss%f') + '.jpg'
        cv2.imwrite(path + filename, image)

    # obtain changes between images
    def _delta(self, t0, t1, t2):
        d1 = cv2.absdiff(t2, t1)
        d2 = cv2.absdiff(t1, t0)
        return cv2.bitwise_and(d1, d2)

    # wait until motion is detected 
    def detect_motion(self):

        # set motion threshold
        threshold = 170000

        # hold three b/w images at any one time
        t_minus = cv2.cvtColor(self.webcam.read()[1], cv2.COLOR_RGB2GRAY)
        t = cv2.cvtColor(self.webcam.read()[1], cv2.COLOR_RGB2GRAY)
        t_plus = cv2.cvtColor(self.webcam.read()[1], cv2.COLOR_RGB2GRAY)

        # now let's loop until we detect some motion
        while True:
  
          # obtain the changes between our three images 
          delta = self._delta(t_minus, t, t_plus)
  
          # display changes in surveillance window
          cv2.imshow(self.WINDOW_NAME, delta)
          cv2.waitKey(10)

          # obtain white pixel count i.e. where motion detected
          count = cv2.countNonZero(delta)

          # debug
          print (count)

          # if the threshold has been breached, save some snaps to disk
          # and get the hell out of function...
          if (count > threshold):

              self._save_image('WebCam/Motion/', delta)
              self._save_image('WebCam/Photograph/', self.webcam.read()[1])

              cv2.destroyAllWindows()
              return True

          # ...otherise, let's handle a new snap
          t_minus = t
          t = t_plus
          t_plus = cv2.cvtColor(self.webcam.read()[1], cv2.COLOR_RGB2GRAY)
          
    # detect faces in webcam
    def detect_faces(self):

        # get image from webcam
        img = self.webcam.read()[1]
        
        # do face/eye detection
        face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
        eye_cascade = cv2.CascadeClassifier('haarcascade_eye.xml')

        gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
        faces = face_cascade.detectMultiScale(gray, 1.3, 5)

        for (x,y,w,h) in faces:

            cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)

            roi_gray = gray[y:y+h, x:x+w]
            roi_color = img[y:y+h, x:x+w]
            eyes = eye_cascade.detectMultiScale(roi_gray)

            for (ex,ey,ew,eh) in eyes:
                cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)

        # save image to disk
        self._save_image('WebCam/Detection/', img)

        # show image in window
        cv2.imshow(self.WINDOW_NAME, img)
        cv2.waitKey(1000)

        # tidy and quit
        cv2.destroyAllWindows()

        if len(faces) == 0:
            return False

        return True

Here’s an example of motion detection in action, whilst I was testing the system:

pi_girlfriend2_motion

And face detection:

pi_girlfriend2_face

Fabulous!

Screen

3. To give the girlfriend a name, and display a suitable photograph of her on-screen

The virtual girlfriend can be granted a moniker simply by speaking into the microphone. The fun bit comes with displaying a picture of her.

The Screen class has a display_images function which retrieves some image search results from Google, based on the girlfriend’s name. We then pick apart the results using the Beautiful Soup Python library, and display the required number of images in seperate web browser windows.

import requests
from BeautifulSoup import BeautifulSoup
import webbrowser

class Screen(object):

    # displays google images
    def display_images(self, search_term, amount):
        try:
            # check params
            if (amount < 1):
                return

            #build url
            url = "https://www.google.co.uk/search?q=image+{}&tbm=isch".format(search_term)
             
            #get webpage
            request = requests.get(url)
            soup = BeautifulSoup(request.text)    

            # get images
            images = []
            for img in soup.findAll('img'):
                images.append(img['src'])

            # ensure enough images
            if amount > len(images):
                amount = len(images)

            # display each image in a browser
            for i in range(0, amount):
                webbrowser.open(images[i], new=0 )
        except:
            print ("Error displaying images")

Arkwood likes the name Sandra, and here’s what he got:

pi_screen_images

Speech

4. To speak to her, and for her to speak back to him

Some of the previous sections have already dealt with using the microphone to record speech to text. But we also need to spit back out text to speech, so that the girlfriend can have her say. For both these requirements we use Google services.

My Speech class has two functions:

from subprocess import Popen, PIPE, call
import urllib

class Speech(object):

    # converts speech to text
    def speech_to_text(self, filepath):
        try:
            # utilise PiAUISuite to turn speech into text
            text = Popen(['sudo', filepath], stdout=PIPE).communicate()[0]

            # tidy up text
            text = text.replace('"', '').strip()

            # debug
            print(text)

            return text
        except:
            print ("Error translating speech")

    # converts text to speech
    def text_to_speech(self, text):
        try:
            # truncate text as google only allows 100 chars
            text = text[:100]

            # encode the text
            query = urllib.quote_plus(text)

            # build endpoint
            endpoint = "http://translate.google.com/translate_tts?tl=en&q=" + query

            # debug
            print(endpoint)

            # get google to translate and mplayer to play
            call(["mplayer", endpoint], shell=False, stdout=PIPE, stderr=PIPE)
        except:
            print ("Error translating text")

Now everything is ready for Arkwood to whisper sweet nothings to his lass, and for her to reciprocate. Roll on our main program…

Bringing it all together

So, to recap, we wanted:

  • Storage – for accumulating chat between Arkwood and his darling
  • Webcam – so Arkwood can present himself
  • Screen – for a photograph of his beloved to be displayed
  • Speech – allowing two-way conversation

And we’ve got it! Now we just need a way of orchestrating these features. May I present to you the Virtual Girlfriend (Mark II):

from webcam import Webcam
from screen import Screen
from speech import Speech
from storage import Storage

webcam = Webcam()
screen = Screen()
speech = Speech()
storage = Storage()

# wait until motion detected
webcam.detect_motion()
     
# if face detected
if (webcam.detect_faces()):

    print ("...")

    # give girlfriend a name
    girlfriend_name = Speech().speech_to_text('/home/pi/PiAUISuite/VoiceCommand/speech-recog.sh')
        
    # show girlfriend on screen
    screen.display_images(girlfriend_name, 1)

    # loop forever in romance
    while True:

        # speak to girlfriend
        my_jabber = Speech().speech_to_text('/home/pi/PiAUISuite/VoiceCommand/speech-recog.sh')
         
        # get girlfriend' response
        her_babble = storage.read_config("chat.conf", my_jabber)
         
        # now girlfriend speaks to Arkwood
        if her_babble:
            Speech().text_to_speech(her_babble)
        else:
            Speech().text_to_speech("What you say baby")

So, the Webcam class kicks in first, ensuring that not only is motion detected but a face is clocked too. If all is well, then Arkwood is prompted for a name for his valentine, and a photograph of her is displayed on the screen. Once done, we simply loop the program, allowing him to talk to her, and – if we find a match in the config file – allowing her to talk to him.

pi_desktop_girlfriend2

Oh what joy! My tiny pervert friend will never be unloved ever again. It’s true what they say, all of life’s yearnings can be solved with a computer.

And Arkwood is already badgering me for some enhancements:

  • Can we load different config files, to suit my mood?
  • Can we use a thermometer attached to the RasWIK wireless inventors kit, so that the program can tell if my breath is getting steamy?
  • Can we get the face recognition working, so that my girlfriend doesn’t run off with another man?

Jesus. Okay, pour me another Irish coffee. Time to get cracking.

Ciao!

p.s. if you want more detail on these features, check out the following:
BBC fibs on Raspberry Pi (storing conversations)
Face detection on Raspberry Pi (motion and face detection)
Simple slide show with Raspberry Pi (displaying photos)
Virtual girlfriend on Raspberry Pi (bidirectional chat)

Advertisements