Tags

, , , , , , , , , , , , ,

Arkwood was in tears.

‘I have to do a slideshow presentation for a new job. But I can’t do it! I stttuuuttter and shake and fall to pieces.’

He grabbed a pillow from the sofa and drenched it in salty water.

‘Don’t worry.’ I said, snatching the sodden cushion, ‘My robots will help you master the art of the slideshow!’

I updated SaltwashAR – the Python Augmented Reality application – so that the robot can talk through a series of slides. Here’s the result:

Hurray! Rocky Robot provides a clear and concise presentation about the streets of Hong Kong and the planets in space.

But how the hell does the new Slideshow feature work? Here’s the Python code:

from features.base import Feature, Speaking
import numpy as np
import os, glob
import cv2
from threading import Thread
from time import sleep

class Slideshow(Feature, Speaking):
 
    FEATURE_PATH = 'scripts/features/slideshow/'
    ROOT_PATH = '../../../'
    SLIDE_OFFSET = 50

    def __init__(self, text_to_speech):
        Feature.__init__(self)
        Speaking.__init__(self, text_to_speech)
        self.background_image = np.array([])
        self.slides = []
        self.blurbs = []
        self.current_item = 0
        self.current_slide = np.array([])
        self.blurb_thread = None
        self._get_slides_and_blurbs()

    def stop(self):
        Feature.stop(self)
        self.background_image = np.array([])

    # load slideshow
    def _get_slides_and_blurbs(self):
        os.chdir(self.FEATURE_PATH)

        for slide in glob.glob('*.jpg'):
            self.slides.append(slide)

        for blurb in glob.glob('*.txt'):
            self.blurbs.append(blurb)

        os.chdir(self.ROOT_PATH) 

        if len(self.slides) != len(self.blurbs):
            self.slides = []
            self.blurbs = []
            print "Unable to load slideshow as number of slides not equal to number of blurbs"
        elif not self.slides:
            print "Unable to load slideshow as no slides or blurbs found"

    # slideshow thread
    def _thread(self, args):
        image = args

        # check slides loaded
        if not self.slides: return

        # reset current item, if at end of slides
        if self.current_item >= len(self.slides): 
            self.current_item = 0

        # get next slide and blurb, if at end of current blurb
        if not self.blurb_thread or not self.blurb_thread.is_alive():
            self.current_slide = cv2.imread('{}{}'.format(self.FEATURE_PATH, self.slides[self.current_item]))

            with open('{}{}'.format(self.FEATURE_PATH, self.blurbs[self.current_item]), 'r') as blurb_file:
                blurb = blurb_file.readline()
            
            self.blurb_thread = Thread(target=self._blurb_thread, args=(blurb,))
            self.blurb_thread.start()           

        # update current background image with slide
        slide_offset_and_height = self.SLIDE_OFFSET + self.current_slide.shape[0]
        slide_offset_and_width = self.SLIDE_OFFSET + self.current_slide.shape[1]            

        if slide_offset_and_height <= image.shape[0] and slide_offset_and_width <= image.shape[1]:
            image[self.SLIDE_OFFSET:slide_offset_and_height, self.SLIDE_OFFSET:slide_offset_and_width] = self.current_slide
            self.background_image = image
        else:
            print "Unable to use slide as size larger than background image"
            self.background_image = np.array([])

    # blurb thread
    def _blurb_thread(self, blurb):
        if blurb: 
            self._text_to_speech(blurb)
        
        self.current_item += 1
        sleep(4)

The Slideshow feature inherits from the Feature base class, which provides threading (all features run in threads so as not to block the main application process from rendering to screen). We also inherit from the Speaking base class, to let the robot’s mouth move when he speaks.

The class __init__ method is passed a Text To Speech parameter, so that the robot can talk. We have class variables to store and track our slides and blurbs, as well as the background image of our application.

We override the Feature base class stop method, so the background image can be cleared when we are no longer interacting with the robot.

The _get_slides_and_blurbs private method is called from the __init__ method. It navigates to the feature’s folder, so as to load all the filenames of our slides and blurbs.

Next up is the _thread method, where all the shit happens (as a coffee merchant would say)…

Notice that we pass the method an image argument. This is the original background image which we can draw our slide upon.

We do a couple of preliminary checks. We make sure that slides have been loaded. If we have reached the end of our slides then we reset to the first slide.

Okay, so next we need to check whether the blurb thread is alive – which basically means that the robot is currently talking. If the robot has stopped talking, then we are ready to retrieve the next slide and blurb from disk, and kick off our blurb thread again.

For example, here’s the first slide we retrieve from disk:

SaltwashAR_Slideshow_HongKong_001

And here’s the first piece of blurb we retrieve:

Here we are in Hong Kong

The blurb is a simple sentence that we pass to the blurb thread, so that the robot can speak it out through the computer speakers.

Finally, we add our slide to the original background image (assuming it is small enough to fit), ready for it to be rendered to screen.

The _blurb_thread method takes care of the robot’s speech, allowing for a 4 second pause after each sentence is delivered. But why is a new thread being created just for the robot’s speech? Well, if we were to let the robot speak in the main _thread method then the background image would freeze until the robot had finished its spiel. That would be rubbish. So instead, the main _thread method continues to render background images whilst the robot speaks.

And that’s about it. The slides and blurb stay in sync, so the robot is always talking about the correct slide.

And if we want to change our slideshow to something else, all we need do is drop new slide images and blurb text files into the feature’s folder. Just be sure to stick to the filename convension of e.g. 001.jpg 002.jpg 003.jpg for slides and 001.txt 002.txt 003.txt for blurbs, as using numbers for filenames ensures they are dished up in the correct order.

I told Arkwood the good news.

‘The robots will now teach you how to deliver the perfect slideshow. You will get that dream job after all! What job is it anyway?’

‘It’s the job of deep fat frier,’ he replied.

Of course, he wants to win the heart of Daphne – the plump spotty girl who works down the chippy. Goodness, I had no idea a presentation on potatoes and oil was needed for such casual employment.

Ciao!

Please check out the SaltwashAR Wiki for details on how to install and help develop the SaltwashAR Python Augmented Reality application.

Advertisements