Tags

, , , , , , , , , , , , ,

Arkwood is a keen botanist. Oh yes. He is particularly interested in the Iris flower.

‘Can you help me classify the Iris flowers I find?’ he asked of me.

I told him that I would be glad to help.

I will add a new Iris Classifier feature to SaltwashAR, the Python Augmented Reality application, so that the robots can tell Arkwood which species of Iris he has found (based on measurements he provides about the plant).

Let’s take a look at the Python code first, then we’ll pour ourselves a nice cup of tea and watch a video of the feature in action.

from features.base import Feature, Speaking
import numpy as np
from neuralnetwork import *
import cv2
from time import sleep

class IrisClassifier(Feature, Speaking):

    IRIS_CLASSES = ['setosa','versicolor','virginica']
    FEATURE_PATH = 'scripts/features/irisclassifier/'
    SLIDE_OFFSET = 50

    def __init__(self, text_to_speech, speech_to_text):
        Feature.__init__(self)
        Speaking.__init__(self, text_to_speech)
        self.speech_to_text = speech_to_text
        self.neural_network = None
        self.background_image = np.array([])
        self.iris_slide = np.array([])

    # start thread
    def start(self, args=None):
        image = args
        Feature.start(self, args)
 
        # if slide, add to background image
        if self.iris_slide.size > 0:
            slide_offset_and_height = self.SLIDE_OFFSET + self.iris_slide.shape[0]
            slide_offset_and_width = self.SLIDE_OFFSET + self.iris_slide.shape[1] 
        
            image[self.SLIDE_OFFSET:slide_offset_and_height, self.SLIDE_OFFSET:slide_offset_and_width] = self.iris_slide
            self.background_image = image
        else:
            self.background_image = np.array([])

    # stop thread
    def stop(self):
        Feature.stop(self)
        self.background_image = np.array([])

The IrisClassifier class inherits from the Feature base class, which provides threading (all features run in threads so as not to block the main application process from rendering to screen). We also inherit from the Speaking base class, to let the robot’s mouth move when she speaks.

There are some constants defined, to handle Iris classification – more on that later.

The class __init__ method is passed Text To Speech and Speech To Text parameters, so that the robot can have a conversation with Arkwood. We have a class variable for our PyBrain Neural Network, which will be used to classify the Iris data that Arkwood provides. And we also have a background image and Iris slide, which will be used to show a picture of the matched Iris.

We have overridden the thread’s start method. Reason being, if we have a picture of the matched Iris to show then we can embed it in each background image served from the webcam.

We have also overridden the thread’s stop method, so we can clear the background image when the robot is no longer in front of the webcam.

# run thread
def _thread(self, args):

    # reqest iris data from user
    sepal_length = self._request_iris_measurement("What is the sepal length in centimetres?")
    sepal_width =  self._request_iris_measurement("What is the sepal width in centimetres?")
    petal_length =  self._request_iris_measurement("What is the petal length in centimetres?")
    petal_width =  self._request_iris_measurement("What is the petal width in centimetres?")

    # check iris data okay
    if not sepal_length or not sepal_width or not petal_length or not petal_width:
        print "The data you have provided is not valid"
        return

Okay, now we are at the _thread method, where all the shit happens (as a village gardener would say).

First, we request the Iris data from Arkwood. In particular, we ask for:

  • sepal length
  • sepal width
  • petal length
  • petal width

The measurements are all in centimetres, as floating point numbers. For example, the Iris setosa species measurements could be:

  • sepal length: 5.1
  • sepal width: 3.5
  • petal length: 1.4
  • petal width: 0.2

Here’s the private method _request_iris_measurement:

# request iris data
def _request_iris_measurement(self, request_message):

    self._text_to_speech(request_message)
        
    iris_measurement = self.speech_to_text.convert()

    try:
        return float(iris_measurement)
    except:
        return None

We use Text To Speech so that the robot can ask Arkwood through the computer speakers for the Iris measurement. And we use Speech To Text so that Arkwood can provide the Iris measurement through the computer microphone. If the provided measurement does not convert to a floating point number then it is assigned a value of None.

If any of the provided measurements have a value of None then we bail out of the thread. We can’t provide classification on dodgy data.

# ensure neural network has been built
if not self.neural_network:
    self.neural_network = build_network()

# check whether to stop thread
if self.is_stop: return

Right, time to build the PyBrain Neural Network (if it has not already been constructed in a previous use of the thread). I’ll dive into the specifics of putting together the network later in this post – but for now, simply know that we have its classification powers at our disposal.

Note that we check whether to stop the thread, in the event of our robot no longer being in front of the webcam (and therefore not able to tell Arkwood the classification of his Iris data).

# classify iris data
iris_index = classify_data(self.neural_network, [sepal_length, sepal_width, petal_length, petal_width])

# update background image with iris slide
self.iris_slide = cv2.imread('{}{}.jpg'.format(self.FEATURE_PATH, self.IRIS_CLASSES[iris_index]))

# inform user of iris classification
self._text_to_speech("Your iris data has been classified as {}".format(self.IRIS_CLASSES[iris_index]))
sleep(6)

# clear iris slide
self.iris_slide = np.array([])

With the PyBrain Neural Network built, we can now use it to classify our Iris data. We simply pass the network to our classify_data function, along with the Iris measurements, and get back the Iris classification index. The specifics of the function is later in this post.

Note there are three species of Iris that we will be classifying:

  • Iris setosa
  • Iris versicolor
  • Iris virginica

With the Iris classification index at hand, we can use our IRIS_CLASSES constant to fetch a picture of the correct Iris species from disk and add it to our iris_slide class variable.

Finally, the robot tells Arkwood the species of Iris that his measurements have been classified to. The robot waits six seconds before clearing the slide.

Phew! That’s a fair chunk of code. Let’s pour ourselves that cup of tea and watch the video of Arkwood and the robot in action:

Hurray! Arkwood has provided measurements for four Iris flowers he has found and the robot has correctly classified them: Iris setosa, Iris versicolor, Iris virginica, Iris setosa.

Please check out the SaltwashAR Wiki for details on how to install and help develop the SaltwashAR Python Augmented Reality application.

Now, before I go, here’s the Python code I promised to build a neural network for classifying data:

from sklearn import datasets
from pybrain.utilities import percentError
from pybrain.tools.shortcuts import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.structure.modules import SoftmaxLayer
from pybrain.datasets.classification import ClassificationDataSet

# network constants
INPUT = 4
HIDDEN = 3
OUTPUT = 1
CLASSES = 3

# get classification dataset
def _get_classification_dataset():
    return ClassificationDataSet(INPUT,OUTPUT,nb_classes=CLASSES)

# convert a supervised dataset to a classification dataset
def _convert_supervised_to_classification(supervised_dataset):
    classification_dataset = _get_classification_dataset()
    
    for n in xrange(0, supervised_dataset.getLength()):
        classification_dataset.addSample(supervised_dataset.getSample(n)[0], supervised_dataset.getSample(n)[1])

    return classification_dataset

# split dataset with proportion
def _split_with_proportion(dataset, proportion):
    x,y = dataset.splitWithProportion(proportion)

    x = _convert_supervised_to_classification(x)
    y = _convert_supervised_to_classification(y)

    return x,y

We start with the import statements for PyBrain Neural Network, along with sklearn (which provides the Iris data set to train and test the network).

Next are some private functions, which we will discuss as we go through the public functions.

# build a neural network
def build_network():

    # get iris data
    iris = datasets.load_iris()
    d,t = iris.data, iris.target

    # build dataset
    ds = _get_classification_dataset()
    for i in range(len(d)):
        ds.addSample(d[i],t[i])

    print "Dataset input: {}".format(ds['input'])
    print "Dataset output: {}".format(ds['target'])
    print "Dataset input length: {}".format(len(ds['input']))
    print "Dataset output length: {}".format(len(ds['target']))
    print "Dataset length: {}".format(len(ds))
    print "Dataset input|output dimensions are {}|{}".format(ds.indim, ds.outdim)

    # split dataset
    train_data,test_data = _split_with_proportion(ds, 0.70)
    
    print "Train Data length: {}".format(len(train_data))
    print "Test Data length: {}".format(len(test_data))

    # encode with one output neuron per class
    train_data._convertToOneOfMany()
    test_data._convertToOneOfMany()

    print "Train Data input|output dimensions are {}|{}".format(train_data.indim, train_data.outdim)
    print "Test Data input|output dimensions are {}|{}".format(test_data.indim, test_data.outdim)

    # build network
    network = buildNetwork(INPUT,HIDDEN,CLASSES,outclass=SoftmaxLayer)

    # train network
    trainer = BackpropTrainer(network,dataset=train_data,momentum=0.1,verbose=True,weightdecay=0.01)
    trainer.trainOnDataset(train_data, 500)

    print "Total epochs: {}".format(trainer.totalepochs)

    # test network
    output = network.activateOnDataset(test_data).argmax(axis=1)
    
    print "Percent error: {}".format(percentError(output, test_data['class']))

    # return network
    return network

Okay, here’s the code for building the PyBrain Neural Network (along with lots of print statements, so we can easily tell how our network is constructed and how it performs).

We fetch our Iris data set from the aforementioned sklearn package, and split it into rows of data (the network input values) and targets (the network output values). Wikipedia provides information on the Iris flower data set.

We fetch a ClassificationDataSet with the desired input (4) and output (1) settings, along with the Iris classes (3 – for each species: Iris setosa, Iris versicolor, Iris virginica).

The ClassificationDataSet is filled with all our rows of data and targets.

Next, we need to split our data set into training (70%) and testing (30%) data using the _split_with_proportion private method. But why do we need a private method – why not simply call the PyBrain splitWithProportion method? The answer is that PyBrain splitWithProportion currently has a bug which means that it returns a SupervisedDataSet rather than a ClassificationDataSet. So we use a private method instead, which takes care of yielding a ClassificationDataSet.

With our training and testing data in place, we call the PyBrain _convertToOneOfMany function so that network output is provided for the three species of Iris.

Now we can get to the real meat, building and training our network.

The network layers are built using our INPUT (4) HIDDEN (3) and CLASSES (3) constants.

We use a BackpropTrainer, to ply our network with the training data over 500 epochs.

But how well has our network been trained? That’s where the testing data comes in. We feed the testing data into the network and inspect the output for percentage errors. If we have trained it well then the percentage error will be 0 (but we may settle for something slightly higher).

Finally, we return the trained and tested network for use in our feature’s thread.

Here’s the classify_data function, which is used to classify the data provided by Arkwood:

# classify data against neural network
def classify_data(network, data):
    output = network.activate(data).argmax(axis=0)

    print "Data classified as: {}".format(output)

    return output

We simply feed the data into our neural network and collect the output. The output will be for one of our Iris species: Iris setosa (0), Iris versicolor (1), Iris virginica (2).

That’s it! The PyBrain Neural Network can now be used to classify the measurements of an Iris flower into one of three species.

Note for the first two classifications in the video, I provided Iris measurements as I found them in the data set – but for the third classification I changed a data set value of 7.7 to 7.6, and for the fourth classification I changed data set values of 3.5 to 3.4 and 0.6 to 0.7. The network was still able to classify the Iris correctly, regardless of these changes (just as any robust neural network ought to!)

The tutorial Machine learning of Iris data using Pybrain neural network was a great help in putting together the PyBrain Neural Network code.