Road sign detection using OpenCV ORB

Tags

, , , , , , , , , , , ,

Once upon a time, I wrote some code which detects roundabout road signs in Google Street View. I created an OpenCV Haar feature-based cascade classifier which can be utilised from a Python script:

roundabouts = roundabout_cascade.detectMultiScale(
    gray, 
    scaleFactor=1.4, 
    minNeighbors=6
    )

It works quite well:

orb_streetview_minneighbors6

But what happens if we drop the minNeighbors parameter from 6 to, say, 3:

orb_streetview_minneighbors3

Oops, we are now detecting a ‘keep left’ road sign, as well as the roundabout road signs. By adjusting the minNeighbors parameter to make object detection less strict, we have inadvertently introduced a false-positive.

Why not just leave the minNeighbors at 6? you ask me. Problem is, by being so strict with this setting we may fail to detect roundabout signs in other Google Street View snaps. So let’s try a different approach – let’s set minNeighbors to 3 and then treat each detected object as a ‘candidate’ for a roundabout sign. This candidate can then be inspected, to ensure it is actually a roundabout sign.

Now, there are many OpenCV techniques we can use to inspect these candidate objects. Here I am going to use OpenCV ORB, which is one of a number of feature detection and description algorithms that we can match images on.

Let me explain the following code:

import cv2

# constants
IMAGE_SIZE = 200.0
MATCH_THRESHOLD = 3

# load haar cascade and street image
roundabout_cascade = cv2.CascadeClassifier('orb/repository/haarcascade_roundabout.xml')
street = cv2.imread('orb/repository/roundabout1.png')

# do roundabout detection on street image
gray = cv2.cvtColor(street,cv2.COLOR_RGB2GRAY)
roundabouts = roundabout_cascade.detectMultiScale(
    gray, 
    scaleFactor=1.4, 
    minNeighbors=3
    )

# initialize ORB and BFMatcher
orb = cv2.ORB()
bf = cv2.BFMatcher(cv2.NORM_HAMMING,crossCheck=True)

# find the keypoints and descriptors for roadsign image
roadsign = cv2.imread('orb/roundabout.jpg',0)
kp_r,des_r = orb.detectAndCompute(roadsign,None)

# loop through all detected objects
for (x,y,w,h) in roundabouts:

    # obtain object from street image
    obj = gray[y:y+h,x:x+w]
    ratio = IMAGE_SIZE / obj.shape[1]
    obj = cv2.resize(obj,(int(IMAGE_SIZE),int(obj.shape[0]*ratio)))

    # find the keypoints and descriptors for object
    kp_o, des_o = orb.detectAndCompute(obj,None)
    if len(kp_o) == 0 or des_o == None: continue

    # match descriptors
    matches = bf.match(des_r,des_o)
    
    # draw object on street image, if threshold met
    if(len(matches) >= MATCH_THRESHOLD):
        cv2.rectangle(street,(x,y),(x+w,y+h),(255,0,0),2)

# show objects on street image
cv2.imshow('street image', street)
cv2.waitKey(3000)
cv2.destroyAllWindows()

First up, I load the roundabout cascade and the Google Street View photograph. I use the aforementioned OpenCV detectMultiScale function to detect all roundabout objects in the street snap.

Now we get to the OpenCV ORB stuff. I create an instance of ORB and a matcher.

Next, I load an image of a roundabout road sign (for matching each detected object with). I use ORB to obtain the keypoints and descriptors of the roundabout image. Here’s the roundabout image:

roundabout

We are ready to loop through all the objects detected by our cascade…

I use the object’s coordinates to obtain its image from the street photograph. I resize the object image, to bring it inline with our roundabout image.

I use ORB to obtain the keypoints and descriptors of the object. Now we can match the descriptors of our object with our roundabout image.

All that’s left is to draw a rectangle around our detected object, on the street photograph. But the object needs to pass the match threshold for this to happen!

So you see, OpenCV ORB has determined which of our candidate objects are actually roundabout signs. If fewer than 3 matches are found between our object and the roundabout image, the object is discarded.

Time for a demo…

Our cascade detected four objects from our Google Street View photograph. Here’s the first object, which shows no matches with our roundabout image:

orb_matches_1

Our next object is a roundabout road sign, but no matches are found. Perhaps it’s because the object is not a full sign i.e. its border has been chopped off?

orb_matches_2

Luckily, our third object is of the same sign, but this time we can see its border. Matches have been found!

orb_matches_3

Finally, our fourth object, with lots of matches:

orb_matches_4

And here’s the Google Street View photograph with rectangles around the objects which passed the match threshold:

orb_matches_streetview

So there you have it – OpenCV ORB has helped us to detect road signs! Notice how ORB’s rotation invariance has matched the arrows on the detected objects, which are upside down compared to roundabout image.

Of course, there are other feature detection and description algorithms to try e.g. where scale invariance may avoid the need for image resizing. There are parameters we can tweak on some of the function calls. There are alternative techniques to evaluate our matches (than simply using a threshold). We need object detection to work well across all Google Street View photographs. But it’s been a promising start.

P.S.

If you want to get your hands on my roundabout cascade, as well as some Google Street View photographs, visit the repository: https://onedrive.live.com/redir?resid=74B6CEA107C215CA%21107

Perhaps when matching an object with the roundabout image, you’d rather the object was set into its surrounding area as thus:

orb_paddedobject

No problem. Simply add some padding to the coordinates:

IMAGE_PADDING = 100
obj = gray[(y-IMAGE_PADDING):(y+h+IMAGE_PADDING),(x-IMAGE_PADDING):(x+w+IMAGE_PADDING)]

Now, you may have noticed from the OpenCV Feature Matching documentation that there is a cv2.drawMatches function. But this be for OpenCV 3.x, and I only have 2.4.9:

from cv2 import __version__
print __version__
>>> 2.4.9

A stackoverflow post provided some options for drawing matches, if you don’t have OpenCV 3.x. I plumped for the code provided by rayryeng.

Follow

Get every new post delivered to your Inbox.

Join 69 other followers