Let's imagine that you can't see to find out what's in that tin you just took out of the cupboard. This isn't a hypothetical scenario if you're visually impaired. Rice pudding, baked beans, or dog food? Doesn't bear thinking about, does it? Later on in this post I'll offer a simple solution for Android devices, but first here's a bit of background...

I have an interest in assistive technology, and for the last few years have been attending Sight Village to geek out on the latest gadgets for blind and partially sighted people. Historically there has been a running theme of hugely expensive software and/or hardware packages to do things like screen magnification and screen reading. This year I felt that things were substantially different - the buzz was about apps for iPhone and Android, and it was telling that there were lots of iPads in evidence. I think Apple have a particular mindshare here because of their Voiceover extensions to iOS:

My experience of screen readers and magnifiers under older Symbian and Windows Mobile devices has been pretty dire - memory and CPU hogging, and prone to crashing the device due to being only tangentially integrated with the operating system. To add insult to injury these packages have tended to be extremely expensive due to the limited size of their market. The move with Android and iOS to support accessibility from the ground up is very encouraging. Let's see what sorts of apps it makes possible...

Here's a very powerful example. SayText is free OCR and text-to-speech software for the iPhone:

But let's head back towards baked beans and dog food...  At Sight Village this year there were several systems for labelling things and then using a reader to identify them. The RNIB's PenFriend is probably the most impressive of these. You might expect that this uses RFID tags, but it's actually based on optical recognition. This makes the labels effectively disposable as the unit cost is very low. Here's John Godber from the RNIB explaining how this system works:

The PenFriend was pretty impressive to play with in real life, but I found myself thinking that many of the items you would be interested in (e.g. CDs/DVDs and food packaging) already have an optical tag on them - the humble barcode (UPC). This took me back to my first adventures with barcode scanning on mobile phones. Google Googles for Android is probably the most impressive app in this category, as it will scan bar codes, carry out on-the-fly OCR of text, translate text into other languages, and carry out a Google search. Here's a video of Hartmut Neven from Google demoing recent enhancements:

I found myself thinking that a paired down version of this could be very handy for a visually impaired Android user. Enter the Scripting Layer for Android (SL4A), which exposes the underlying Android APIs to a range of scripting languages. This makes the barrier of entry very low indeed for doing interesting things with your device. Now you can write trivial Python, Perl or Ruby scripts that use geo-location, accelerometer, compass, speech synthesis, speech recognition, scan barcodes with the camera, and interact with the Internet. Here's a video demoing some simple Python scripting on the phone itself:



The SL4A community has helpfully put together a large number of tutorials, including this nice example of barcode scanning for books from Google's Matt Cutts:


import android
droid = android.Android()
code = droid.scanBarcode()
isbn = int(code['result']['SCAN_RESULT'])
url = “http://books.google.com?q=%d” % isbn
droid.startActivity(‘android.intent.action.VIEW’, url)

This six line script is all it takes to start up the Android barcode scanner, look up the UPC code and then carry out a Google Books search in the Android web browser. Unfortunately if you turn on the Android TalkBack screen reader, it insists on reading out everything it comes across, including the URL of the Google site being visited.

So, here goes with talking barcodes!

This script is a quick and dirty hack to scan a barcode, search the Google Products database for the barcode, pull out some interesting attributes (title, price, description etc) and speak them.

import android
import urllib2
import re

from htmlentitydefs import name2codepoint
name2codepoint['#39'] = 39

def unescape(s):
  return re.sub('&(%s);' % '|'.join(name2codepoint),
    lambda m: unichr(name2codepoint[m.group(1)]), s)

droid = android.Android()
droid.ttsSpeak("Ready to scan bar code")

code = droid.scanBarcode()
barcode = int(code.result['extras']['SCAN_RESULT'])
print "Barcode: ", barcode

url = "http://www.google.com/products?q=%d" % barcode
handler = urllib2.urlopen(url)
response = handler.read()
handler.close()

# clunky code to pull out the interesting bits - was hoping to use xml.dom.minidom instead
rtitle = re.compile(r'<h3 class="result-title">.*</h3>', re.M|re.S).search(response)
rattrs = re.compile(r'<p class="result-attributes">([^<]+)</p>', re.M|re.S).search(response)
rdescr = re.compile(r'<p class="result-desc">([^<]+)</p>', re.M|re.S).search(response)
rprice = re.compile(r'<span class="main-price">([^<]+)</span>', re.M|re.S).search(response)

output = ""
if rtitle is not None:
  output = re.sub(r'.* >(.*)</a></h3>', r'\1', rtitle.group(0))
if rattrs is not None:
  output += re.compile(r'<p class="result-attributes">(.*)</p>', re.M|re.S).sub(r'\1', rattrs.group(0))
if rdescr is not None:
  output += re.compile(r'<p class="result-desc">(.*)</p>', re.M|re.S).sub(r'\1', rdescr.group(0))
if rprice is not None:
  output += re.compile(r'<span class="main-price">(.*)</span>', re.M|re.S).sub(r'\1', rprice.group(0))

print unescape(output)
droid.ttsSpeak(unescape(output))

With a little more finessing I think it should be possible to replace the crufty regular expressions with callouts to the Python DOM library. I should mention that the HTML entity decoding above is courtesy of dluce's posting to Stack Overflow. A Python guru could probably replace most of this with a couple of well chosen one liners.

Ironically, whilst the Google Products database search did just fine with a bunch of books and CDs, it didn't match my test case tin of baked beans. I think a useful next step would be to link the script up with the barcode database provided by upcdata.info, as a database of last resort. This loses the immediate potential for comparison shopping that we get through the Google database, though. I like to picture a blind person at the supermarket being asked if they need any help - "Thanks but I'm OK. Think I'll get this from the local shop down the road, my phone says it's cheaper there" :-)

Now it's over to you to think of some other applications for SL4A, like this nice example of onboard rocket telemetry gathering...