Last week, Microsoftreleased an appdesigned to help theblindand visually impaired “see” what is going on around them. Called Seeing AI, the app applies computer vision to describe people, text, and objects by analyzing whatever is in view of a user’s smartphone camera. Itrelays that information back to the user with a computer aide that talks through their phone’s speakers. Microsoft claims its app is capable of reading short text and documents andanalyzing products, scenes, people, and( eventually) currency, but there’s only one to actually find out.
I tested to see how well it is able to acknowledge and describe different settings and objectsand I was pleasantly surprised by the results.
Seeing AI: Reading short text
In my first exam, I asked the app to read a few paragraph from a Daily Dot article on HyperLoop One. I blew up the article on my 27 -inch monitor, grabbed an iPhone 7 Plus, pressed on the app’s” Short Text” icon, and pointed my camera at the screen.
The app immediately started reading text to me in a female robotic voice. That promising start was quickly must be accompanied by persistent exasperation. The voice on the other terminate awkwardly hop-skip large parts of text and constantly recurred itself.
Here’s the block of text I asked Determining AI to read.
Elon Musksvision for high-speed travellingin giant steel tubings just got one stair a little bit closer to reality.
Hyperloop One completed thefirst full-scale exam of its hyperloop pod in vacuum conditions, the companyannounced so early. The full-sized hyperloop vehicle got nowhere near airline hastens, topping out at 70 mphduring its run down the companystest track in Nevada.
This is what it came up with.
Elon Musk’s vision for travel. Elon Musk’s high-speed travel in giant steel. Elon Musk. Elon Musk’s eyesight for high speed travel in giant steel got one pace a little bit closer to reality. One completed test of its in vacuum circumstances. The corporation announced earlier today the airlines hastens topping out at 70.
Clearly, the app struggled to read text from a screenor it’s just really, really into Elon Musk. But I didn’t give up on the “Reading” part, figuring most of the app’s troubles came downto the poor visibility of a monitor and layout of the webpage. So I grabbed one of my favorite Haruki Murakami fictions Colorless Tsukuru Tazaki and His Years of Pilgrimage and devoted the app another spin.
Without skipping a beat, the app read two paragraphs from the book without a single correct. Sure, the lane itfluctuated its voice was a bit awkward, and its dry tone was devoid of any feeling, but the words were all there.
” Short Text” has a lot of potential. It’s extremely fast and working for you when text has a simple formatting. But its incompatibilities maintain me from recommending it.
Seeing AI 😀 ocument
The ” Document ” feature was created to provideaudible counseling to assist the visually impaired take photos of documents. It recognise when something is a document and uses audio cuesto help the user align the paper into their camera’s position. For instance, it might mention ” left margin not visible .” Once you’ve gotten it aligned, the app automatically snaps a photograph and scans the document’s text and formatting onto your phone. From there, it should read whichever sections of the text you swipe on.
It arose to me that this could be a fantastic style to overcome the limitations of the” Short Text” feature. To test that assumption, I took the same text from the article before, printed it out, and devoted it a go applying Document.
Here’s a screengrab of the document.
The app nailed it, all the way down to the punctuation.
Unfortunately, I was not able to get onto to read the text to me. Hopefully Microsoft set this feature in the future.
Seeing AI: Product
Thereare a number of apps capable of scanning barcodes and pulling up product datum, so I seemed fairly confident Insuring AI would do a good job this round. It didn’t disappoint.
The most intuitive element is how it guides consumers to product barcodes. You will start hearing a beep if a barcode comes into view of your camera. That beeping get faster and faster as you get closer to the label. It takes some getting used to, but once you’ve gotten the hang of where your camera is in relation to the item you are comprising, it works pretty well.
I grabbed a pair of Sennheiser headphones to see if it would tell me which model I was carrying. It was spot-on.
I then drew out some fiber caps to see its potential for providingimportant information about drugs and supplements. Not only did it tell me the brand and product, it also gave me the capsule count.
I’m pleased with the results.
Seeing AI: Person
A hallmark feature of the app is for it to tell users who’s around them, how those people looking, how age-old “they il be”, andhow “they il be” feeling.
The app told I was ” happy” in this photo of me smiling.
Unfortunately, it added six years old to my age. It got a little bit closer to my actual age when I frownedbut said I appeared more “neutral” than “sad.”
Here I am with a super-frown. That seemed to fix things( though definitely not my image ).
I then tried to trick the app with a pair of sunglasses, but it one-upped me with its responseand penalized me farther by saying I was3 1 years old.
Despite my bitter attitude, I’d devote Person a solid B rating.
Seeing AI: Scene
As the epithet implies, this experimental fix tells a user what’s going on around them. Just aim your camera and snap a photo of any scene and it will analyze it for you. The results of my exam ranged from on-point to wildly inaccurate.
Here I am sitting on a bench in the park. The app’s description isn’t wrong, but detailed information would have stimulated it more useful.
It did a great job with this photo of some guy’s puppy, merely forgetting the words “cute” and “beefy.”
This next description could be very useful for the visually impaired. Perhaps as camera technology advances( HoloLens ?), it will be able to tell users how far away they are from objects.
It had some trouble identifying a treadmill.
Overall the app did a solid undertaking, especially considering it was just released this week with a number of features still in beta. I could see this being a useful tool for the blind and visually impaired, but there’s still a lot of work for Microsoft to do to iron out some of itsannoyances and inconsistencies.
That being said, Insuring AI is a genuinely novel ideaone designed tohelp the blind and visually impaired insure the world by listening.
Seeing AIis available today for free on iOS. Microsoft has not announcedplans to launch the app onAndroid . em>