An Indian-origin scientist, Ramesh Raskar, and his team from the Massachusetts Institute of Technology Media Lab, have modified an everyday LCD screen into a TV that can “see” the world in front of it in 3D.

“It means a viewer can control on-screen objects by waving their arms in the air without touching the screen, let alone a mouse or keyboard,” Raskar said.

He said, “This is a level of interaction that nobody’s ever been able to do before”.
Raskar along with his colleagues Matthew Hirsch and Henry Holtzman created a prototype of the screen called bi-directional or BiDi that allows users to manipulate or interact with objects on the screen in three dimensions. The model will be presented at Siggraph Asia later this week.

“It will also function as a 3D scanner. If you spin an object in front of screen, the software will stitch together a 3D image,” he added.

Raskar’s team was inspired by the way manufacturers of LCD panels, including Sharp and Planar Systems, are experimenting with adding optical sensors between a panel’s pixels so that it can act as a touch-screen interface, journal ‘New Scientist’ reported.

But such displays have poor vision, like a camera with no lens, says the researchers “They can clearly image objects that are in direct contact with the screen, but anything further away is blurred”

They set out to modify the concept to let the screen see the world in front of it more sharply.

“Placing a tiny lens slightly in front of each sensor would do that, but the layer of lenses would adversely affect the images produced by the display, thus we used a standard 20-inch screen to show how a basic feature of all LCD screens can perform the job of a lens array,” they added.

When the screen is “looking” around it, most of its pixels are shut off by the liquid crystals. But a regular grid of hundreds of pixels spread across the screen use their liquid crystals to create a tiny hole that acts as a pinhole camera lens, focusing an image of the scene in front onto a thin translucent film a few centimetres behind the LCD.

Those images are detected by a camera inside BiDi, allowing the device to know what is happening before it.

The LCD screen’s pixels must also do their usual job of presenting images to the user, though. They oscillate between their two tasks many times per second, too fast for the viewer to notice that while they are watching the screen, the screen is also watching them. “We take the normal LCD layer and put it to double duty,” Raskar said.