Computers and cameras: Seeing more than humans can

It opens up a technology that offers a computerised supervisor that is always watching

Some officers played the role of prisoners, acting like gang members and stirring up trouble, including a mock riot. The latest in prison gear got a workout — body armour, shields, riot helmets, smoke bombs, gas masks. And, at this particular drill, computers that could see the action.

Perched above the prison yard, five cameras tracked the play-acting prisoners, and artificial-intelligence software analysed the images to recognise faces, gestures and patterns of group behaviour. When two groups of inmates moved towards each other, the experimental computer system sent an alert — a text message — to a corrections officer that warned of a potential incident and gave the location.

The computers cannot do anything more than officers who constantly watch surveillance monitors under ideal conditions. But in practice, officers are often distracted. When shifts change, an observation that is worth passing along may be forgotten. But machines do not blink or forget. They are tireless assistants.

The enthusiasm for such systems extends well beyond the nation’s prisons. High-resolution, low-cost cameras are proliferating, found in everyday products like smart phones and laptop computers. The cost of storing images is dropping, and new software algorithms for mining, matching and scrutinising the flood of visual data are progressing swiftly.

A computer-vision system can watch a hospital room and remind doctors and nurses to wash their hands, or warn of restless patients who are in danger of falling out of bed. It can, through a computer-equipped mirror, read a man’s face to detect his heart rate and other vital signs. It can analyse a woman’s expressions as she watches a movie trailer or shops online, and help marketers tailor their offerings accordingly. Computer vision can also be used at shopping malls, schoolyards, subway platforms, office complexes and stadiums.

All of which could be helpful — or alarming.

“Machines will definitely be able to observe us and understand us better,” said Hartmut Neven, a computer scientist and vision expert at Google. “Where that leads is uncertain.”

Google has been both at the forefront of the technology’s development and a source of the anxiety surrounding it. Its Street View service, which lets internet users zoom in from above on a particular location, faced privacy complaints. Google will blur out people’s homes at their request.

Google has also introduced an application called Goggles, which allows people to take a picture with a smart phone and search the internet for matching images. The company’s executives decided to exclude a facial-recognition feature, which they feared might be used to find personal information on people who did not know that they were being photographed.

Despite such qualms, computer vision is moving into the mainstream. With this technological evolution, scientists predict, people will increasingly be surrounded by machines that can not only see but also reason about what they are seeing, in their own limited way.

The uses, noted Frances Scott, an expert in surveillance technologies at the National Institute of Justice, the justice department’s research agency, could allow the authorities to spot a terrorist, identify a lost child or locate an Alzheimer’s patient who has wandered off.

Kinect, an add-on to Microsoft’s Xbox 360 gaming console, is a striking advance for computer vision in the marketplace. It uses a digital camera and sensors to recognise people and gestures; it also understands voice commands. Players control the computer with waves of the hand and then move to make their on-screen animated stand-ins — known as avatars — run, jump, swing and dance. Since Kinect was introduced in November, game reviewers have applauded, and sales are surging.

To Microsoft, Kinect is not just a game, but a step toward the future of computing. “It’s a world where technology more fundamentally understands you, so you don’t have to understand it,” said Alex Kipman, an engineer on the team that designed Kinect.

Mirror, mirror

Daniel J McDuff, a graduate student, stood in front of a mirror at the Massachusetts Institute of Technology’s Media Lab. After 20 seconds or so, a figure — 65, the number of times his heart was beating per minute — appeared at the mirror’s bottom. Behind the two-way mirror was a web camera, which fed images of McDuff to a computer whose software could track the blood flow in his face.

The software separates the video images into three channels — for the basic colours red, green and blue. Changes to the colours and to movements made by tiny contractions and expansions in blood vessels in the face are, of course, not apparent to the human eye, but the computer can see them.

“Your heart-rate signal is in your face,” said Ming-zher Poh, an MIT graduate student. Other vital signs, including breathing rate, blood-oxygen level and blood pressure, should leave similar colour and movement clues.

Faces can yield all sorts of information to watchful computers, and the MIT students’ adviser, Picard, is a pioneer in the field, especially in the use of computing to measure and communicate emotions. For years, she and a research scientist at the university, Rana el-Kaliouby, have applied facial-expression analysis software to help young people with autism better recognise the emotional signals from others that they have such a hard time understanding.

John Ross, chief executive of Shopper Sciences, a marketing research company that is part of the Interpublic Group, said Affectiva’s technology promises to give marketers an impartial reading of the sequence of emotions that leads to a purchase, in a way that focus groups and customer surveys cannot.

“You can see and analyse how people are reacting in real time, not what they are saying later, when they are often trying to be polite,” he said. The technology, he added, is more scientific and less costly than having humans look at store surveillance videos, which some retailers do.

Darker possibilities

Picard enunciates a principled stance, but one that could become problematic in other hands. The challenge arises from the prospect of the rapid spread of less-expensive yet powerful computer-vision technologies.

At work or school, the technology opens the door to a computerised supervisor that is always watching. Are you paying attention, goofing off or daydreaming? In stores and shopping malls, smart surveillance could bring behavioural tracking into the physical world.

More subtle could be the effect of a person knowing that he is being watched — and how that awareness changes his thinking and actions. It could be beneficial: A person thinks twice and a crime goes uncommitted. But might it also lead to a society that is less spontaneous, less creative, less innovative?

Google could have put face recognition into the Goggles application; indeed, many users have asked for it. But Google decided against it because smart phones can be used to take pictures of individuals without their knowledge, and a face match could retrieve all kinds of personal information — name, occupation, address, workplace.

“It was just too sensitive, and we didn’t want to go there,” said Eric E Schmidt, the chief executive of Google. “You want to avoid enabling stalker behaviour.”