Power of Big Data

Tech blog

The recent US polls were a milestone in psephology. The Democrats have built a futuristic analytics team to muster and process massive amounts of data about voters. Big Data played a big role in Obama’s victory and left clueless Republicans struggling to react. 

 

To gauge public opinion, traditional polling methods depend on sample surveys or focus groups; they reach out to a small cross-section of the population, which by definition leads to off-the-mark results. Outgrowing this approach, the Democrats have developed tools to assess and talk to each voter individually, achieving an unprecedented breakthrough in ‘micro-targeting’.

To see Obama through, they gathered as many as one thousand variables, you read it right, on American voters drawing from voter registration records, consumer data warehouses and past campaign contacts. The stunning effort even tapped into the TV viewing habits of individual households locked in set top boxes by negotiating with cable operators. Though they were legally prohibited from accessing personally identifiable information there was enough data to see which way people were swinging.

Democrats engage in a similar effort in all the elections they face from presidential to local congressional districts. Crunching all this data, which is updated constantly, they are able to predict the outcome of different polls with alarming accuracy six months before they are held.

They are obviously not content just knowing which way voters will go, but want to turn them in their direction. Volunteers and carefully targeted advertisements go after the segments of voters they think they have the best chance of swaying. While the Democrats high-tech poll machinery has evolved over the last few years, in the recent presidential election it lived up to its potential.  

As a flurry of professionals - specialists in text analytics, computational advertising, and online experiments – joined Obama team, Romney campaign, which had outsourced similar work, watched in awe wondering what they were up to. They were baffled to see Obama incomprehensibly buying advertisements in Republican strongholds or obscure media outlets where Democrats went hunting for small pockets of voters. They never fully
understood the game till the results came out. The recent issue of MIT Technology Review has a three-part series on how Democrats have refined the art and science using Big Data technologies. While the feat is admirable, it also raises questions on how the information people put out about their digital selves could be used to manipulate them.

Businesses are accessing similar data and deploying similar technologies to reach out to their customers.  In a famous incident this year, the US retail giant Target tracked the
purchasing behaviour of a customer, who happened to be a teenager, and rightly deduced that she was pregnant. The discount vouchers it sent to the customer landed with her father, who did not know that his daughter was pregnant till he read
the Target mail.