Contradictory surveys: Which one should we believe?

I’ve been an amateur pollster for the last 15 years and have tracked both US and Indian polls closely. It wouldn’t be far off the mark to say that polls in USA are much more reliable than Indian polls. There are many reasons for this, which we won't get into now.

But at the very least we expect the pre-election polls to get the trends right, if not specific numbers and percentages. And given that predicting a national trend is tough, at the bare minimum, pollsters should be able to tell which way the wind is blowing at the states level reasonably well. 

Unfortunately, this time the opinion polls at the states level are all over the place. Let’s compare the results of two polls that show trends from three key states:

Karnataka     

	BJP	Cong	JD(S)
CNN-IBN & CSDS	36%	46%	12%
NDTV  & Hansa	40%	39%	13%

                 
Tamil Nadu

	NDA	AIADMK	DMK	Cong
CNN-IBN & CSDS	22%	32%	23%	12%
NDTV  & Hansa	16%	41%	27%	5%

               
Bihar

	NDA	RJD-Cong	JD(S)
CNN-IBN & CSDS	43%	28%	16%
NDTV  & Hansa	32%	27%	23%

                              
I could give examples of many more polls and many more states, but the above numbers are enough to drive home my point - the results from some of the best Indian pollsters appear to be all over the place when it comes to some of the bigger states.

On the one hand, Hansa/NDTV is giving BJP a 1% narrow lead over Congress in Karnataka, while CSDS/CNN-IBN is showing Congress with a 10% point lead in the same state! Which of the 2 polls should we trust? And why is there such a massive discrepancy between two supposedly credible polling agencies? Is it due to the sample size or demographic/sampling variation or variation among projected voter turnout among different religious & caste groups in Karnataka? One certainly needs to take a closer look at the methodology of both CSDS & Hansa.

In Tamil Nadu, both CSDS/CNN-IBN and Hansa/NDTV are showing AIADMK with a reasonably large lead, but they are nowhere close to each other when it comes to predicting the vote share of AIADMK, NDA & Congress. Then there is CSDS/CNN-IBN giving AIADMK only 32% votes, whereas Hansa/NDTV poll predicts 41% for AIADMK.

Needless to say, there is a big difference in the number of seats projected by both the polls for AIADMK.In Bihar both the polls are in agreement on 2 points - 1. NDA is leading... 2. Cong-RJD alliance will get 28% votes...But they differ widely on 2 imp stats: 1. Vote share of NDA..2. Vote share of JD(U)...

CSDS/CNN-IBN is projecting NDA to poll a whopping 43% votes in Bihar, a full 11% more than what Hansa/NDTV poll estimates... the latter also predicts a much better showing for Nitish Kumar's JD (U) than CSDS estimate.

Only May 16 will tell us who is right and who is wrong! But these numbers make one thing clear - all the polls should be taken with a pinch of salt, especially in a large, diverse, multi-ethnic society and country like India.