pn854. Three Polls, Very Different Results. What to Watch Out for in Polls

 Over the past two weeks we have published the results of two polls taken at about the same time (pn851, pn853), and now we publish the Newshub-Reid results. The 1NewsKantar poll showed Labour still ahead of National while the Roy Morgan poll showed National and ACT could form the government.  Today, however,  the Newshub-Reid poll shows Labour and the Greens to be well ahead.  How could the Morgan poll be so out of line with the other two results?)

Four  polls were asked about the preferred Prime Minister over roughly the same time  period. The NZ Herald wrote that Jacinda Ardern's support was "plummeting". I wonder whether they'll have it "soaring" when they report the Newshub-Reid poll.

Earlier, the Herald and Helen du Pressis-Allan used the dubious approval-disapproval  rating (also called the confidence index or the net favourability rating)  to show Christopher Luxon ahead of Ardern despite the preferred PM question showing otherwise. 

I published the results and my criticism of the approval-disapproval  rating method in pn845. "Up the Pole.  Ardern and Luxon Approval Ratings Meaningles, Misleading and Deceitful" and repeat them here to save you having to look them up.    The Approval-Disapproval A-D (or confidence, favourability) score is obtained by deducting the percentage of those who disapprove from approve while ignoring those who  don't know or refuse to answer.  Luxon's +22 was "better" than Ardern's +15. Hence, he had overtaken her.

I called the statistic dubious for four reasons: 

1) The high proportion (11% for Ardern and 37% for Luxon) of those polled who "did not know" or refused to answer (see table). 

2) The dubious means of calculation.  Subtracting disapprove from approve gives some sort of average but it is highly influenced by relative percentage size, and does not take the undecided into account. Despite the PM's approval rating being significantly more than Luxon's, the statistic only gives Ardern a +15 while Luxton is much higher  on  +22.  1News tells us this is a first time ever for a Leader of the Opposition. 

3) The approve-disapprove question is dichotomous. Those polled had to state one or the other.  The question does not allow for degrees of approval or disapproval. Those who disapproved or approved only a little counted the same as those who disapproved or approved a lot. A far better measure would have been an approval-disapproval scale, e.g.,  from 1-5, 1-10, or -5 to +5, with 0 being a neutral position.  

4) The statistic's  result should be broadly similar to the results of the preferred PM question (Ardern 35% and Luxon 17%), but it is the stark opposite. Well behind on preferred PM question Luxon is well ahead on the approval-disapproval rating. In reality—if not in the poll— both results cannot be simultaneously true. 

Not all poll questions are equal. The party vote question is a more robust measure than preferred PM and far more robust than the personal approval ratings. 

We should not take too much notice of a single poll.  It is better to look at several over time, while bearing in mind the pollsters' methods, overall reliability, and possible biases.  

As for the polling companies and journalists, take a pinch of salt and watch out for their biases, too.

New Zealand has four main polling companies,  Curia Taxpayers Union used by the National Party, Talbot Mills formerly UMR Research  used by Labour, the Australian and NZ company Roy Morgan, and 1NewsKantar, previously Colmar Brunton.  Morgan results tend to be more favourable to National and Kantar to Labour.

-- ACW