Predicting How We Predict the Future - The Next Generation of Polling
By Richie Hecker
What did the Presidential election teach us about predicting the near future?
For years, we’ve been told that polling is sacrosanct. Polling predicts — until it doesn’t. What we saw happen this election cycle is the limitation of polling. The media and pollsters all predicted Clinton would win. Yet, Trump won. On Election Day, The New York Times placed Clinton with an 85% chance of winning. Even Steve Schmidt, Republican strategist, predicted a Clinton outcome three weeks before the election. “I think she is trending over 400 [Electoral Votes],” he stated on Morning Joe.
Then it all shifted in an overnight instant. Trump emerged the winner, shocking nearly everyone. What happened?
Let's first take a look back to the year 2008, which was the most accurate presidential election for polling in decades, with 92% of pollsters correctly picking the winner, according to fivethirtyeight.com. It's been a mere eight years since the "good old days" when polling was accurate, and social media was benign.
According to Henry De Sio, Chief operating Officer for the successful Obama 2008 campaign, a big part of the problem lies in a failure to recognize the ideals that now informs voting behavior:
“Back when there were some two-dozen hopefuls at the start of the 2016 presidential primaries, I believed the edge would go to the candidate who most closely identified as the “Changemaker” in the field. That’s because Changemaker qualities — an innovative mind, a service heart, an entrepreneurial spirit, and a collaborative outlook — embody the new definition of leader. We see these attributes in ourselves, and this is what won Barack Obama the election in 08’. The 2016 contest was confused around this point. Pollsters didn't account for the Changemaker Effect in their surveys; the media didn't report through that lens; and this crop of candidates didn’t appeal to us as Changemakers. Trump did run an unconventional campaign, however, which differentiated him decisively from all the other candidates. This likely ticked a box for many voters searching for the Changemaker in the race. But the conventional measurements simply did not pick up on these undercurrents.”
To learn valuable lesson about polling and predictions from the November Election, the first thing to examine is where errors can occur in polling:
Margin of Error: Every poll has a margin of error. This is combination of two factors. First is sampling error, which provides a number representing how close an estimate should be to the real value based upon the size of the sample. Second is “non-sampling error,” a combination of factors that could also influence the accuracy of an estimate. The sampling error is easy to compute, but the non-sampling error is tricky to estimate because so many factors can affect the accuracy of an estimate. As a result, most pollsters simply report the sampling error as the “margin of error,” ignoring all of the other sources of error, and leading to a high likelihood that unusual factors will have an impact on voting. Typically, the sampling error for political polling is around plus or minus 3%. The typical sample size for political surveys is about 1000 voters. In a normal predictive environment, this has some accuracy. However, given how rabid our election season was, there were many people who were secret Trump supporters that simply didn’t want to acknowledge that in public. These individuals may have been undecided voters who voted for Trump at the end, people who wanted to vote for him all around and be discrete about it or people that would have normally voted Democrat but didn’t show up, in protest or in apathy. Voter turnout is difficult to predict. In a normal election, people are more likely to follow patterns. This election was emotionally charged, and people broke normal frame. People may not answer their phone, say if there is something stressful happening in the media, and in the case of Republicans, they may not answer because the polling apparatus is seen as ‘biased mainstream media.’ Voter sentiments that were not captured in polling increased the non-sampling error and were ultimately revealed only at the voting booths.
Likelihood of Voting: Every poll response receives a weight reflecting the likelihood that the person responding will vote. Since the best predictor of future behavior is usually past behavior, people voted in the last Presidential election were usually weighted as being more likely to vote than those who did not. But this year, the “enthusiasm gap” resulted in a high number of disaffected potential Clinton supporters not voting, especially across the rust belt, while a larger number of people who did not vote in the last election made the effort to vote this year.
Voter Sampling: The method and nature of the sample is important. Some surveys use mail, others Internet, while others use landlines or cell phones. Mixed-mode or mixed channel surveys yield a broader set of results that can be weighed and combined. For example, telephone polls made over landlines disproportionately favor older voters, while younger voters are more likely to respond to mobile phone polls. Balancing the results of multiple poll methods will have a better outcome than any one by itself. To understand what the results really mean, it is also important to match the voter pool sampled with demographics and then balance the pool to reflect the likelihood of turnout by demographic segment. This often is not disclosed when disclosing poll results. This is mainly because it is complicated and detailed math and statistics don’t make good television. One survey can inform the population, which can be skewed as a result of who it was targeting.
We need to always be testing new methods at getting to what people really think and how they will act. This presents an opportunity for entrepreneurs to come up with new ideas and new technologies to predict outcomes. At the end of the day, data-driven tools are superior forecasters than even the most enlightened gut-feeling. There are startup research companies that use innovative approaches to yield more accurate opinion gathering and predictions. Let’s look at several that beat the pundits and media at their own game and how we can learn from the experience.
In February 2016, the Silicon Valley firm 1World used a Triangulation Method to predict better than any other group in the country the Republican primary results in South Carolina. 1World correctly picked the order of candidate placement as well as the exact percent of vote for the winner, Donald Trump.
Additionally, 1World predicted the General Election popular vote within a 0.5% for both Clinton (47.7% predicted, 48.1% actual) and Trump (46.1% predicted, 46.4% actual); again what is notable here is that no other traditional polling firm in the world got closer on this prediction.
So how did 1World do it?
The 1World triangulation method started with analysis of polling data published by other organizations. About a dozen polls for each election were entered into a database that weighted the results according to the sample size, date of the poll, and other factors, yielding a combined estimate for each candidate. These results typically reflected a number of undecided voters.
“A key component of 1World’s accuracy is that our polls consider a wider variety of factors than most pollsters normally consider,” said Brad Kayton, COO and GM of Data Analytics of 1World. “Our triangulation method includes the key ability to do spot opinion gathering across a network of digital properties and segment the data based on location and sentiment of where the user came from.”
Another startup takes a completely different view. Instead of transforming polling, they look to monitor and “listen” to social media. “Polls failed for one simple reason: People will not tell you what they think anymore,” explained Julien Newman, CEO of political outreach startup Turnout. “And the solution is simple, don’t poll at all. Tap into where people are already talking about this stuff and measure that.“
A third startup’s approach is to eliminate the opinion of polling and social media altogether, and instead to follow the trail of money. Maxim Lott & John Stossel launched ElectionBettingOdds.com, which analyzes the cash-money betting odds at online book-makers to predict elections. “People should look at betting instead, since it’s the best predictor. If there are any better predictor of the future, you could make money betting based on that.” Betting markets failed to predict Trump and Brexit, but still got it right before networks did; bettors gave Trump a 90% chance at 11pm. Historically, studies have found bettors’ predictions to be more accurate than polls; for example, bettors correctly predicted Clinton’s win over Sanders in the primary, consistently putting his odds very low, at less than 20%. says Maxim Lott.
Another powerful technique is running a counter-poll. A counter-poll approach is a psychological technique to get guide people to acknowledge what they are thinking indirectly. This is a great way to validate assumptions. For example, in a situation where voters are likely to hide information, ask respondents who they think their neighbors are voting for and why. This will often lead to transference where voters will transfer their own ideas onto their neighbors. In a situation where a lot of voters didn’t want to acknowledge they were voting for Trump, this can be telling. Make sure the results of this counter -poll match your statistically analyzed self-reported opinion poll. If they match, stand by the winner. Kellyanne Conway, Trumps campaign manager and long-time professional pollster, used this indirect approach to get at the ability to win the rust belt states, “the inside flush” to get above 270 electoral college votes, while others missed it. Most of the media was actually calling the Trump campaign lost in the closing days, such as with Trump’s election eve visit to Michigan. Not so.
So how can the one predict the future? The best approach is to utilize multiple systems with checks and balances. Start with making sure your polling is across multiple channels and that your demographics reach an accurate sample of the population. Match your channel of collection (i.e, cell phones, internet, mail) with historical turnout data broken down by demographics. This way you can accurately measure who is likely to turnout by demographics in each location. This is more valuable than generalized data. Next, pay attention to what is going on in social media and the nature of the conversation. Look at whose comments are most engaged. This will be telling to see whose message is resonating. Finally run a quick counter-poll and check-in at your local bookmaker. Then blend what you learn and make a judgment. If they all match, you have a winner. If there lacks clarity from polling, the counter-poll may provide the most accurate information and finally check real time sentiment by checking betting odds.
The predictions sector will change over the next election cycle more than we’ve seen in the last 50 years. If 2008 was the good old days of polling and predicting, today’s world of interfacing with the electorate takes you down the dark and windy path of social media ranting, fake news, and the gambler's den to grope for answers. The traditional system broke down, research methodologies had not kept up with current times. Technology, including harnessing social media channels, and innovative analytical approaches, can offer us the salvation from the polling issues we just saw in the 2016 election. In every problem lies an opportunity, and with so much at stake the one thing you can surely predict is that smart minds will be at work to provide better and effective solutions to opinion gathering. Expect to see a rush of entrepreneurs entering the space with new methods, technologies and tools and ultimately make a claim that they can predict the future. Would you bet on that?
About 1World Online
1World Online has developed and maintains a Consumer Intelligence Platform providing simple but effective engagement applications, unique-engaging content, ongoing market research, and insightful end-user analytics. 1World is headquartered in San Jose, CA and has offices in Boston, MA, India, Mexico, Europe, Japan, Ukraine, and Singapore. Please visit us at http://www.1Worldonline.com or contact us at email@example.com.
To learn more, please fill out the contact information below: