David Waldrop and I were sitting in the National Press Club’s legendary Truman Lounge watching two huge television screens showing CNN’s coverage of the second Republican debate.
Sitting on a table in front of us was a laptop computer running Linux and a data analysis tool called LUX2016 and it was getting as much attention as the screens on the walls. Waldrop, who is CEO and founder of ICG Solutions in Chantilly, Va., was pointing out the changes in the constantly moving bar graphs on his laptop.
Those graphs displayed on Waldrop’s computer were reporting the readings from Twitter as users everywhere reacted to the debates and to the candidates. The readings tracked the number and nature of the tweets so it could update the results every 10 seconds. On the graphs, we saw segments for each 10-second period along with a color code for statistically important changes.
The color changed according to the variance measured by standard deviations. As a result, if the public on Twitter increased its response to a candidate by 10 standard deviations, the segment was colored red.
Equally important to how much of a response a candidate received was the duration and consistency of the response. For example, we might see a big response for 30 seconds when a candidate said something dramatic, but then a lower number other times.
But not everything worked exactly that way. One candidate, former Hewlett-Packard CEO Carly Fiorina, began picking up steam as the debate began, but instead of dropping when she wasn’t talking, her response continued to grow. By 10 p.m., it was obvious that no matter what happened for the next hour of the debate, Fiorina had won.
Later, as we analyzed the numbers and looked at samples of the actual social media activity, it was clear that we were witnessing a turnaround in politics. The numbers didn’t appear to come from a social media team, but rather from real people.
Equally surprising was what happened with some of the other candidates. Notably, Republican front-runner Donald Trump’s numbers dropped, slowly at first, then quickly.
By the end of the night, Trump’s numbers reflecting his social media support were some 900 points down from Fiorina. Afterward, Waldrop and I discussed what this drop might mean. Had we seen the beginning of Trump’s decline?
As it turned out, it wasn’t the beginning. What we were seeing was perhaps the first public evidence of a greater decline that had begun three weeks earlier. The next day Ben Schrenckinger, reporter for Politico Magazine, published the results of his own study using different data, but showing the same result that we had seen.
Our results also showed data that might give some of the other candidates reason to rethink their campaigns.
Big Data Analysis Tool Predicts Republican Debate Winners, Losers
Wisconsin governor Scott Walker wasn’t eliciting any significant response at all, no matter what he said. The debate audience simply didn’t seem to react to Walker or his message.
Other candidates including New Jersey governor Chris Christie and former governor Mike Huckabee weren’t doing as poorly in the results as Walker, but the Twitter response to those candidates was mostly tepid with some occasional but limited activity. The other GOP candidates were at least holding their own.
Of course, none of this is forecasting the eventual outcome of 2016 election. It’s much too early to even know who will be running in 2016, much less know what the sentiment of the voting public might be.
But we did learn the sentiment of the viewing public on that night of the debates and it was later supported by independent pollsters and by the usual pundits. What was critical was that I learned the results at least 12 hours before anyone else did.
Had the candidates or their parties been watching these results, they would have known what was resonating with the public, what wasn’t resonating, and in some cases they would have been able to make timely changes.
While there wasn’t any coaching going on during the debates, over the course of any election campaign there’s a lot of communication between candidates and their strategists. If they find that something the candidate is saying isn’t resonating, or worse, is turning voters off, the candidates will change their message quickly enough.
But something else important came out of our work with LUX2016 during the debate. As part of the process, we discovered just how agile this cloud-based big data analysis tool is.
“LUX is a real-time analytics platform,” Waldrop explained. “It includes a complex events processor with plug in modules [and] an intuitive easy to use interface.” Waldrop said that you can combine inquiries “as if they were Lego building blocks and stack them on top of each other.”
Waldrop said that one thing that makes LUX so powerful is the fact that it’s data agnostic. While we were analyzing Twitter messages as an indicator of voter sentiment, we could have been measuring anything that’s quantifiable.
The difference between LUX and many other big data analysis platforms is the fact that it runs in real-time. We watched voter sentiment change as it happened, but Waldrop said that it could have been watching security events or the temperature of bearings on a factory full of manufacturing machines.
LUX started out in the intelligence community and Waldrop won’t discuss how those clients were using the software, but what’s really important is that it’s no longer necessary to analyze big data weeks or months later. Now analysis can be done in real time.