Using Big Data to Discover the Truth Isn't as Easy as It Looks
NEWS ANALYSIS: Having lots of data isn't really the way to discover truthful information because first you have to make sure it's the right data and that it's also the most relevant data.NEWS ORLEANS—It was clear when I stood up to speak at the session on big data at the Society for Professional Journalism's Excellence in Journalism 2016 conference here that I wasn't addressing your average trade show audience. The hundred or so people in front of me were all professional journalists, which meant that they were expecting a no-nonsense, practical look at how they could use vast data archives to find the truth on a wide variety of topics, ranging from political corruption to the spread of the Zika virus. With me in the front of the room was Pam Baker, the highly respected author of Data Divination: Big Data Strategies and Louis Lyons, chief operating officer of ICG Solutions, the company that created the LUX2016 data analysis engine and which helped with our examination of viewer reaction to last year's Democratic and Republican primary debates. Our discussion started out with my description of how we used data analysis to figure out who won last year's debates well before the major news organizations had polling data to release. But as valuable as our first effort to use big data to support a feature article was, the fact is that data analysis goes far beyond what I was able to do in eWEEK's first attempt.
This is no surprise because the analysis of large data sets is still in its infancy, and while data analysis can be used by most new organizations to gain important insights, finding the way is still hard.