Given last year's revelations about the National Security Agency's (NSA) massive surveillance and data analytics conducted on Americans, along with continuing stories about local police scanning thousands of license plates per day, it might sound absurd to say that government lags behind the private sector in the use of Big Data analytics.
But those examples tend to be outliers among the nation's sprawling bureaucracies, especially at the state and local levels. In general, the private sector is well ahead of the public sector in the use of Big Data analytics, according to a recent report titled "Realizing the promise of Big Data," sponsored by the IBM Center for the Business of Government.
While the report's author, Kevin Desouza, an associate dean for research at Arizona State University, cited multiple examples of it being used in government, he found that the overall promise of Big Data analytics is largely unrealized so far in the public sector. He called it, "a new frontier" for government at all levels.
There are multiple reasons for that, Desouza concluded, after interviewing 22 federal, state and local chief information officers (CIO). One of the most significant is that those outside of IT don't yet understand the concept, or are fearful that the public will perceive Big Data as Big Brother.
"Outside of IT no one gets the term," one CIO told Desouza. "The few (managers) that do have some views on it have been put off due to the privacy concerns because of the NSA's surveillance program."
Those concerns are legitimate, but lack some context. Kelly Stirman, director of product marketing at MongoDB, said what many other experts say: "The private sector knows much more about you than government ever will. But people have been willing to trade their information for convenience."
That is what Desouza found — that the private sector has been able, through things like terms-of-service agreements for mobile apps, "to collect immense amounts of information on individuals with limited pushback."
With government, it is different. There were ferocious protests last year after revelations from former NSA contractor Edward Snowden about the extent of the agency's surveillance of U.S. citizens, including reports that on a single day the NSA had collected hundreds of thousands of email address books from vendors including Yahoo, Microsoft's Hotmail, Facebook, Google's Gmail service and others.
Beyond that is the reality that the private sector tends to be more nimble than government and is better able to attract top talent. Government is more likely to be constrained by both political pressure and bureaucracy.
Kimberley Williams, chief strategist, public sector, at Informatica, said none of this should be a surprise, since the roles and motives of the private and public sector are different.
Sign up for Computerworld eNewsletters.