While government has been using Big Data for years in areas like intelligence, the census, taxes and Social Security, "what is new to government is the realization that Big Data can and should be used to drive, reform and refine public policy, as well as defend and protect," she said.
A.J. Clark, president of Thermopylae said that investments in counter-terrorism after 9/11 meant that, "certain elements of the public sector led the Big Data movement. Statistics from agencies involved with Remote Sensing — satellite imaging, full motion video collection, aerial imaging such as Light Detection and Ranging (LiDAR) — show that there has been a watershed of big data growing in their environments," he said.
But, "many of the advances in big data did not transcend the agencies they originated in and the public sector as a whole did not benefit," he said.
Williams said another factor is that the private sector's use of Big Data is driven by the profit motive and competition, while, "government is not profit-driven and maintains a captive audience. So, the drivers of Big Data projects in government are inherently different and frankly not as urgent, except for intelligence and law enforcement."
She estimates that most of the public sector is about five years behind the private, "with no imperative to catch up and the reality that Big Data projects may remain an unfunded mandate or underfunded project/program."
Other experts agree, although there are mixed opinions on how far behind. Clark estimates it to be 18 months, while Chris Petersen, CTO and cofounder of LogRhythm, said he thinks it amounts to "years," but that the gap is less in the area of cyber security.
CIOs admit that they are behind. They told Desouza that they simply don't have the technical capability or staff with enough expertise to conduct Big Data analytics. Not one had made use of unstructured data.
"Isn't that a critical element of Big Data? If so, then we are not doing anything in the Big Data space, as we have not touched unstructured data. All of our data has some structure, and most of it is highly structured," one of them said.
But it has exploded in the private sector. The Boston Globe reported more than two years ago that Massachusetts alone is home to more than 100 companies focused on Big Data.
Pam Dixon, executive director of the World Privacy Forum (WPF), estimated last fall that there were about 4,000 data brokers in the U.S., all of them collecting and selling data on people's activities, both online and off -- license plate scanning is not confined to law enforcement departments.
Sign up for Computerworld eNewsletters.