And there are hundreds of stories of private firms using Big Data to gain market share, improve their bottom line, curb fraud and theft and become more efficient.
Compass Group Canada, which operates more than 2,000 food service locations in that country, recently began using software from Boston-based Lavastorm Analytics to analyze data on money or merchandise either being misplaced or stolen by employees or customers. They found the analysis much more efficient than analyzing thousands of hours of video footage.
Desouza also cited Merck's use of weather data in July 2012 to anticipate greater demand for its allergy pill, Claritin, in May 2013, and Google's early detection model of flu outbreaks, called Google Flu Trends that, in 2009, was able to track trends in the H1N1 flu epidemic "days faster" than the federal Centers for Disease Control.
That doesn't mean there is no progress in the public sector. Boston's former police commissioner, Edward Davis, recently joined the board of Mark43, a tech startup founded by three Harvard graduates that makes software to analyze data on crime and gangs, and improves management of police records on suspects.
The good news for the fledgling company — but not so good for the reputation of government services -- is that the field is relatively wide open. The class project that led to the founding of Mark43 was to come up with a program that could use analytics to track gang relationships in Springfield, Mass.
"We didn't know much about law enforcement," said CEO and cofounder Scott Crouch. But, "the first thing we noticed was that their law enforcement software was awful."
Improving the technology and the productive analysis of data by government clearly could make government more efficient and offers the hope of curbing the classic bugaboos of "waste, fraud and abuse."
"Analytics now holds great promise for increasing the efficiency of operations, mitigating risks, and increasing citizen engagement and public value," Desouza wrote in his report.
But it will take some changes, experts say. "Human capital is clearly a major problem," Williams said. "Government has a serious lack of scientists, an aging workforce, ongoing budget and pension challenges and an inability to attract technology-savvy employees."
Government is also hamstrung by long-term contracts with vendors supplying proprietary technology, Stirman said. "Because government agencies are so large, they get into contracts worth hundreds of millions, and they can't easily untangle themselves even if there are problems."
The hope for change, he said, lies in the fact that, "most of the (Big Data) technologies are open-source, which means you don't have to get locked in with a vendor with proprietary technology. Anyone can learn it — you can become an expert without going through a big certification process. It's really important that government take advantage of it," he said.
Sign up for Computerworld eNewsletters.