Big Data tools can provide value for security professionals in a forensic sense, according to Teradata enterprise risk management director, Sam Harris.
When monitoring communications and network data, Harris said data is being increasingly machine generated, and difficult for people to look at and discern what is happening.
"As a result, some incursions begin, such as spear phishing or malware that comes over the perimeter defences," he said.
Harris admits that Big Data "means a lot of things to a lot of people," but he said the notion that all big data is unstructured is not correct.
"All data has structure, but not the structure you are anticipating," he said.
Fortunately, if a company is continuously monitoring its network, Harris said there are a number of things to look out for, such as communications that are using a protocol the company would normally not use on the network.
"That's a telltale sign that you have malicious code on your network," he said.
Advanced and persistent
With signs of next generation threats using big data and analytics to breach into systems, Harris recommends a proactive response to counter this increasing level of sophistication.
"If you look at empirical evidence, there is a supported view that the volume and sophistication of threats is increasing," he said.
APTs (advanced persistent threats) are singled out as being particularly challenging because they tend to have a "designer malware" aspect to them.
"They do not come with signatures that are already identified in signature-based malware detection tools," Harris said.
Thus, when these types of threats are introduced to a computer environment, Harris said they are "particularly insidious" because they are difficult to detect until they start to act.
"There are telltale signs of activity that there is malware present in the environment, and continuous monitoring and detection techniques are key to rooting out these types of APTs," he said.
Sign up for Computerworld eNewsletters.