Every day, we create 2.5 quintillion bytes of data-so much that 90 percent of the data in the world today has been created in the last two years alone. This data comes from everywhere: from sensors used to gather climate information, posts to social media sites, digital pictures and videos posted online, transaction records of online purchases, and from cell phone GPS signals, to name a few. This data is big data. But the big news in big data isn't about size - it's about complexity and diversity of the data structures.
Each day websites generate many terabytes of raw, complex data about customers' viewing and buying habits - also called interactional data. These Web logs must be transformed and refined in preparation for analysis. The analysis provides insight into customer preferences. Harnessing and integrating big, diverse data with other data provides deep customer insights. When combined with other transactional data about customers, the insights support marketing campaigns that provide the right offer, delivered at the right time, to the right consumer.
It does seem obvious that a very high proportion of data is multi-structured - in a variety of different formats: Much of our workday is spent reading or writing e-mails, reports, or articles and the likes, in conversations, or listening to live or recorded audio. This clearly makes a case for tapping multi-structured sources. Big data is more than a challenge; it is an opportunity to find insight in new and emerging types of data, to make your business more agile, and to answer questions that, in the past, were beyond reach.
A big-data environment brings new challenges and opportunities for analysing, managing and mastering the massive volume, velocity, variety and complexity of information created by existing systems and emerging sources. Businesses have been immersed in this environment for several years as they contend with new types of information from Web interactions, mobile devices, social media, machine data and other sources.
However, only the largest and most analytics-focused companies (such as Google) have had the time, resources and tools to implement big-data analytics. But now, many more can successfully and affordably leverage a platform alongside their data warehouse to extract valuable strategic insights from new and newly accessible information sources and feed business innovation at a previously unimagined pace.
Extreme Data, Extreme Pain?
The ability to glean information from large amounts of data is rapidly becoming a strategic imperative as the number of data sources grows exponentially and information expands rapidly in both volume and complexity. In an April 2011 report by Gartner titled " 'Big Data' Is Only the Beginning of Extreme Information Management," Gartner concluded that, "Concern over big data represents the first manifestation of the extreme challenges that will overwhelm existing information management practices and technology ... The ability to manage extreme data will be a core competency of enterprises who are increasingly using new forms of information (text, social, context) to look for patterns that support business decision (Pattern-Based Strategy)."
Sign up for Computerworld eNewsletters.