Investments in big data are getting bigger, but big results are still eluding most organizations, according to a new survey of big data professionals.
Conducted by business analytics firm SAS, the survey reports that 83 percent of respondents call their company’s big data investment as “moderate” or “significant.” However, many of these projects are still in the early stages: Just 33 percent say that they have derived value from those investments, while another 49 percent say it’s just too soon to tell.
“The good news is that large organizations understand the importance of big data solutions and are taking steps in the right direction,” says Matthew Magne, global product marketing manager for data management at SAS. “But while the trend is encouraging, many of these companies still have a long way to go; data access and quality remain challenges.”
According to the survey, respondents generally believe that big data will provide their company with a competitive advantage. But they cite several challenges to achieve that goal: data integration, complexity and integrity, as well as unstructured data capture and management. Other issues are associated with culture, skills and access to appropriate technology.
Among the other survey findings:
Data access
- For many organizations, data access belongs to the "data crunchers" – data scientists and business analysts. Only 13 percent have self-service access without IT supervision.
- About a third of respondents have instant access to the data they need. But 28 percent say it can take up to a week, 8 percent a month, and 3 percent a month or more.
- Just under half of companies surveyed have a chief data officer. Others in IT are typically in charge of the organization's data.
Moving to the cloud
- Cloud adoption has been slow – just 19 percent of organizations are accessing data in the cloud. The rest continue storing data on-site.
- Hadoop is on the rise – 56 percent of those actively engaged in a big data initiative have purchased a Hadoop-centered solution. Of those, 87 percent have implemented it.
- The most popular Hadoop distribution is Cloudera at 44 percent, followed by Hortonworks at 29 percent.
- Another 40 percent are considering Hadoop to replace an existing data warehouse.
- Six in ten companies have increased spending on data infrastructure over the past two years.
“The movement to cloud storage is still somewhat guarded, even among those already entrenched in big data. Other research indicates that the biggest concern with cloud storage is security,” added Magne.
Machine learning and artificial intelligence
- About one in five companies have implemented machine learning or artificial intelligence (AI) technology.
- Another 23 percent have tried machine learning or AI on an experimental basis.
- The largest segment, 42 percent, are exploring AI solutions but have yet to invest.
“Nearly all respondents are seeing traction from their data-driven projects, with about half saying they have many data success stories,” said Magne. “Overall, the survey shows that big data initiatives are making progress and continuing to gain traction.”
