Top 10 Priorities for Big Data Management
July 7, 2015 11:40 AM
1. Demand business value from big data
The first step toward value is to manage big data. If you dont capture, store, process, and deliver big data, you wont have it to repurpose for valuable applications in customer intelligence, operational efficiency, business monitoring, and so forth, said TDWI. Eighty-nine percent of survey respondents say BDM is an opportunitybut only if you seize it. Know the common paths to business value and follow them. The primary path to business value from big data is discovery analytics. A second path joins new big data with older enterprise data to extend complete views of customers and other business entities. A third path taps streaming big data to enlighten and accelerate time-sensitive business processes.
2. Use big data to create new applications and extend old ones
Streaming big data can enable business monitoring applications, and big data can expand the data samples that data mining and statistical analysis applications depend on for accurate actuarial calculations and customer segments or profiles, TDWI said.
3. Get training (and maybe new staff) for big data management
The focus should be on training and hiring data analysts, data scientists, and data architects who can develop the applications for data exploration, discovery analytics, and real-time monitoring that organizations need if theyre to get full value from big data, said TDWI. When in doubt, hire and train data specialists to manage big data, not application specialists. Most BI/DW professionals are already cross trained in many data disciplines; cross train them more.
4. Collaboration is key to all data management, especially when managing big data
Due to big datas diversity, diverse technology teams will need to play coordinated roles, said TDWI. As a business asset, big data should be managed for broad access and leveraged by multiple business units and stakeholders. It takes a lot of collaboration to be sure everyone knows their role and has their needs met.
5. Beware the proliferation of siloed big data repositories
After all, the goal is to integrate big data into your well-integrated enterprise data and BI/DW environments, not proliferate 21st-century spreadmarts, said TDWI. Someone (probably not you) should decide whether big data platforms will be departmentally owned (as a lot of analytic applications are) or shared enterprise infrastructure supplied by central IT (similar to how IT provides SAN/NAS storage, servers, the network, etc.). Even if big data management begins in a silo (and you do have to start somewhere), make integration with other enterprise data management systems a second-phase priority. To make the integration happen, look for big data platforms (both open source and vendor built) that enable the integration points.
6. Define places for big data in architectures for data warehousing and enterprise data management
For example, an obvious place to start is to rethink the data staging area within your data warehouse, said TDWI. Thats where big data enters a data warehouse environment and where it is usually stored and processed. Consider moving your data staging area to a standalone big data management platform (on Hadoop, a columnar DBMS, or an appliance) outside the core data warehouse. Architecture can enable or inhibit critical next-generation BDM functions such as extreme scalability, complete views, unforeseen forms of analytics, big data as an enterprise asset, and real-time operation.
7. Reevaluate your current portfolio of data platforms and data management tools
For one thing, big data management is, more and more, a multi-platform solution (as are most data warehouse architectures), so you should expect to further diversify your software portfolio accordingly to accommodate big data fully, said TDWI. For another thing, survey data shows that the software types poised for the most brisk new adoption in the next three years are Hadoop (including HDFS, MapReduce, and miscellaneous Hadoop tools) and complex event processing (for streaming real-time big data). After those come NoSQL DBMSs, private clouds, and data virtualization/ federation. If youre like most organizations surveyed, all these have a potential use for your BDM solution, so you should educate yourself about them, then evaluate the ones that come closest to your BDM requirements.
8. Select data platforms that have special support for big data
There are many types to consider, including relational DBMSs, columnar DBMSs, data appliances, and other engineered systems, as well as Hadoop, NoSQL DBMSs, and the many other options, said TDWI. However, matching your BDM requirements to vendor products isnt always about the entire platform; it sometimes comes down to specific functions such as in-memory processing, in-database analytics, complex event processing, MapReduce, and robust interfaces to other data platforms.
9. Embrace all formats of big data, not just relational big data
Create a plan for your BDM maturation process, said TDWI. You have to start somewhere, so start with relational data, then move on to other structured data, such as log files that have a recurring record structure. Carefully select a beachhead for unstructured data, such as text analytics applied to call center text in support of sentiment analysis. Look for mission-critical data thats semi-structured, as in the XML documents your procurement department is exchanging with partnering companies. Then continue down the line of big data types.
10. Develop and apply a technology strategy for big data management
The strategy needs to spell out a wide range of road maps, standards, preferences, and stewardship guidelines, depending on what your organization needs and has a culture for, said TDWI. For example, you could lay out a road map for maturing from structured to semi-structured to unstructured data, as noted above. Since big data comes in many forms, you need to determine the preferred platforms and interfaces for capturing, storing, and processing each form. You should design a workflow for managing big data forms in their original state, plus processing that into other states for customer intelligence, data warehousing, discovery analytics, and so on. Big data isnt the storage problem it used to be, but you still have to plan capacity carefully, as well as related issues such as the acquisition and upgrade of data management platforms. Assuming you have an enterprise-scope data strategy and data architecture, you need to determine the many places diverse forms of big data should take in those. Finally, all the above must be supported by influential business sponsors (through stewardship and governance) so that big data management aligns with business goals for maximum business value.
More Information and Thank You
Special thanks to TDWI Research, which published this Top 10 list in its complete report, "Managing Big Data," by Philip Russom. Check out more Insurance Networking News slide shows here.