Each year, we reflect upon the business intelligence and enterprise information management industry and offer a review of the noteworthy trends. Our review emanates from five sources: our customers, industry conferences, articles, social media, and software vendors. This year has proved to be an interesting one on many fronts. Here are our observations for 2013 and predicted trends for the remainder of 2014.
For purposes of this article, we have adopted the broader definition of “BI” which includes both data preparation (data integration, data quality, data warehousing, master data management, text and content analytics) and data usage (reporting, analytics, dashboards). This is in line with Howard Dresner’s umbrella definition of “BI” as "concepts and methods to improve business decision-making by using fact-based support systems."
1. Interest in Big Data and Hadoop Grew
We’ve all heard a lot about big data and Hadoop over the past several years. This is understandable because data volumes continue to grow, data formats continue to evolve, data is being collected at increasing rates, and technologies must innovate to support these changes a cost-effective way. Furthermore, these technologies have been very well-marketed; vendors have invested significantly in creating buzz.
"NewVantage Big Data Executive Survey," a 2013 study of 50 corporations and government agencies, revealed that 91 percent of executives indicated that they have a big data initiative planned or in progress. The surveyed executives indicated that 88 percent intended to spend $1M or more by 2016 on big data. Of those, 50 percent intended to spend $10M or more by 2016. So, big data is not just buzz - business adoption is underway.
However, in the same study, only 32 percent of the executives reported that they have a big data solution in production. Like most emerging technologies, the capabilities are still not well-understood by many organizations. They are also difficult to implement, partly because of the immaturity of the tools and partly because of the lack of experienced resources in the market. Furthermore, the technologies perform certain tasks very well but are typically not a wholesale replacement platform for existing technologies. One might say that big data is still in its infancy, and it would seem that incorporation of these concepts into mainstream data organizations will take longer than many analysts originally predicted, years ago.
A recent study by data management expert Richard Winter has highlighted the fact that fitting the right technology for the right purpose can have a significant impact on the total cost of ownership for a solution. In one situation, a five year comparison of TCO was conducted between a traditional RDBMS solution and a similarly functioning Hadoop solution for data refinement of sensor output from diesel engines. The study yielded a cost benefit of nearly 3:1, in favor of Hadoop. In a second study, however, a five-year comparison was made between the total cost of ownership of a traditional RDBMS solution and a similarly functioning Hadoop solution to support analytics of a large financial organization. The second study yielded a cost benefit of nearly 3:1, in favor of the RDBMS.
This clearly demonstrates the importance of matching the right technology to the business need. It also establishes the need for evaluation guidelines between Hadoop and RDBMS for any organization considering making a change. Criteria for selecting the proper solution , whether it be traditional RDBMS, Hadoop, or a hybrid of the two, should include:
- Understanding the data management and analytic requirements of the organization.
- Evaluating all costs of ownership, including hardware, software, development and support.
- Considering an architecture that leverages the strengths of both platforms.
One notable byproduct of the lack of clarity about which technology approach is most suitable for organizations has been a recent softening the BI/DW software market. Some organizations have chosen to delay or reduce their investment in BI technology until a clear path has been defined in the space.
Over the coming years, we expect that the role that each technology plays will begin to crystalize, and benchmarks and rules of thumb will be established. Until additional technologies emerge that combine Hadoop and RDBMS capabilities, we expect that a hybrid architecture will be most common.
2. Momentum Continuing with In Memory and Desktop Analytics
With the rise of 64-bit architectures and the ever-decreasing cost of memory, we recognize a continued emergence of in-memory analytics as well as in-memory databases.
The past several years have seen the rapid ascent of in-memory analytic technologies. Tools such as Tableau, IBM TM1, Spotfire and QlikView offer the promise of near-instantaneous analytic response. Such tools continue to enable business users to explore, evaluate and monitor their data with increasing efficiency.
Furthermore, as the cost of memory continues to decrease, in-memory databases such as SAP HANA and Oracle Exalytics have gained market share, and memory optimized databases, such as Kognitio and Teradata have optimized their platforms to efficiently leverage more data in memory. This results in significantly faster database operations for data residing in memory while allowing larger data sets to reside on disk. This hybrid capability allows administrators to tune their environments to their specific analytic workloads at a reduced cost versus in-memory databases.
In-memory analytics will continue to evolve and bridge the divide between business users and data within their enterprise. However, such tools do not address inherent challenges integrating, conforming and cleansing such data. Although we see the trend toward in-memory databases increasing, we also recognize that data volumes tend to outpace the growth of memory. Therefore, we don’t foresee all enterprise data being stored in memory anytime soon. Rather, we believe hybrid/temperature-based approaches (where frequently accessed data is stored in memory, while less critical data is stored on disk) will be more frequently adopted.
3. Master Data Management Becoming a Competitive Imperative
Master data management is more important than ever, with the proliferation of data sources both on-site and in the cloud and increasing importance of connecting with customers, fine-tuning products to meet their needs and managing suppliers and vendors. Customer data, in particular, has become more unwieldy than ever as a result of the increase in the number of channels that businesses offer. In addition, social media interactions, sensor data and increasing methods of communication have all driven an increase in the complexity to truly understand unique customers and connect with them in the most appropriate way. Entity resolution in an attempt to achieve the “golden record” has never been more important.
Enabling technologies have been present in the marketplace for years, and their capabilities have evolved to address this need. Companies such as Informatica, SAS and Stibo have solid MDM offerings, for instance. However, in spite of the maturity of MDM software and delivery methodologies, the ability to deliver successful MDM projects has still proved challenging.
Some of the reasons for this incongruity include:
- MDM requires cultural buy-in. Business sponsorship and an environment of collaboration between the business and IT is necessary. Yet most organizations consider MDM to be an IT problem. MDM technology, on its own, is not enough to solve the problem, and many IT-only/technology-driven MDM projects fail to achieve their stated goals.
- To succeed at MDM, one must have implemented at least a modicum of data governance - it is a prerequisite due to the political challenges of data ownership, survivorship and precedence. Many organizations, however, struggle with even the most basic data governance. As a result MDM never takes root, and if it does, struggles to continue.
- Justifying an MDM project is often a challenging process. Many organizations struggle to define and articulate the business case for an MDM project. For example, rationalizing products does not always lead to significant revenue increases or cost reductions. A proper business case is essential to justify such a large undertaking, and since the mission statement is not always clearly understood or articulated, many MDM projects go unfunded.
Managing project scope is another challenge. MDM projects often span myriad potential data sources and data elements, yet the true business value may be realized for a select few. This down-selection process may involve a breadth of stakeholders and it is not always a trivial process to achieve consensus. The tendency, therefore, is to include a broader range of elements than can be effectively managed. This produces a higher-than-ideal project risk.
All of the above challenges can be overcome. As extracting the value out of data emerges as a cornerstone objective for organizations, the essential visibility and desire to address data challenges will swell. Over the coming three years, we expect that more organizations than ever before will embark on MDM projects with the essential top-down business support, in conjunction with the necessary bottom-up IT expertise.
4. Greater Focus Placed on Operational BI
Over the past few years, we have observed an increased focus on the reporting and analysis of lower latency data. As corporate BI programs mature, this is a natural evolution. Using BI to support and enhance operations within a company yields considerable value. For instance, the customer/user experience may be enhanced, cross and up-sell opportunities maximized, and B2B messaging leveraged to streamline operations.
Companies that succeed with operational BI realize that it is a different class of system. Operational BI generally requires lower data latency, higher data selectivity and a larger amount of query concurrency than traditional analytic workloads. These factors often necessitate a different architecture than is traditional for data warehouses.
In addition, support for operational BI is often different. If a daily data load on a traditional data warehouse fails, it is often acceptable to address it within hours, not minutes. For operational BI, a 24x7 (or similar) support model is more often required, because load failures can immediately impact the user or vendor experience and, consequently, the organization’s bottom line.
We see the trend toward operational BI and lower latency analytics continuing as organizations broaden their focus from enterprise data warehousing to enterprise data management.
5. Continued Interest in Adopting Agile BI Practices
We’ve always been aware of the high cost and long lead times for implementing BI solutions. In addition, BI projects often have a difficult time obtaining proper levels of business participation and sometimes exclude the business entirely. Software-as-a service BI offerings and departmental solutions enable businesses to move forward without IT, putting even more pressure on BI programs to deliver results faster. Businesses are looking for new ways to implement BI and are finding that many agile practices (smaller focused iterations, daily scrum meetings, short sprints, embedded business representatives, prototyping and integrated testing) help accelerate BI projects and bolster communication between business users and IT. Certain technologies are also helping influence this shift. Data virtualization, for example, allows a “prototype, then build” capability and doesn’t require physicalizing all the data required for analysis.
However, agile was created for software development, not BI, and early adopters are learning that there are many differences. For example, the tools to automate software code testing are far more numerous and mature than what one might find for ETL mappings and data warehouses. Also, foundational activities such as defining a data architecture or creating a data model don’t always fit cleanly in the “sprint” model of Agile.
We expect to see BI practitioners continue to refine which agile principles are effective with BI and which ones don’t translate as well. We also expect to see a rise in enabling technologies, such as desktop analytic software, data virtualization and automated testing/data validation.
6. Failed BI Programs and Projects Remain a Formidable Challenge
Although the industry has learned and documented the reasons BI projects tend to fail, this knowledge hasn’t done much to stem the tide of failed projects. From our perspective, the overarching reason is that implementing successful BI projects is, simply put, difficult. Achieving success requires a balance of strong business involvement, thorough data analysis, scalable systems and data architectures, strong performing teams, comprehensive program and data governance, established standards and processes, excellent communication, and BI-focused project management. In addition, explicit success criteria must be defined and the solution must remain focused on achieving the defined goal.
From our perspective, we don’t necessarily see this trend changing unless organizations learn to:
- Institute and enforce enterprise data management practices.
- Ensure high levels of business involvement for BI projects.
- Spend more time marketing capabilities and the roadmap to the business.
- Thoroughly screen new hires and consulting partners.
- Design data architectures based on workload requirements
- Leverage a repeatable best-practice project methodology that resonates with the organization.
- Institute measurable, value-driven success criteria for every BI project.
All of the above factors may be addressed with focused business and IT leadership and a desire to collaborate to achieve common goals. Organizations that do so will reap the rewards of a truly powerful and dynamic BI environment.
7. BI in the Cloud Grows and Embraces “Hybrid IT”
The concept of “hybrid IT” or “hybrid Cloud” has recently emerged, where organizations manage some of their IT resources in house but leverage cloud-based solutions for IT needs. This hybrid approach allows an organization to centrally manage strategic capabilities such as data governance while leveraging the benefits of cloud computing where they produce the greatest benefit (e.g., Salesforce.com). By stepping away from the all-or-nothing approach, organizations are learning to evaluate and measure the benefits of BI in the cloud.
Likewise, BI in the cloud has continued to grow and evolve over the past several years, but progress has been gradual. According to Howard Dresner’s “2013 Wisdom of Crowds Cloud Business Intelligence Market Study,” more than 30 percent of respondents considered BI in the cloud to be critical or very important.
Vendors are developing technological innovations to address traditional deficiencies and hindrances such as data security, lack of robust ETL capabilities and performance challenges (from both hardware and bandwidth). They are also addressing concerns that “infrastructure-as-a-service” may be too complex for many BI organizations to consider.
To address these concerns, we have observed the emergence and evolution of a number of vendors in the SaaS space. Solutions such as MicroStrategy Cloud, Microsoft Azure, Informatica Cloud, Pervasive Cloud Integration, and GoodData are beginning to address these difficulties. These products show great promise, and over the coming years we expect them to continue to mature and validate the technical and fiscal viability of the space.
Over the next three years, we expect that the trend to push pieces of BI to the cloud (in a hybrid IT fashion) will continue . This trend will continue to increase industry understanding as to which cloud-based BI capabilities are most effective and how they are best integrated into an organization’s environment. It will also help BI cloud vendors to focus their efforts on the most salable solutions.
8. Multiple BI Tools are Here to Stay
The emergence and acceptance of data discovery tools, along with low-barrier entry to cloud offerings has increased, not reduced, the fragmentation of analytic and reporting tools in the enterprise.
We expect this trend to continue over the coming years. The specific analytic and reporting tools will increasingly become less of a deciding factor in a BI program’s success. This poses an interesting dilemma for organizations because it diverges from the concept of “single version of truth” that has been one of the textbook tenets of success throughout the years.
This trend places increasing importance on data disciplines such as: data modeling, data quality, MDM, metadata management, data integration, data masking and enterprise data management/governance. The broader the range of tools used to access an organization’s data, the more important it is to ensure that the data is presented in a consistent, well-documented, easy to understand and high quality fashion.
9. Data Security is More Important than Ever
Personally identifiable information (or PII) and sensitive financial and health data is being collected and stored by more organizations than ever. The rate at which such data has increased is astounding, yet only recently has attention been drawn to the impact of such data being collected and stored by companies and government agencies. Recent media attention to the inner workings of the NSA brought about increased awareness of the issues. Data breaches of Target, LivingSocial and Sony have further highlighted the innate challenge that we face to adequately secure data that may be used to harm customers and citizens. The security challenge is exacerbated by the emergence and proliferation of electronic medical record data. Not only the prevalence, but also the sensitivity of data is on the rise.
This increases the focus on data security as a practice within IT organizations -- specifically within BI organizations. BI programs must continue to evolve their competencies in areas such as: HIPAA standards, data security, test data management, data masking, enterprise data management, breach detection and, in some cases, analytic solutions that are specifically targeted at ensuring enterprise data security.
The primary obstacle is that data security falls into the category of low probability of occurrence with a significant impact. Organizations, therefore, struggle to justify expenditure from an ROI perspective. That said, we believe that this trend toward increased data security will be an evolution that will be fueled by high-profile security breaches along with increased government-mandated compliance requirements. This trend will evolve faster in industries such as health care and government where the impact of a data breach is greatest.
10. Growing Promise of DW/BI Automation
A variety of DW/BI automation vendors are increasing in prominence in the BI marketplace. Companies such as WhereScape, Kalido and BIReady promise to automate the data warehouse build process. The goal is to reduce costs while improving the agility (and ultimately user satisfaction) of BI projects.
DW/BI automation not only promises reduced development time, but also helps identify impact of changes from the underlying data sources or business rules. When data attributes change, there can be significant impact to the data model, database tables, ETL jobs, BI semantic layers and end-user reports. Understanding the scope of this impact can range from difficult to nearly impossible, depending on the size of the BI environment. Since these products leverage a common metadata backbone, they can quickly identify the impact of changes across the architecture. They can also enable versioning and change control.
These technologies, however, are still relatively new to the market, and the technical hurdle to support the broad range of business challenges, data architectures and technology mixes pose a significant challenge. We expect to see continued vendor consolidation in this space. We also expect to see an increased prevalence of these technologies in midmarket companies looking to reduce costs and increase performance of their BI programs.
Final Thoughts
This past year was certainly a year of change and maturation in the BI industry. We realize that a broad range of technologies, trends and capabilities emerged in 2013 and show promise for 2014.
Internet connected devices, for instance, will drive the value of data to consumers. Mobile devices will enable continuous data collection for health care and location-driven analytics. Sensors will drive finer grained collection of data across common processes. Long-term, this will elevate the awareness about the value of data to society. Data and analytics is becoming increasingly mainstream, and this promises to heighten the perceived value of BI across the globe.
This industry review is based on what we observed from real-world projects and challenges that our customers face. This doesn’t always align with what industry analysts and vendors choose to promote. Rather, our observations tend to be practical and focus on trends and technologies that are currently making a difference on the ground. We are curious to know if you agree with our opinions. Please let us know what you think: info@datasourceconsulting.com.
This story first appeared at
Dave Crolene is a partner at Datasource Consulting LLC. With 20+ years of experience in business intelligence and data integration, David has worked for two leading BI vendors, managed data warehouses for an electronics giant, and consulted across a range of industries and technologies.
Steve Dine is the managing partner and founder of Datasource Consulting LLC. He has extensive experience delivering and managing successful, highly scalable and maintainable data integration and business intelligence solutions. He is a faculty member at The Data Warehouse Institute and a judge for the Annual TDWI Best Practices Awards. Contact Steve via email at sdine@datasourceconsulting.com or on Twitter: @steve_dine or LinkedIn.
