Many of the larger companies I have worked with or discussed with my colleagues have made some progress on their way to Enterprise Data Warehousing. Hardly anyone, however, has actually completed an implementation of the one single point of truth, integrating all relevant structured data and analysis in one place.
The challenges start with justifying the investment into a long-running, complex and expensive initiative in a dynamic, ever-changing business environment with quite a lot of work to complete before business sees the first benefits. Then the exponential growth of available data (structured, semi-structured and unstructured), the improved capabilities for advanced analysis on large amounts data, the demand for (near) real-time analysis have led to a shift of investments away from the classic Enterprise Data Warehouse.
Clearly, there is value in integrating data and in ensuring data quality. But does it pay off to have all data in one place?
Does it make sense use the same technology for all the different purposes? Current main stream DBMS certainly provide everything needed for an Enterprise Data Warehouse, but are they well suited to handle the increasing amount – and, more importantly, increasing variety of data generated by components of the Internet of Things (IoT) or coming from Social Media platforms?
The main early driver of data warehousing is largely gone: Modern operational systems have sufficient capacity to serve operational reporting and analysis needs. In fact, real-time reporting and analytics is gaining importance and, consequently most of these operational systems now provide their own analytic functionality.
Advances in computing power, network speed, the advent of in-memory systems, and other technology developments have enabled ever larger amounts of data to be processed and stored, enabling large Social Media platforms, or enabling Internet of Things applications. Business is now able to apply “Big Data” technologies to answer different questions on different, often very large sets of structured, semi-structured or unstructured data.
Adding the availability of standard, open interfaces to that, we now see a shift of paradigms in data warehousing and analytics, away from seeing everything as a data source towards seeing a collection of Analytic Services. An analytic service independently processes and stores information, it allows direct access to analytic functionality that allows reporting and analysis on that information, it exposes analytic functionality, data and meta data through APIs and it allows to extract detailed, aggregated or otherwise preprocessed data for use in other systems. Examples for what we consider Analytic Services include:
What independent analytic services don’t provide is cross-functional, cross-application analysis, KPIs embedded into a rich context of information, reliable historical or auditable information. And clearly, there are practical limitations in terms of network bandwidth, processing power or storage capacity that make it worthwhile to still maintain controlled redundancies simply to optimize analysis.
That is where data warehousing comes back in with the capability to combine analytic services to provide high quality, cross-functional, interdisciplinary analysis that really has the potential to add unique business value.
Distributed Data Warehousing
Distributed Data Warehousing is not a new idea and it complements traditional data warehousing by applying a structured approach to combining available analytic services, providing more agility and flexibility without really having to compromise data integrity. Integration points are on three levels:
This approach puts more emphasis and more urgency on the actual business value that can be generated by delivering data and analysis to the business users, utilizing analytic services that are already available and then gradually refining analytic capabilities, deepening the integration and optimizing execution over time. Distributed data warehousing can, of course, be complemented with other approach to more agile development, such as SCRUM/Agile project management, or with the broader approach of Gartner’s bi-modal BI.
Note that this not a discussion of “Cloud” vs. “On-Premise”. Conceptually, it doesn’t really matter where a specific analytic service is located – everything becomes “hybrid”.
How does BW/4HANA fit in?
With SAP BW/4HANA, SAP has released a much leaner variant of their Business Warehouse product line, deeply integrated with the SAP Hana in-memory database. If you look behind the curtains, most of the original BW logic has been moved into the database layer significantly improving BW performance. At the same time, deep integration into Hana allows BW to utilize more advanced functionality of not just the Hana database, but the Hana platform, enabling integration of predictive analytics, text mining, spatial analytics, etc.
This door is just opening – there is a lot for SAP left to do in terms of development and integration and there is a lot for clients left to explore.
Please refer to my “7 Reasons” Blog for more information about why S/4HANA does not replace BW (and why we still need Data Warehousing).