Earlier this month, SAP announced the new version of its flagship data warehousing application: BW/4HANA. This version embodies the new era of enterprise data warehousing and heralds a new dawn.
It’s all about reduction!
The need for persisting the same data over and over again has gone: gone are the Cubes and the LSA physical storage layers as we knew them. The direction of development for SAP Business Warehouse (BW) has been clear since the introduction of BW on SAP HANA (BWoH). More and more support for working with virtual layers has gradually been introduced which has reduced the need for physical replication. This has made BW more adaptable and significantly decreased the costs of keeping BW up and running. SAP BW/4HANA will enable us to further reduce the need for data replication. There is a clear roadmap for an enterprise data warehouse without the need to replicate data: the ERP data stays in the ERP system (SAP S/4HANA) and is made available through a virtual enabler (Smart Data Access) for analytics reporting using SAP HANA CDS views.
History of innovations
Online Analytical Processing (OLAP) models were introduced to create physical data structures which are more suitable to answer complex questions compared to how data is organised in data entry systems. The downside of using these tables optimised for ‘analysis’ was that the data had to be copied from the source system to the reporting structures – usually whilst populating several ‘intermediate’ layers of data storage as well. Soon, enterprise data warehouse systems had interfaces from ERP systems, and other systems, which would take all night to complete. To make sure all data is up-to-date in the business critical Business Intelligence (BI) reports, the load processes have to be monitored overnight so issues can be resolved before the business gets online. This monitoring of load processes for BI still requires a lot of manpower in large enterprises.
Thanks to innovations such as columnar databases, in-memory databases and the ongoing increase of processor speed, complex questions can now be answered directly from running a query on source tables. There is no longer a need to reorganise the data into a different model just to do ‘analytics’. OLAP as a tool to create physical structures for data storage has certainly become obsolete. As duplicate storage of the same data in different models becomes unnecessary, there are fewer load processes to run on a daily basis. This in turn results in a significant reduction in operating costs of the Data Warehouse.
Data modelling in modern data warehouse applications
BI applications typically use data from a variety of sources. This data is not necessarily logically connected and transformations are required to present the data in a consistent and integrated way to analytical tools. Previously this integration was achieved during the Extract, Transformation and Load (ETL) processes. As data was being passed through from A to B, it was transformed to fit in an Enterprise Data Warehouse model. Even though data can now be exposed in a virtual way, the Enterprise Data Model remains a key enabler for enterprise wide analytics. The difference now is that it is no longer necessary to implement physical structures for all elements of the Data Model.
With the introduction of BW on HANA, SAP had already redefined its framework to cater for both virtual and physical data in a data model. SAP called this framework the LSA++, which is the successor of the Layered Scalable Architecture (LSA) which was used ‘back in the day’ to define the physical structures. The old LSA is well and truly dead in this age of virtualisation. The introduction of BW/4HANA will put the emphasis on virtual data access even more.
I believe we will see the first announcement of a fully virtual ECC integration in BW very soon. This is likely to be a greenfield implementation of S/4HANA and BW/4HANA with a modest set of analytical business functions delivered through BW.
Switching off the last ECC-to-BW load
It will be some time still until the first existing mature ECC / BW implementation will be transformed to a fully virtual Data Warehouse. Migrating to SAP HANA (ERP and BW) will be the easy part of the process. ECC to HANA migration and BW to BWoH migration have become common practice and the effort and time involved can be accurately planned.
That is only the starting point though. Converting all the logic currently embedded in multiple-layered load processes into virtual views will be an epic job. In addition, a big effort is required to convert now obsolete objects (Cubes, classic DSO’s) to the new objects. And let’s not forget the effort involved in replacing BEx, which is still widely used, as a front-end tool. These three factors (object conversion, load process conversion and report conversion) mean that the total effort to convert to virtual is huge for existing implementations.
How to prepare for the future
Regardless of what system you are currently on, you can already start to work towards a future migration to BW/4HANA. Make sure that you get rid of all 3.x objects, as there really is no excuse for still having those. They will not be allowed in BW/4HANA. Once you are on BWoH make sure you use Advanced Data Store Objects – so get rid of your Cubes and Classic DSO’s. Next you can start to virtualise specific sub-sets of your data. Below are some criteria which can be used to select the most suitable areas to start:
- Increase ability to respond to changing requirements
- Reduction the overnight load effort and storage requirements
- Ability for the business to analyse data in real time
- Effort required for converting the physical model and processes into virtual models
Needless to say that all new development should be based on a ‘virtual first’ approach – and physical data movement should only be considered if, for whatever reason, virtualisation cannot be applied.