Agile data warehousing with SAP BW/4HANA

9 May 2017

Jan van Ansem

Jan van Ansem

Principal consultant SAP Data Warehousing

Traditionalists believe that it isn’t possible to deliver value to the business through the agile methodology on enterprise data warehousing (EDW) projects. However, we may still be riding around on a horse and cart if we didn’t challenge ‘the accepted’. I’m not only going to challenge but also illustrate how moving to agile will increase user adoption and ultimately the success of EDW projects. 

Jan-Agile-EDW-Content-Picture.jpegAgile software development has quickly proven its value in delivering high quality applications that meet customer needs. When it comes to meeting the customers’ needs, EDW projects often fail to meet expectations, as research has illustrated time and time again. Interestingly, ‘Agile’ is not widely adapted as a project methodology for EDW projects.  

There is a lot of prejudice against the Agile methodology; perfectly captured here in ‘Fearful of Agile?’ In EDW, this prejudice is hidden behind the argument that it is not possible to present anything meaningful to the business in the early stages of a project, or in the short ‘sprint’ cycles, as building the data foundation simply takes too long.  

In the age of virtual data marts, fast databases and flexible reporting tools, this belief no longer holds true. With the assistance of SAP BW/4HANA and SAP Lumira I’ll prove my point...  

Display data quickly and worry about the modelling later 

Setting up an EDW takes time: 

  • Business rules and data definitions must be agreed throughout the organisation;
  • Improving the quality of data is cumbersome;
  • Designing truly actionable reports requires a thorough understanding of business objectives, processes and data.

An architected EDW aims to provide a ‘single version of the truth’ in a central repository which is used for all corporate reporting. One of the pitfalls of such a project is that reporting only starts when the data layer is in place. Waiting until this point means business users do not get to see anything meaningful until weeks, or even months, into the project. By then some decisions might have been made which could be costly to change.

With BW/4HANA, it is very easy to involve the business user at a much earlier stage. Recently I worked across a project where we took a data sample from an old system as flat files and loaded this into SAP HANA. The first samples were small and loaded in a couple of days, however, we soon scaled this up to data dumps containing entire tables with close to a billion rows. We built some crude virtual models on top of the data structures and before the week was over business users could start building reports in Analysis for Office and Lumira.

The models in BW/4HANA were incrementally enriched with formalised data structures, by associating the fields in the Calculation Views with InfoObjects – the building blocks of an architected EDW in BW/4HANA. Assumptions about master data relationships and data integrity could immediately be evaluated with business users. Once all the data elements were associated with InfoObjects, we started building the structures for physical data storage and loaded the dataset into BW/4HANA.

By following this approach, we not only made sure that business users were fully involved from the start, but we also had a very high success rate of data loading; almost completely avoiding the time-consuming task of reloading data.

Let the business do the reporting 

Business users involved in EDW projects are often quite data savvy. They have advanced Excel skills and often some experience in creating reports in an analytics tool. They rarely need much guidance to work with tools like Lumira or Analysis for Office.  

A great way of working with the business users on reporting is by organising weekly reporting workshops. This provides enormous value to business users: 

  • Weekly tasks can be commonly agreed to work on;
  • Questions can easily be answered;
  • Users can share their experiences e.g. what they did, what they couldn’t (easily) do and their learnings.  

In this environment, the elements which are difficult to do in the reporting tools will easily be identified. This is where a back-end solution in the EDW system might be required – usually in the form of a virtual or physical Data Mart.  

Put the business user in the driving seat of data acquisition

It is impossible to predict what sources of data business users will want to use in the future. Some of the new sources might only be temporarily of interest (for example, market research for a specific campaign). However, when a new source is identified, the business users want to see it incorporated in to their reports now.  

An easy way to give the business users control of what sources they use is through Lumira’s data blending function. This function allows you to build a report based on different sources (for example BW/4HANA and Excel) and merge or link different datasets. A downside is that the non-BW/4HANA dataset is only available in the context of Lumira reporting and not for other tools which you might have running on the EDW (planning, predictive, etc).

BW/4HANA offers two mechanisms which allow business users to pull non-EDW data into the context of the EDW (within certain restrictions). There is the ‘Workspaces’ feature in the BW application component and the ‘Agile Data Preparation’ function on the HANA platform. Both functions provide the business user more flexibility to use ad-hoc sources and gives IT the ability to set parameters for data acquisition and monitor the usage of resources of non-EDW data. When managed correctly, this is a great way to ensure that the business stays engaged with the EDW instead of walking away in frustration and finding their own solutions in shadow BI systems.  

Let’s embrace change and make EDW projects successful

It is time for the EDW world to improve its track record and the Agile project methodology will boost the success rate significantly. The traditional excuses for not using Agile are no longer valid.

BW/4HANA provides the ideal platform from which to work in an agile way: business users can interact with the data from day one, have flexible tools to do their own reporting and can be in charge of their own data acquisition, within the parameters set by IT. What’s not to like? Go forth and be agile!

View comments


  • Juergen Haupt - 14 Mar 2018, 18:08:17
    a nice, crisp contribution how the EDW approach must evolve - Thank you

Bluefin and SAP S/4HANA - welcome to the one horse race

We use cookies to provide you with the best browsing experience. By continuing to use this site you agree to our use of cookies.