Imagine you need to load ‘external data’ into your data warehouse. This external data is transactional data and it refers to ‘external’ master data codes. You will have to find a way to translate these codes into your SAP Master Data codes. The external master data comes from different 3rd parties, all maintaining their own master data. This means the link between external and internal codes has to be maintained as a BAU process. You want end users to maintain this link, so they can map new external codes to SAP Master Data codes, and they should be able to make sure all transactional data can be loaded successfully.
Is that too much to ask for? Not at all. It’s perfectly valid from a business process point of view and something clients ask about quite often. So, what does SAP offer to support this process? An easy-to-use, out-of-the-box solution? In many instances when I evaluate options with a client none of the SAP products offer a good fit and a bespoke solution needs to be built.
Extract Transform and Load (ETL) tools
Before we look at what options SAP has to offer, let’s consider the position of ‘ETL’ tools. Traditionally, ETL is regarded to be a technical tool, to enable a dataflow between different systems and platforms. The ETL tool is looked after by a technical team and business users don’t have to worry about the complicated mappings and transformations which happen ‘in the background’. The business users are happy as long as the data is available in time in the reporting system and as long as the data in the reporting systems reconciles with the data in the source systems.
Probably 99% of the data load processes can be automated with an ETL tool without interference from the business user. But if the 1% of the remaining dataloads require some form of manually intervention than it will cost you a disproporationed amount more to make this happen.
So what does SAP have to offer – and why is it not working?
Although I’ve looked into a number of options I don’t claim that my list of alternatives is complete. There might be others out there which I’m not aware of so if you can think of any of have any comments/suggestions be sure to let me know. The list below illustrates how for each product I looked at, why the rationale didn’t work for my scenario.
SAP Master Data Management (MDM): This comes with a user interface and provides ‘mapping’ functionality. If you use MDM as a strategic tool for master data management, then this might be a good tool for maintaining mapping data for external data load. If you’re not intending to use MDM as a strategic tool for master data management, then this seems might be a too costly and/or complex solution for loading that 1% of external data.
SAP BO Data Services: SAP’s ETL component on the NetWeaver platform. It does not come with a business user interface nor does it provide a staging area for mapping tables. You can build your own mapping tables (as ABAP dictionary tables) and an interface on top of it, for example in WebDynpro or Visual Composer. This seems a logical choice if you have decided on SAP BO DS as your strategic ETL tool.
SAP BO Financial Information Management (FIM): This tool provides a user interface on top of SAP BO Data Services. According to the product description, it does everything you would need to allow your business users to maintain mapping tables. One important shortcoming is that FIM does not support loading data into SAP BW. It can load directly into SAP BPC though.
SAP BRFplus: A SAP Business Rule Management product which has mapping functionality for data loads. The product is not as mainstream or prominent as the above mentioned products. I have never seen it in action. It might be worth looking at if you have requirements around business rule management.
Other SAP transactions:
Legacy System Migration Workbench (LSMW): Can be helpful for a one-off load of legacy data, although it certainly is not a state of the art tool for mapping data. The user interface is standard SAP GUI. It is not a recommended option for ‘Business As Usual’ dataload scenario’s.
Table clustering (SE54): Useful tool to maintain a set of (ABAP dictionary) mapping tables. The default user interface is the table maintenance functions in SAP GUI. Other user interfaces can be developed on top of this.
Winshuttle: Allows you to use Excel as a ‘mapping tool’ to load external data. I have never seen this in action but the demo’s on the website are quite slick. I have some considerations which would require further investigation (Can it run as a scheduled batch process, can it handle larger volumes of data, is it possible to implement more complicated business rules, for example based on lookups on other master data tables?).
How to make it work
So if you find that none of the out-of-the-box solutions work in your scenario, you’ll have to build your own solution. A combination of SAP BO Data Services and WebDynpro (for complicated front-end requirements) or Visual Composer (for simpler scenarios) have proven to work really well. The actual ‘ETL’ can be implemented in SAP BO Data Services. It is a reliable, easy-to-use tool. When you develop the front end in WebDynpro you have virtually unlimited flexibility. You do need to have ABAP WebDynpro skills in your project though, which might be hard to find. Visual Composer skills are easier to find. Even when you think of building your final solution in WebDynpro it pays off to start with a protoype in VC. You might even find that the VC solution is good enough to serve the purpose.
SAP is missing a trick here. I hope they recognise this and will provide an out-of-the-box solution in the not so distant future. Given that SAP BO Data Services is the strategic ETL tool, it would be a logical choice to change the existing User Interface (FIM) in a more generic toolset. Don’t be put off by the name: ‘Financial Information Management’ does not mean it can only handle financial data. If SAP would change the product so it can load in other targets than just BPC than we might have just the product we would be looking for. Maybe, to avoid confusion, they can change the name of FIM. That, for sure, should not be a problem for the guys in Waldorf.