History of SAP BW integration with competitors
When SAP built out their Business Warehouse product, they must have realised that customers would want to consume information from products outside of the SAP suite. So when they built their SAP BW 3.1C product in 2002, they implemented technology to enable customers to easily extract information out of BW.
SAP chose to implement the open standard that Microsoft wrote between 1997 and 1999 called MDX. MDX is a standard for "Multi-Dimensional eXpressions" - that is, the ability to model the kind of structures that a Data Warehouse represents – e.g. store, by product, by supplier, by time.
If you're interested in the technical details of MDX then Wikipedia has a much better explanation. In the end, MDX is a pretty good means of representing a question to a Data Warehouse. The problem is that it was designed for questions that have simple answers. So you can ask BW a question that causes billions of rows of information to be processed and so long as it returns a simple dataset, you're fine.
The problem that has plagued SAP since 2002 is that if you ask MDX a question which has a complex dataset to return, then the design of MDX means that the response will be pretty inefficient. Back in 1997 or even 2002 this didn't really matter because it was mostly used to return small and simple datasets. But in 2010, the MDX interface is used for some pretty serious data volumes. I have one customer who pulls 800 page reports off it with millions of records. And other customers who use 3rd party products like Business Objects, MicroStrategy and QlikView to pull millions or billions of rows out.
Wait a minute, didn't SAP buy Business Objects?
Ah yes. The uncomfortable truth is that SAP bought Business Objects and renamed it SAP BusinessObjects – and when it did so, I wonder if SAP had done the due diligence to realise that they bought a company which had fundamental integration problems with its own product line.
So what SAP did was to invest money into the MDX interface and the SAP BusinessObjects XI product range to improve the integration of MDX. But what really happened was they started polishing the proverbial and the MDX integration improved for various customer scenarios – but not consistently.
There is some supposition here but I had a customer in global escalation around this time for the integration between SAP BusinessObjects XI and NetWeaver BW 7.0. I believe SAP realised that the MDX interface was doomed for large data volumes and they built integration with SAP BusinessObjects Data Federator, which consumes large data volumes – and wrote an interface into SAP NetWeaver BW called the "Data Federator Façade".
Introducing BICS – BI Consumer Services
At this point, I think SAP gave up on MDX and wrote a new interface called the BI Consumer Services – or BICS for short – probably based on the DF Façade, that they wrote for the above customer. BICS is a Web Services model that fixes all the problems MDX has – performance, data volumes, authorizations and hierarchies. BICS was first seen in the enhanced version of NetWeaver BW 7.0 in 2009 – in Enhancement Pack 1 (EhP1) and the first product to support it was Xcelsius 2008 – SAP's dashboarding product that they inherited from Business Objects.
The BICS interface has since been used by its own Java component for reporting called BI Java, by Crystal Reports, and most recently by the "BI4" product range that has been released today. BI4 is the replacement for the SAP BusinessObjects XI 3.x product – affectionately known as most customers as BOXI or Webi.
What does BI4 bring to the NetWeaver BW customer?
More on BI4 some other time, but for the BW customer it brings strength of integration using the BICS interface. You can expect to get the same speed and quality of data transfer as you get in SAP BI Java or other interfaces. It probably isn't as strong still as the traditional BEx interface that some customers known and love, but it is much better than the old SAP MDX interface.
If you were suffering in SAP BusinessObjects XI around performance, hierarchies or security then BI4 on top of NetWeaver BW may well solve some of your problems and give you a much better, performant, user experience. It is available now and it's very easy to see if that helps in your scenarios.
But what are MicroStrategy and QlikView going to do?
I don't see that SAP have made BICS explicitly closed, but equally they don't appear to have explicitly worked with their competitors to improve their integration with NetWeaver BW, using the BICS interface. So the competitors are still using MDX – with all the challenges it brings – and SAP has moved on. Will it be investing the development effort that it did into MDX? It seems unlikely, although some products like BPC 7.5, still use MDX. We will see if the forthcoming BPC 10 uses BICS – which would be a clear indicator.
But, it is clearly MicroStrategy and QlikView's responsibility to build their own best-of-breed integration, and if they still want to sell to SAP customers then they need to get on the bandwagon and build out their own integration into BICS.
Is SAP deliberately trying to lock out innovation in competitors?
Actually I sense that this isn't the case. SAP had to fix the integration between Webi and BW and it has done so – but at the expense of using an industry standard like MDX. I can't specifically blame SAP for this because the intention appears to be to fix the problem first. Unfortunately this leaves competitors in a fix – and one that they will rush to sort out, if they have any sense.
And whilst SAP has built out best of breed integration between its products, it is behind in other areas – namely usability and onDevice integration – there are too many products, with too many names and too many use types. The real question is: can SAP sort these problems out before its competitors sort out integration? The race is on.