Head of Business Analytics & Technology
Is SAP deliberately locking out competitors with BI4?
23 Feb 2011
Business Intelligence (BI), Business Objects, Emerging Technologies, Integration, SAP NetWeaver Platform, SAP
History of SAP BW integration with competitors
When SAP built out their Business Warehouse product, they must have realised that customers would want to consume information from products outside of the SAP suite. So when they built their SAP BW 3.1C product in 2002, they implemented technology to enable customers to easily extract information out of BW.
SAP chose to implement the open standard that Microsoft wrote between 1997 and 1999 called MDX. MDX is a standard for "Multi-Dimensional eXpressions" - that is, the ability to model the kind of structures that a Data Warehouse represents – e.g. store, by product, by supplier, by time.
If you're interested in the technical details of MDX then Wikipedia has a much better explanation. In the end, MDX is a pretty good means of representing a question to a Data Warehouse. The problem is that it was designed for questions that have simple answers. So you can ask BW a question that causes billions of rows of information to be processed and so long as it returns a simple dataset, you're fine.
The problem that has plagued SAP since 2002 is that if you ask MDX a question which has a complex dataset to return, then the design of MDX means that the response will be pretty inefficient. Back in 1997 or even 2002 this didn't really matter because it was mostly used to return small and simple datasets. But in 2010, the MDX interface is used for some pretty serious data volumes. I have one customer who pulls 800 page reports off it with millions of records. And other customers who use 3rd party products like Business Objects, MicroStrategy and QlikView to pull millions or billions of rows out.
Wait a minute, didn't SAP buy Business Objects?
Ah yes. The uncomfortable truth is that SAP bought Business Objects and renamed it SAP BusinessObjects – and when it did so, I wonder if SAP had done the due diligence to realise that they bought a company which had fundamental integration problems with its own product line.
So what SAP did was to invest money into the MDX interface and the SAP BusinessObjects XI product range to improve the integration of MDX. But what really happened was they started polishing the proverbial and the MDX integration improved for various customer scenarios – but not consistently.
There is some supposition here but I had a customer in global escalation around this time for the integration between SAP BusinessObjects XI and NetWeaver BW 7.0. I believe SAP realised that the MDX interface was doomed for large data volumes and they built integration with SAP BusinessObjects Data Federator, which consumes large data volumes – and wrote an interface into SAP NetWeaver BW called the "Data Federator Façade".
Introducing BICS – BI Consumer Services
At this point, I think SAP gave up on MDX and wrote a new interface called the BI Consumer Services – or BICS for short – probably based on the DF Façade, that they wrote for the above customer. BICS is a Web Services model that fixes all the problems MDX has – performance, data volumes, authorizations and hierarchies. BICS was first seen in the enhanced version of NetWeaver BW 7.0 in 2009 – in Enhancement Pack 1 (EhP1) and the first product to support it was Xcelsius 2008 – SAP's dashboarding product that they inherited from Business Objects.
The BICS interface has since been used by its own Java component for reporting called BI Java, by Crystal Reports, and most recently by the "BI4" product range that has been released today. BI4 is the replacement for the SAP BusinessObjects XI 3.x product – affectionately known as most customers as BOXI or Webi.
What does BI4 bring to the NetWeaver BW customer?
More on BI4 some other time, but for the BW customer it brings strength of integration using the BICS interface. You can expect to get the same speed and quality of data transfer as you get in SAP BI Java or other interfaces. It probably isn't as strong still as the traditional BEx interface that some customers known and love, but it is much better than the old SAP MDX interface.
If you were suffering in SAP BusinessObjects XI around performance, hierarchies or security then BI4 on top of NetWeaver BW may well solve some of your problems and give you a much better, performant, user experience. It is available now and it's very easy to see if that helps in your scenarios.
But what are MicroStrategy and QlikView going to do?
I don't see that SAP have made BICS explicitly closed, but equally they don't appear to have explicitly worked with their competitors to improve their integration with NetWeaver BW, using the BICS interface. So the competitors are still using MDX – with all the challenges it brings – and SAP has moved on. Will it be investing the development effort that it did into MDX? It seems unlikely, although some products like BPC 7.5, still use MDX. We will see if the forthcoming BPC 10 uses BICS – which would be a clear indicator.
But, it is clearly MicroStrategy and QlikView's responsibility to build their own best-of-breed integration, and if they still want to sell to SAP customers then they need to get on the bandwagon and build out their own integration into BICS.
Is SAP deliberately trying to lock out innovation in competitors?
Actually I sense that this isn't the case. SAP had to fix the integration between Webi and BW and it has done so – but at the expense of using an industry standard like MDX. I can't specifically blame SAP for this because the intention appears to be to fix the problem first. Unfortunately this leaves competitors in a fix – and one that they will rush to sort out, if they have any sense.
And whilst SAP has built out best of breed integration between its products, it is behind in other areas – namely usability and onDevice integration – there are too many products, with too many names and too many use types. The real question is: can SAP sort these problems out before its competitors sort out integration? The race is on.
Jamie Hylton 23 May 2012
Is there any update on this issue since last year?
John Appleby 22 Aug 2011
Ian - agree with you on all the points.
DF is set to consolidate but I think they will still keep a standalone version - they have been good so far with SAP-BO offerings in that respect.
An 800-page report is a data dump and that's what it is used for - but try changing non-functional requirements to reduce performance with a new system!
Ian B 22 Aug 2011
Is SAP locking out competition with B4? Certainly, with the partial integration of Federator now meaning that instead of providing a database like interface via ODBC, the only federation option is to use a Universe, which of course means having to use BO reporting tools.
For now you can still use Federator R3 but my guess is that will be the last of that product as we know it. Either the database view gets incorporated into Data Integrator for "real time" functionality, or it will disappear altogether.
Oh, and 800 pages is not a "report" its an extract, regardless of how pretty it has been made. A report is for human consumption and nobody reads 800 pages of data.
Yogesh 19 Apr 2011
Great blog and comments. Currently advising a pharma customer to make a final decision on their Corporate BI Strategy - and boy the resistance coming from current BI team that is primarially based on Oracle-MicroStrategy (yesterday's best-of-breed BI products). No one seem to be willing to see the writting on the wall that with the roll out of SAP ECC (and BW 7, BO BI 4), they have no option but to migrate to BO. Let us see who wins in the end.
Ken 14 Apr 2011
Why did they remove MDX/XMLA access? It seems like they destroyed what was a standard open platform in the creation of a system for certain large volume tools.
Now I'm expected to use SQL to query a cube for a report in reporting services? If anything is not suited to analysis queries, it's SQL.
John Appleby 09 Mar 2011
Thanks for the considered and well thought out response - it's great to see SAP engage in the conversation. Also thanks for clarifying a couple of my minor historical errors. I surmise 2 things from this:
1) MDX, whilst great for small display volume analytics (000s of rows max) isn't suited for large scale data mining. DF Facade and BICS are, but are proprietary interfaces for various reasons. It is possible to do 3rd party integration using Data Federator - and actually we've done just this in some projects to get data out of BW fast. Of course you lose some features if you do this - hierarchies, authorisation variables and navigational attributes.
2) This means that SAP indeed has locked out 3rd party vendors from direct integration with BW for the purposes of large volume analytics (unless they implement through DF). This isn't to say that SAP has behaved badly in its approach, but the outcome is the same.
The end result is that players like MicroStrategy and Qlikview - who rely in some implementation scenarios on pulling large volumes of data out of BW In Memory, are in a bit of fix.
Now looking forward into the future, HANA has a direct SQL interface, which will cure some of these problems (I would think), but if you have HANA... why would you want MSTGY or Qlik?
Thomas Zurek 08 Mar 2011
Hi John, all,
thanks for writing this blog and initiating such a lively discussion. I like to shed some light into the thinking that has been behind the integration between Business Objects tools and clients with BW. My goal is to remove some of the speculation that has come up in the course of the discussion.
First of all, let me emphasise what has already been said above, namely that MDX is well suited for interactive analysis meaning the case that the result of an MDX query is displayed 1:1 in a suitable UI. The latter typically implies small result sets as an end user can "digest" only manageable portions of data in order to reasonably analyse. Tools that follow this paradigm normally see good performance as the logic is calculated down in a server which is architected for those calculations. Microsoft's Reporting Services or (native) Excel follow that rule and are equally suited to run on both, SAP BW and Microsoft Analysis Services. On the other hand, tools suffer from poor performance if they use MDX to retrieve significantly large amounts of data to only then calculate on top of that data. Some of the 3rd parties that have been mentioned above and that allegedly suffer from a competitive disadvantage simply follow such an approach and it is the latter that lets them fail rather than anything else.
Initially (i.e. around the time SAP acquired Business Objects), WebI followed that approach too. This was absolutely reasonable from the (previously independent) Business Objects' perspective as the semantic layer intended to provide semantics on top of arbitrary data sources. Here, "arbitration" implies "no assumption on capabilities of the underlying data source". This, in turn, implies that you need to apply semantics and the underlying calculations only once the data is in your own realm of control within the call stack. As soon as the SAP - Business Objects acquisition happened, the entire call stack (for a WebI query on top of BW) came under the control of one single company. Therefore, it was natural to redistribute the tasks and the processing with this new opportunity in order to do the best possible job for the end user.
There had been 2 options to do so: (a) provide a rather dumb (semantically poor) but volume enabled interface to BW that would allow WebI to keep its initial approach and/or (b) adjust WebI (and the underlying semantic layer processing) to the fact that BW already holds a fairly rich set of semantics within its repository (infoproviders, BEX queries, hierarchies etc.). Finally, both approaches were implemented which led to the "Data Federator - BW integration" for (a) and the BICS universes (or direct BICS access) with BI 4.0 (Aurora) for (b). Why both? Well, there is advantages and preferences at both ends, e.g. some customers like the modeling approach behind the relational universes and prefer (a), others have heavily invested in BW modeling, require the SAP specific semantics, want to continue to do so and prefer (b).
Now, why not opening up BW's "DF Façade" to 3rd parties, it has been asked. Well, the reason is simple: it has not public interface quality. For example, it cannot handle standard SQL albeit being close to SQL. The Data Federator compensates the missing SQL functionality (like HAVING clause processing or converting to a "select options" based WHERE clause). This is how it is currently architected. It means that the "DF Façade" is a technical layer that is not complying to any standard and also might be adjusted or changed in the future (which would not meet the expectation for an API). But besides, it also means that Data Federator can expose BW infoproviders via standard SQL and that means that 3rd parties can in fact query BW infoproviders via standard SQL. It requires Data Federator as it adds value (SQL compensation, SQL connectivity like JDBC, SQL parsing and translating it to a proprietary API, optionally also user management, security). You can always challenge this setup but it does the job and it does it well - see here for an example: http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/13952.
Furthermore, there have been questions on BICS. Firstly: it has not been invented for integrating Business Objects tools. Secondly, it is not based on the "DF Façade". Thirdly, it has surged in the course of NW 7.0 and BEX Web. Nowadays, Analysis Office uses a .NET version. BICS is not necessarily superior to MDX but rather different in its focus. SAP-internally, we have analysed and discussed a lot on this and it would fill many pages and many authors to summarise. However, I like to provide one single and fairly trivial but in performance terms unfortunate difference between the two: type casting! The MDX standard requires to format even key figure values as strings - e.g. meaning a cast float (ABAP) -> string (ABAP) - to be then sent over the network to the Java client - meaning string (ABAP) -> string (Java) - to be then reconverted into a Java type - e.g. string (Java) -> float (Java). The negative performance impact of such conversions is neglectable in case of small result sets (a few 1000 cells/values) but increases the more data is transferred. See the discussion above. BICS, in turn, uses type casts that do not deviate via an intermediate string result. It sounds trivial but the impact is not.
My comment has become quite long, I need to stop here as I'm sitting on a train that will arrive soon. I've intended to describe the sound technical motivation behind some of the decisions that have been discussed in this blog and have given room for interpretation into one or the other direction. Maybe I've succeeded here and there.
Tim Fisher 08 Mar 2011
Very interesting discussion.
I guess this is all part and parcel of SAP's new(ish) drive towards taking a more enterprise architecture driven view of the world. I think they want to paint this as a means for their customers to use SAP solutions in a more collaborative way, so building on their heritage business and potentially opening up the wider ecosystem to all their friendly new products, tools and usage types!
I'd also guess a rather useful benefit of this would also be the purchase and integration of new products into the SAP familiy becomes a more rapid and less painful process...but possibly I'm a cynic...
Mark Chapman 07 Mar 2011
Great summary. Wish this had been available just a few weeks earlier! Would have been very useful in startup conversations for my current project. I suspect the discussion around MDX's suitability for large data volumes/complex structures are VERY pertinent to my current Microstrategy discussions. We may see more on Thursday 10th if you're still up for a side trip after Boston
Mark. Aka @chapmanmt
Ali Qahtani 04 Mar 2011
Nice blog and great comments... I still think BICS has a long to go as it still has some limitations in terms of memory (that's why there is a safety limit on BI Java for the number of cells). I wish SAP invested more on BEx 3.x interface as it was flawless and they could've made it more open for SBO tools but I see it dying.
- Ali Q @alisqahtani
Ethan Jewett 02 Mar 2011
I guess I would frame this in a different way: Is there an expressive analytic expression language standard that provides good performance for a wide range of queries? Unfortunately the answer is no.
MDX is the closest thing we have (ODBO and XMLA are based on MDX), but it has not really been proven to provide good performance and there is not a consistent standard against which to implement. Even in Analysis Services (the defacto reference implementation), behavior and performance have changed significantly through multiple versions.
SQL (the basis for JDBC and many other interfaces) is another option, but it is not well-optimized to express analytical queries.
Because of this state of affairs, I can't fault SAP for looking for a third road and coming up with BICS. It would be nice if SAP would publish the specification for use by third parties, and it would be super-nice if SAP would start the long process of moving BICS through a standards body. But that takes a lot of work and may not be in SAP's best interest.
Lastly: Just to be clear on the BPC front - BPC NW does indeed use a lot of MDX on the front end and uses Netweaver BW's MDX engine for *some* computations. My understanding is the the development of BPC NW drove several major performance improvements in the BW MDX engine.
However ... because of performance issues with early versions BPC NW now internally converts most MDX statements to use native ABAP access to the underlying cube. This is closer to BICS than anything else, though technically it uses the RSDRI function modules to do the dirty work. BW also does not support MDX's new UPDATE statement, so writebacks all go through native ABAP interfaces.
I don't have any special knowledge of this, but if anything I would expect BPC NW to continue supporting MDX on the front end and also continue phasing it out on the back-end inasmuch as doing this improves performance.
Vitaliy @Sygyzmundovych 02 Mar 2011
Amyn, here is my understanding (with my typical disclaimer that for HANA and IMCE everything can change before they go to GA):
- BICS had been used only for BEx Webtemplates 7.0 (BI_Java-based), not for Webtemplates 3.x and not for BEx Analyzer 3.x or 7.0. BICS was used by Visual Composer as well.
- IMCE client libraries: JDBC driver, ODBC driver, DBSL for ABAP application server, client module for Python, ODBO provider for MDX (indeed "using partner product from Simba Technologies" as stated by SAP), SQLDBC library
- Afaik initially IMCE will only provide the MDX features required for supporting MS Excel
PS. I had been one of the reviewers of Larry Sackett's book before it was published :) Indeed I would recommend it, as Larry did great job writing this book.
-Vitaliy aka @Sygyzmundovych
John Appleby 28 Feb 2011
I think you're getting confused (unless I grossly misunderstand) - BICS came with BI Java but was remodelled with BOBJ in mind. It is different to the original BEx RFC interface.
Have you used MDX in anger? It's just not a suitable interface for large data volumes.
Yes, SAP is committed to MDX - it is an important integration layer for third party vendors and otherwise. Yes, HANA supports it - as it should - but that doesn't mean that it's suitable for large scale complex integration. It just means that HANA isn't a step backwards from BW in terms of its integration options.
Amyn Rajan 28 Feb 2011
Some good points John. However, you should remember that BICS has existed before SAP acquired Business Objects. BICS is the interface that BEx uses to connect to SAP BW. Therefore, there is nothing new about BICS. A few other points:
1. From my understanding, SAP is very committed to making MDX a good interface for connecting from third party applications. The MDX query language is absolutely necessary to connect from Microsoft Excel to SAP BW. In fact, MDX is so important, it is even being used as an interface for SAP's next generation HANA product: http://blogs.simba.com/simba_technologies_ceo_co/2010/12/sap-hana-now-available-supports-mdx-and-sql.html
2. If you want to understand the MDX interface better, SAP Press has a book about the MDX query language for SAP BW written by Larry Sackett
3. Thomas Zurek is the VP for R&D of SAP BW. His blog is a great source of information about what is happening within BW. I recommend you read: http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/13566%3Fpage%3Dlast%26x-maxdepth%3D0
John Appleby 25 Feb 2011
Thanks for the comment Vitaliy - well thought out.
Well I don't think SAP ever fixed MDX for SBO. They made it better, for sure, but every customer I have upgraded to BW 7.0 EhP1 has had some performance problems which had to be worked out through the UAT phase. SAP usually then produced some fixes that sorted out their scenarios - they are usually very responsive in this respect.
I thought BICS originally came in EhP1 for BW Java and it was equally thought of for SBO? Does it predate SBO connections? BICS had its teething troubles too in its early time.
Matt Hawkins 25 Feb 2011
Very thoughtful analysis, I look forward to your discussion on BI4.
Vitaliy @Sygyzmundovych 25 Feb 2011
John. It's a valid topic.
Last year I had been ranting about SAP closing the door for other BI products integrated with BW, but then realized that actually SAP is shooting its own leg. It is much easier for the customers to replace back-end DW system from BW to something else, than it is for them to migrate users to new BI platform, especially if they have BI corporate standard.
Paradox is that SAP still did great job for other BI vendors, while fixing BW's MDX for SBO. 3rd parties are not saint either. The quality of their BW connectors is equally important. I saw it with OBIEE, where it was OBIEE's BW connector fault not to pass filters into MDX statements, and therefore selecting massive amounts of data to transfer. MDX is not much worse than BW Analytic Engine's SQLs, which are similarly primitive SELECT - FROM - WHERE - GROUP BY.
Btw, BICS was developed for Visual Composer and BW's own Web 7.x run-time.
-Vitaliy aka @Sygyzmundovych