“Can I just have a query to download everything from the Enterprise Data Warehouse to Excel?” It’s the dreaded question which everyone who has been working in BI long enough has heard before. A business user has lost their patience and/or their trust in the Enterprise Data Warehousing solution and decided to matters into their own hands. Within no time, new Excel reports will be produced, with some good looking front sheets providing a “dashboard” or highly formatted report.
The more tech savvy users might even download some BI software and put the data in their own database. Team members admire the fancy graphs and the ease with which the solution can be adapted to meet changing business requirements.
The formal name for these BI applications developed and managed by non-BI professionals is called Shadow BI and although they initially appeal to the business, there is a real danger that in the longer term these applications do more damage than good to the enterprise as a whole. There can be good reasons for shadow BI but in my experience 9 out of 10 shadow BI solutions are implemented for the wrong reasons. In this blog I will both look at reasons why most organisations find themselves with so many Shadow BI solutions and I will provide some guidelines how to prevent the uncontrolled growth of Shadow BI solutions, and how to manage Shadow BI properly.
Friend or foe?
Before I go into the detail of ‘why’ and ‘how to manage things better’ I’ll answer the question in the title. In short, Shadow BI is not the enemy. It is the equivalent of a Silicon Valley start-up: An application might fail or live on and fulfil a niche function in the business without causing a storm.
Sometimes a solution becomes very successful and will make a significant contribution to the BI ecosphere. In any case, at some point in the lifetime of a shadow BI application some professional guidance is required. The business users deserve the best support BI can give them, either by showing better alternatives to shadow BI solutions or by adopting a solution so it can be scaled up to be rolled out to a wider audience.
But: There are dangers associated with Shadow BI. If shadow BI is not probably managed then the organisation is exposed to risks associated with things like poorly tested functionality, data integrity, adoption to future (technology and business) developments and last but not least, legal issues if the software is not used within the boundaries of the software license agreement.
So if properly managed, shadow BI is a friend. Now let’s have at why we have Shadow BI in the first place.
Reasons why business users build their own BI applications
I have no research data available to back this up, but based on my fifteen years of experience in BI I believe there is one main reason and several smaller causes contributing to Shadow BI. The core reason is: Changes to central BI solutions take too long to implement. In addition to this there can be other reasons such as requests which would require a major investment; the costs charged to the requesting business user’s department are deemed too high or, worst of all, business users simply don’t know what services can be delivered through central BI.
In many cases the business users feel they have no choice. They have tried to get what they want through central BI and failed that way, and are now left to their own devices. The reason why central BI fails is more often than not poor organisation (of the change processes) or poor implementation of the BI system. In other words, most business user requirements could be met within an acceptable time if there is an efficient governance procedure and if the BI is implemented in accordance with best practice guidelines.
Preventing unnecessary Shadow BI
Understanding the root cause of the problem means a solution is simple: Streamline the governance of central BI and use an adequate framework for BI application development. Below are some suggestions to do just that.
Streamline BI governance
BI applications are principally different from data entry applications. So why is it they are often governed by the same rules? It beats me. The only reasons I can think of are that this is historically grown and there is a reluctance to change agreed procedures – despite the fact that business users are turning their backs to central BI and the return on investment on BI applications is often poor.
The most common change requests for analytical application bring significant business value to a small group of people, are easy to implement (see next bullet point) and can easily be corrected if something goes wrong. On the other hand, changes to data entry systems are usually part of a more complex change in processes within an organisation, impacting several teams, various screens and batch processes and would cause a disaster if they go wrong when implemented.
When there is a significant risk to the business then it is perfectly legitimate that a request is reviewed by design authorities, change advisory boards and test teams prior to implementation. But when the risk is negligible and changes can easily be undone the approval process could be simplified and delegated to senior developers and retrospectively reviewed by design authorities.
Finally, it is helpful when business users, Bi teams and service partners agree what a ‘reasonable time’ is for implementing relatively simple changes to BI. It takes a bit of getting used to, but once there is a routine, it makes it so much easier to manage expectations within the business.
Use best practice guidelines for Enterprise Data Warehousing
Recent developments have greatly improved the options for adding virtual models on top of physical structures. In addition, new industry standards have made it a lot easier to include data from a wide variety of sources into the context of BI reporting. Making good use of these technologies within an enterprise data warehouse framework means the central BI solution becomes easily adaptable, without compromising the standards. It even allows business teams to pull some of their own data (locally held on spreadsheets or in files) within the context of BI reporting, something unheard of until very recently!
As Data Warehousing technology is changing rapidly it is of key importance for BI teams to keep their knowledge current. The required time for development can significantly be reduced by consistently applying best practice standards and by using the new features of the Data Warehousing platform.
Responsible Shadow BI
When you have streamlined your governance and you run your data warehouse according to current industry standards you will see an immediate reduction of shadow BI. Once in a while, there will be users asking you things you still can’t deliver. They may require applications you cannot support or datasets you cannot easily incorporate in the existing data model. In those cases, a business led pilot for trying out a solution not supported by central BI can be useful. Rather than leaving the business team to their own devices, the central BI team should still be involved. Perhaps the central BI team cannot provide the application, but some input might still be appreciated. And equally important, central BI will learn from the experience, and might see new opportunities when new technology becomes available. As long as there are clear rules of engagement, Shadow BI could be beneficial to both the business user community and to central BI.