Neil Wylie, SAP Business Intelligence Consultant, explains the common pitfalls for those seeking to do more with their Business Intelligence suite... and how to correct them.
“The art of the possible”. This is a phrase I’m sure many of you are sick of hearing. BI solutions are often described using this phrase, purporting to deliver incredible benefits (ignoring the phrase’s origin in Realpolitik – setting realistic goals). Whilst it’s true that pretty much any report is possible, given enough time and expertise, in practice setting lofty goals from the outset tends to result in failure.
Whilst evangelists preach from the rooftops (or more accurately, on LinkedIn) about the BI benefits of big data, machine learning and artificial intelligence, many organisations are struggling to implement effective solutions, encountering “data paralysis” and struggling with large volumes of low-quality data. They’re confident there’s value in there but getting it out appears a daunting task. It needn’t be.
We all have big aspirations and want to get there quickly. However, starting out with a grand vision rarely works. Phil Knight started Nike in his parents’ basement and sold shoes out of the back of his car before growing it into the globally-recognised company it is today.
Starting small allows you to minimise complexity and keep things simple, resulting in useful BI and delivering value as quickly as possible.
For many organisations, a data warehouse is the solution to their data governance needs. Importing and storing all data within the warehouse creates the “Single Source of Truth” that can then be used with no doubts as to reliability or quality. However, implementing this can be a long, expensive process, and if not done correctly can result in wasted time, effort, and money when the project is mothballed.
This is exacerbated by the fact that solution design often takes an end-to-end approach, working in the same direction as the data “flow”, from source to reporting. This is known as the waterfall approach and often makes building the reporting layer difficult.
For an exploratory BI project, it makes sense to start smaller. The reporting layer should be fed by a datamart, which itself would normally sit on a data warehouse. This simplifies the gathering of data required and shortens the dreaded extract, transform and load (ETL) process, which often eats up budget whilst providing little return.
Successful delivery of this project acts as a trial run to evidence the value of a BI strategy to executives, potentially opening the door to future BI projects, which may then justify the need for a data warehouse and a longer-term data strategy.
Fig 1. 'The reporting layer should be fed by a datamart'
Well-defined reporting requirements
At the start of any BI project, a requirements-gathering process should be performed with users to determine and document the key reporting requirements. These should be strictly adhered to with minimal change requests until the project is complete. This ensures that quantifiable value from BI is realised for stakeholders as soon as possible, avoiding data fatigue or paralysis. Any future project concepts that arise can be saved for later.
Build down from the Reporting layer
Once the reporting requirements are known, it is then possible to start the design and build process. Close attention should be paid to data required and how often it is accessed. Additionally, care should be taken to ensure processing is pushed down as much as possible. For example, it may make sense to split reporting requirements into several smaller, trusted data sources instead of trying to squeeze them into one large, complex model that is more likely to suffer from performance issues or have greater hardware requirements.
When a user creates a report, the first thing they choose is the trusted data on a particular subject matter. Other data sources covering other data subjects can easily be blended into the report as required, without having to create a large data source to cover all eventualities.
Not just an “IT project”
Frequently a BI project is seen as an “IT Project”, which is a mistake. It should instead be viewed as a business-wide project requiring collaboration from all persons it will affect - preventing misunderstandings and ensuring the solution meets the business’ needs. Regular interaction between different areas of the business fosters collaboration and is more likely to identify weaknesses in the solution early, instead of in the user-acceptance testing (UAT) phase.
Following these steps will usually yield better results than jumping straight in with a warehouse and trying to do everything at once. Some BI professionals will try to design solutions based on older approaches that simply aren’t effective any more. Here, we use modern methodologies and tools is more likely to result in successful BI solutions that add value to the business. Start small, define your requirements strictly and collaboratively, and you’ll be making smart, data-driven decisions in no time.
If you want to benchmark your own BI environment against likeminded organisations, check out our recent publication Adventures in Analytics