Business and research activities involving "Advanced Analytics" are currently flowing against the global economic tide and there are 3 conspiring reasons for that:
1) An explosion of data
The exponential growth in the volume of data stored digitally (last estimated, by IDC, to be around 800 exabytes … and that was in 2009) is a major reason. This explosion has led to the advent - or recognition at least - of "Big Data" and analytics is a major way we make sense of all of that.
2) A growing awareness of the benefits
A consolidating driver is a greater recognition - within business and research - of the ways in which advanced methods can find actionable patterns in increasingly complex and voluminous data flows to create business benefit and demonstrable returns. Vendors, Analysts and Consultants have been articulating this for the last few years and business/government focused conferences like Predictive Analytics World (PAW) are helping spread the word - and raise expectations!
3) An increase in available technology
Technology vendors and open-source communities have been busy developing tools and applications. In recent times the commercial vendors have been mainly focusing on:
Making analytical software which is more accessible to the research and business user through more focussed user interfaces and better usability. KXEN, A relative newcomer set out to make advanced modelling more accessible to the business user.
Creating faster analytical engines like SAP HANA.
Building platforms. Often these are intended to make model development and deployment quicker and easier through automation. They are also about providing more automated decision making and enabling workflow/collaboration between the various user roles. SAS's Model Manager and IBM's Analytical Decision Management are two of the leading examples.
The open-source community, most notably in the form of R, has been providing a plethora of advanced statistical and mathematical algorithms for free. Commercial vendors like Revolution Analytics are harnessing the productivity of the R community to create tools that make that open-source more scalable and accessible to a broader (i.e. less technical) user base. Think Red Hat and Linux. And let's not forget that the most famous distributed platform for big data is the open-source Hadoop.
What is Advanced Analytics? Who does it?
For many, Advanced Analytics is any data analysis which goes beyond simple reporting. We can safely say that any analysis which looks at a number of variables at once, "multivariate analysis" in the statistical sense, and especially any analysis which results in a prediction/forecast in a predictive model is *definitely* Advanced Analytics.
Let's take a simple example to illustrate
Let's say our business objective is e to try and understand what is driving (and can therefore predict and prevent) churn among mobile customers a Telco. We might start by looking at some simple graphs and reports to explore differences in customer churn by geography, tariff and mobile usage e.g. outbound call levels.
Our overall churn rate is 10% per month and standard reports would show us the difference across the various dimensions (variables):
In this analysis we can see that the East region has the highest churn rate (across the regions) at 12%.
Advanced Analytic techniques can look at all of these variables at once and give us that multivariate view of how they combine to predict churn. A relatively simple Decision Tree could look at all the variables and show us.
This deeper analysis - and Decision Trees typically go much deeper than this - reveals that customers in the East who are on Tariff A have a churn rate of 30% and Low usage customers on Tariff C have a churn rate of 20%.
For many years the craftsmen of Advanced Analytics; Statisticians, Data Miners, Mathematical Modelers and more latterly Data Scientists, typically sat in the corner of an office. Alongside insights and reports the predictive models they constructed typically produced static data; predictions of which customers will churn, be a good/bad credit risk, etc. accompanied by a score.
Of course the fruits of some analytical efforts did inform successful strategies and some automated analytics including models, risk models in particular, did get deployed into operational systems and some in real time. Companies like Tesco, Harrahs (now Caesars) and Amazon, have embraced these approaches to greater effect. But these organisations are the exceptions to the rule. As such they have becom the "pin-ups" for thousands of PowerPoint presentations and a number of worthy books (including Tom Davenport's excellent "Competing on Analytics").
So what's new?
For all the reasons discussed we are seeing much more demand and execution of Advanced Analytics. This includes a desire to make insights really actionable by deploying them dynamically into systems more than has traditionally happened. This can include real time personalisation on web sites using tools like Omniture's Test&Target tool.
We're also at the start of an era of diversity in which we are finding new and interesting applications for analytics. The "pin-ups" are typically improving their CRM with analytics. The CRM analysts are predicting stuff like:
Which customers will churn?
What is the next best action for a customer? e.g. make them a specific offer
What is the predicted lifetime value of a new customer?
Increasingly though we are seeing examples of advanced, and often predictive, analytics in applications as varied as:
Anticipating failures in infrastructure like water pumps and pipes at South West Water in the UK
We're seeing analytics drive resource planning in Police Forces like Richmond VA
GlaxoSmithKline build predictive models to support patient recruitment and planning for Clinical Trials
Hewlett Packard have models to help prevent employee churn
However there is a snag. The demand is already starting to outstrip the supply of people who can provide the analytical support required. More business-focussed user interfaces help but a chunk of the demand needs to be met by with people skills. For example more technologists need to be able to integrate outputs from analyses into their operational architectures. The analysts don't normally cover this part. Good analytics is driven by the business objectives, e.g. reduce Churn, and a good analyst will typically be able to assimilate the subject matter often by engaging the business stakeholder. However this is a different skill to figuring out how to connect and update a model (or many models) in real time to an operational system.
Despite the risk of a shortfall in expertise these are exciting times in the world of analytics. While the general economic outlook isn't great you probably noticed a fourth reason that these approaches are on the rise; they can optimise a variety of business processes and in so doing reduce waste and cost.
John McConnell is the founder of Analytical People. He has been delivering Analytical Consulting Services across a broad range of business and research areas for over 20 years. The type of projects he is involved in range from ad-hoc analyses through to multi-user high-end, automated, analytical solutions delivery with Statistical, Data Mining and Predictive Analytics methods and technologies. He is also a regular conference speaker on the role of advanced analytics in the context of both business and research.
Through the '90s he worked for SPSS in a variety of international Professional Services delivery and management roles. Since 2000 John has been involved in a number of ventures in Europe and North America which have applied advanced analytical methodologies. In 2004 he co-founded Applied Insights which specialised in the application of Advanced Digital Analytics. Applied Insights was acquired by Foviance in November 2008 and Analytical People was launched.
Working across industries as diverse as news publishing, retail, financial services, consumer packaged goods, international shipping and within the public sector he has been able to develop and transfer - often cutting edge - learning and methodologies to create innovative solutions and measurable outcomes in a variety of fields. His recent modelling work with Reed Business Information (RBI), for example, resulted in RBI winning "Best use of data" and the "Best customer retention strategy" at the magazine publishing industry conference in 2008.
John has a BSc in Mathematics, Statistics and Operational Research from the University of Manchester Institute of Science and Technology, UK
Disclaimer: The opinions expressed here are the views of the writer and do not necessarily reflect the views and opinions of Bluefin Solutions Ltd.