Insights

Tony Harper

Querying 450 billion rows - revolution or just evolution?

The marketers want us to believe we are living through an all changing revolution driven by the power of in memory computing. All the talk of revolution is causing organizations to delay investment decisions while they work out what the future holds.

As it's not the first time marketers have heralded an all changing revolution, forgive me if I stick to my belief that this is simply another step in our information evolution and far from all changing. All we need to work out is which bits are still relevant and change the bits that aren’t. This is far easier and manageable than changing everything.

But first - my experience of information evolution so far...

I should explain that my parents ran a small data processing company. When I was 6'ish there were boxes of punch cards around the house and we made weekly trip into CDC in Birmingham to get them ‘processed’.

The first evolution in our house was a telex machine. This arrived when I was about 8, it had a QWERTY keyboard, communicated all across the world and stored messages on punched paper tape. We were amazed that the data from a whole box of cards could be stored on a few feet of punched tape and we didn’t have to drive it anywhere …….

What did we discard? Drawing diagonal lines on boxes of cards, so if dropped they could be reassembled back into the original order.

What did we keep? Sticking labels on the paper so we knew what it was.

The jump to magnetic storage in our house was the closest to a revolution I’ve seen mostly because the change for us happened very quickly and resulted in a business change, but again there were many disciplines that persisted. So in hindsight I could argue it was nothing more than an evolutionary step. In our house this evolution took the form of my first computer, a ZX81, our first 5 ¼ floppy disk on a Commodore 64 and a our first 10mb Winchester hard drive on an Apple IIe.

What did we discard? Huge amounts of paper storage, using external computing power.

What did we keep? Secure off site back-ups, fan fold print outs with hundreds of pages, batch processing runs etc….

The next evolution step I remember was an Amstrad PC that among other software ran Dbase. This DBMS removed the need to manage files as such. Freed from the detail of file management and the disciplines required to cram several thousand records into 64k of memory, as a teenager I was able to write a simple sales order processing application.

What did we discard? File handling, Linked lists, record pointers, end of record markers, etc. (clearly these are still used in data transfer)

What did we keep? Application flow design, functional program design, all 9’s records etc.

Skipping a few years, what do these early steps on the information evolution tell me?

With BWA and H.A.N.A SAP have developed some fantastic in memory technology, they have also shown us how it can address today's issues. The keynote at Teched talked about querying 450 billion rows of data, something that would be tricky (;-)) with today’s typical SAP BW hardware. But, as has been noted elsewhere, like this blog, there is more to Data Warehousing and BI than accelerating OLTP data.

The saying “garbage in, garbage out” still applies

In realistic terms, what are we likely to discard and what are we likely to keep in our typical heterogeneous source system (Sorry, SAP there are other systems out there) data warehouse landscape in the next 3-5 years?

What are we likely to discard? Star schema / extended star schema, aggregation tables and other modeling tricks for getting around the limitations of row based databases in the Reporting Layer of the LSA. While retaining the Reporting Later concept just accelerated i.e. in memory.

What are we likely to keep? From the Layered Scalable Architecture (LSA), almost everything else! We still need to get the information into the in memory environment, So, we still need to harmonize it with other systems data in our landscape. We still need to then apply business logic to it (although there is an argument that says this can now be done at run time) we will then need to surface the cleansed, harmonized, transformed data to the users.

There is no doubt that the technology available from SAP is getting more extensive, So over the next few weeks, I'll discuss the options, choices and decisions layer by layer of the LSA.

However, as one sage in the industry put it: “What SAP don’t get is, we’ve always been able to buy BusinessObjects / Sybase IQ etc…. We just didn’t.”

The same business decisions that led organisations to choose SAP BW with all its failings still apply, and BW’s key strengths, mean that the talk of it’s demise may just be a little early. But more on this later.

Enjoy these changing times but don’t believe the hype. With a little planning this can be managed so that investment and reputations are preserved.