Amazon Web Services: disrupting software defined infrastructure

26 September 2016

Chris Kernaghan

Chris Kernaghan

Consultant

Infrastructure as Code (IaC) is the ability to define a software stack in a file and have the system build it without putting your hands on it.  Following some experiments with the Amazon Web Services (AWS) X1 instance, I wrote about how the X1 could disrupt the SAP infrastructure landscape.  Recently I've had the opportunity to really test it's potential.

The brief was clear: migrate Noble Drilling’s SAP BW system to SAP BW/4HANA.  It was a job that necessitated a quick turn around and required full flexibility and control. Where else would I run a project like that but in Amazon Web Services (AWS)? We had to set up a few servers as the source and the target systems for the migration.  Here’s how I accelerated the process of this build.

What’s the norm?

Traditionally infrastructure is built by hand according to a set of written instructions which define how the software is installed. The software stack is defined in specification and architecture documents, which are looked at once and, depending on who’s building it, are followed to varying degrees of accuracy.

When there are multiple servers to be built, then the use of multiple resources to parallelise the installs is a normal and reasonably valid course of action. The downside to parallelism is an increase in the risk of inconsistencies in the build. These inconsistencies often have a disproportionate effect on the stability of the servers. 

How much time does the ‘norm’ take?

Not only does building servers manually by multiple people increase the amount of inconsistency in the build,  building servers manually takes a lot of time. Last October I built a Hadoop Cluster of 100 nodes and we did some calculations on what it would take to build by hand, each node was estimated to take 8hrs to build by hand. This gives us 800 man hours, which would take 20 people 1 work week to complete, with no guarantee of consistency at the end.

What benefits does this disrupter, IaC, bring to the party?

Having worked in the life sciences industry for many years and being experienced in their compliance requirements, I am always looking for ways to improve consistency and reducing variances between environments. A few years ago I found a philosophy which matched my own regarding the interplay of operations, change and projects called DevOps. This philosophy plays on many levels, but it has a clear focus on creating a common language between people and teams often this is done using common tools. The use of IaC is a part of that philosophy and it is one of the most important paradigms for system administrators to get to grips with for the following reasons:

  1. Increase in build consistency - servers are built the same way every time the script is run.
  2. Decrease in risk brought about by human intervention - the script does not change behaviour according to events.
  3. Servers can easily be brought under change control - as servers are defined and built to definition then configuration is easy to establish. The build scripts should also be under change control.
  4. Hand-over between parties can be simplified - as the scripts define and log the process of the build, handover should be simplified.
  5. Build time is reduced - as the build is automatic and can be parallelised across all servers, the time taken to build is dramatically reduced.

An illustration of IaC

One of the easiest IaC examples to get started with is Cloudformation. Cloudformation is a tool that is part of AWS, with it you can define the specific properties of an AWS instance - for example which Virtual Private Cloud to (VPC) be placed into or which region to work out of, or the disk layout of the instance.

The basis of the Cloudformation process is the definition file which is in a JSON format and there are many examples (found here) on how to define your servers. 

SOURCE: AWS Blogs

One of the most powerful features of Cloudformation is the ability to install software on the instance once it is up and running - this is done in the UserData section. This enables the system builder to define what software to install on the system - in the case of some of my examples, I used the UserData section in my Vora build to install and configure my Hadoop Cluster.

I could use the Cloudformation UserData section to launch a scripted SWPM instance and use that to control an SAP System copy process, or simply use the Cloudformation template to start building SAP HANA instances. In the case of this build for Noble Drilling I used the UserData section of the source system build to download the export files from S3 and also to install MSSQL Server. This vastly shortened the build time as these activities took place overnight and did not tie up a resource unnecessarily. In this instance it was not consistency which drove me to use Cloudformation but a desire to use my resources more effectively in an extremely time critical situation. 

Whatever the use case, one thing for me is certain:  AWS Cloudformation improves my builds immeasurably not only from a quality and consistency point of view, but also from a quality of life point of view. Using these scripts means that I don’t have to slave over the build. I simply run the file and it does what I have asked it to (sometimes quite literally) and, just as importantly, it also runs consistently.

Whilst I have concentrated upon the AWS Cloudformation platform, it is important to remember that these functions are available in different platforms - for example, Hyper-V, VMWare, Xen. In addition to platforms there are also applications available such as Chef, Puppet, Ansible - known as configuration management tools.

SAP ecosystem’s progression towards DevOps

Since discovering DevOps several years ago, the infrastructure management capabilities have not progressed into the SAP field as quickly as I would have hoped. IaC is one of those technologies and processes which is easy to ignore, especially when builds are not an everyday event - although this is, I feel, the wrong attitude. I always do my best to define my artifacts in code, because then I can reuse them again and again to my advantage. AWS have created not only an amazing platform in Cloudformation, but they have also created amazing scripts which create working and usable instances quickly and effectively.

As infrastructure platform providers increase the level of abstraction available and further increase the capability of the programmatic tools available they increase the amount of work that developers can accomplish without the help of specific infrastructure skills. To some traditional infrastructure people this poses a direct threat to their positions as they haven't adapted to this new IaC paradigm, and as a result they have been replaced at agile startups by an architect and a developer who understand infrastructure. This shift is coming to enterprises, sooner than many would like to believe so it’s time to stop burying heads in the sand and get on board. 

 

View comments

Comments

Blog post currently doesn't have any comments.