Economics of SAP HANA Adoption in the Cloud – one real customer example

It’s with a great honor that I write today, sharing another of my customer experiences on SAP HANA adoption.

And it’s with a great honor as I’ve just been named a “SAP HANA Distinguished Engineer” (you can read the nomination announcement at: http://www.hdespot.com/2016/06/24/antonio-freitas-sap-hana-distinguished-engineer-hacker-and-troublemaker/).

Feel truly humbled that the HDE community (http://www.hdespot.com) valued my contributions toward building and sharing knowledge on SAP HANA.

On the words of my friend Tomas Krojzl (http://www.hdespot.com/author/tomas_krojzlcz-ibm-com/), one of the greater expectations upon a SAP HANA Distinguished Engineer is to openly share knowledge on this area to help the product and the industry grow and advance.

So, to honor that principle, coming today with a very short note on a concrete customer example, that I was privileged to guide over the last 8 months from proposal to delivery.

Today my focus will be on the “economics of SAP HANA adoption in the Cloud”.

Without identifying the company these values relate to, I’ll provide relative values on this company’s costs when operating their SAP Systems on-prem with a legacy Risk/AnyDB config, in the Cloud with Windows/AnyDB, and finally in the cloud with Linux/HANA.

Setting the scene

I know that one of the biggest concerns of most companies evaluating adopting SAP HANA, is the economic impact in terms of infrastructure and managed services.

I’ve been writing about “SAP HANA openness” for 3 years now, and the key implication of the increased openness of SAP HANA has been allowing lower entry costs and lower operating costs of this infrastructure.

Although in the past most of my focus was on on-prem deployments of SAP HANA (look for example the following whitepapers I’ve contributed to: http://www.emc.com/collateral/emc-perspective/h14459-making-sap-hana-mainstream-datacenter-part1.pdf and http://www.emc.com/collateral/white-papers/making-sap-hana-mainstream-datacenter-part-2.pdf), today I have to agree with a lot of the reasoning I’ve seen from John Appleby (http://www.hdespot.com/author/john-applebybluefinsolutions-com/)  where he said that any customer today considering starting an SAP application implementation, if he did not chose to do it on HANA and on the CLOUD, he was not providing a good service to his company.

I struggled with that a bit at the beginning, as I had an idea of cloud built at the image of the first cloud providers, where there were a lot of concerns in regards to security and compliance (especially when thinking on non-US companies putting data in US datacenters), but also around service assurance (Application level SLAs for Performance and Availability), which led in many cases to see their operational costs go up due to the increased complexity of managing the cloud provider’s flaws.

You all know that I’m passionate about what I do, and that I struggle to do a job or work for a company I don’t believe in.

Well, having come to Virtustream I’ve found a company that merges the best of Cloud (agility, cost savings, flexibility) with the best of IT Outsourcing (security and control, managed services, application level SLAs).

So, over the last 9 months I’ve been, on one side “learning Virtustream”, and on the other presenting Virtustream to customers and then guiding them to migrate their systems to the Virtustream Enterprise Cloud.

It is necessary some months on the job for you to start to see results of your own work, and it with great gratitude that I’m seeing the first customers I’ve consulted with, going into production on the Virtustream cloud.

And this provides me a unique opportunity to do a “before and after” analysis, and that is what I’m coming to share here now.

I hope to get the specific customer I’m writing here about to become a public reference soon, but until that happens, here is a bit of their story and also some insights on the numbers behind the story.

               Looking for “The Right Cloud”

First of all I have to say that this is a very innovative company, as their operating model (being this company operating in a very traditional manufacturing area), already included a consumption based billing to their end customers.

So, for them, paying only for what you actually consume, was already embedded on their DNA.

15 years ago they were running their SAP systems on-prem, and being a very agile company with a very light IT department supporting their operations all the countries they operate (way over 50), it was clear for them that the internal IT needed to focus on activities that truly added relevant value to the business.

Then this led them to sign an infrastructure outsourcing contract with one of the big manufacturers, running all their SAP Systems on a Risk/AnyDB platform.

After 10 years with this provider, they felt that they kept lagging on the business expectations, in terms of response time, cost of change, and ability to experiment and innovate.

On their words “every single new request was painful to get”. It “took weeks to be able to schedule a meeting as the provider always come with large team to each meeting” and the simple factor of reconciling agendas was a nightmare. Then the costs associated with small changes plus the lead time necessary for any change to happen, sometimes just killed the business case, as the business units would have lost the window of opportunity.

So, coming to the end of this 10 years outsourcing contract, they decided to go back to the market, and hearing all these new things about cloud, asked for cloud offers from all the known players in the market.

Their conclusion was disappointing. The model all providers were presenting, resembled a lot the experience they already had with that IT Outsourcing over the last 10 years. Have to mention here that this being an European organization, hosting their most critical production systems in a US datacenter was out of the question, and that the requirements on security and compliance were very strict.

So, instead of confirming with the “least bad” option among the responses they got to their RFP, they hired the consulting services of one of the market analyst firms, which looking at their magic quadrants immediately pointed them out to a player they haven’t invited to the tender.

That player was Virtustream, and I was assigned to support that opportunity.

               Defining the right cloud: First impressions

Having this customer today already in production on the Virtustream cloud, gives me a chance to go back and ask them: when did you made up your mind that Virtustream was the right option? What made you decide?

And this customer has been kind enough to be open and transparent about this, as in his own words “we have not contracted a supplier, we have brought in a strategic business partner to our business”, and so open and transparent communication is a fundamental baseline for the collaboration their company expects from Virtustream.

First of all, would like to share the goals they have documented on the RFP they sent to all the respondents:

• Improvements in the agility and flexibility of the organization in a changing market
• Improved quality of service and reliability of business processes
• Improved relationship between business and IT
• Scalability of operations
• Significant cost savings through true consumption based billing
• Have a fully managed service and have internal IT as service managers, not IT engineering and operations

And their first impressions were like this:

• We wanted agility
• We wanted solutions out of the box
• We wanted solutions specifically designed to our problems
• We wanted fast turnaround times
• We wanted a knowledgeable business partner that was able to advise us on the best way for our company
• We wanted a simple consumption model
• We wanted to pay just for what we actually use, and not for what has been allocated to us
• We don’t want to over-size, but rather to right-size and grow as we go
• We wanted to have control and visibility on where our data resides
• We wanted s strategic innovative partner with capacity to support our own innovative ideas

And this is what they felt none of the providers that had responded could offer, and that they saw all through their interaction with Virtustream.

A key aspect here was that instead of responding to the RFP they had written, which of course was a reflection of the IT outsourcing they had been on for the last 10 years, Virtustream challenged most of those requirements, and instead proposed them a different approach:

• Some of the key benefits of cloud is agility and cost saving
• The only way to get these is to have “brutal standardization” both at the infrastructure level and in the processes
• So, let’s look at what we have to offer out of the pocket in our standard catalog, and then evaluate the real business requirements for the applications to be hosted on the cloud
• Whenever is acceptable for the business the standard SLAs offered by the standard offerings of the provider, let’s adjust the requirements on the RFP, so that instead we can just “pick from the menu” and build an order specific for you
• If there are some very unique business requirements that our standard offer and SLAs cannot cover, we’ll build a custom solution to you

Here I have to say that having the CIO in the discussion, and being a very senior person with a very deep understanding of the business, advised by a team of direct reports also very knowledgeable and pragmatic, and enabled to make the decision to adapt all requirements to the standards offered.

One of the biggest barriers I see for many customers to have a positive adoption of cloud for their enterprise mission critical SAP Systems like SAP ERP, is that they design RFPs that do not ask for what they truly need! I understand this, as one cannot ask for something they do not know. So, the reality is that most RFPs I have seen to host production mission critical systems on the cloud, either ask for a cloud model based on the first generation of cloud providers (read my blog on this at: http://sapinfrastructureintegration.blogspot.com/2016/05/sap-hana-adoption-in-cloud-made-simple.html) that are not suitable for this purpose, or have been written for an IT outsourcing model which has proven to under-deliver over many years.

So I have to say that this customer was truly innovative, by being available to open a blank sheet of paper, learn how Virtustream operates and delivers services, and then rebuild their requirements from a different starting point. This was a key success factor for the levels of satisfaction this customer is observing now.

The outcome was the presentation of a financial offer 1 week after the first meeting, signing a contract in 3 months after the first meeting, and having all systems migrated to the cloud less than  3 months after contract signature (migration included multiple system landscapes, being one of them a SAP ERP 6, and the other a SAP BW, core to the company's business).

And all of this with a very smooth migration process, and better performance, visibility and control on the cloud than they ever had with their IT outsourcer.

So, the first impressions on the engagement, and the way the whole process was handled, showed this very innovative customer that this company was the right choice for them.

Joking a bit, they said that “mentally” they decided to buy 1 week after the first meeting, and the rest of the time was just to confirm and validate that decision as what we were delivering was completely different from anything else they had seen so far, so they needed to learn more about us to be confident of their purchasing decision.

Also they mention that one of their bigger concerns was the fear of losing control, or not being able to get proper support and attention as the “the provider was all on the cloud”, including managed services, and so through this process only happened a hand full of face to face meetings.

He recognized that having not only the systems “on the cloud”, but also the “managed services on the cloud”, enabled them to get access to the best professionals in the world regardless of where they are located, and that the responsiveness and “presence” of the team throughout the complex migration project (this was the first OS/DB migration this customer had gone through) was better than anything they have experienced with their outsourcers over the last 10 years.

We had the pleasure of being invited by the CEO to a discussion, where he explained the importance of these systems for their operations, and that without them they would be completely stopped. This conversation was critical to build trust and a common understanding of the goals, as this was a strategic decision for that company, that could not be taken lightly.

               What about the economics?

So far it is all very nice, but what about the economics?

One of the key decision factors for this customer was also to get “significant cost savings” on IT infrastructure and managed services.

And here is where the story becomes interesting.

This customer had the strategic goal of implementing S/4, as they do understand the benefits of HANA in things like real-time treasury management, as they need to manage the cash balances across countries, and having to move money between countries with bad market conditions, can dramatically erode the company’s profitability.

Also they buy commodities as raw materials for their production processes, also a situation where operating in real time can make the difference between profit and losses.

Then the question was: how much will it cost me to migrate to HANA, consume an infrastructure for HANA, and operate HANA?

There was a whole lot of unknown, and a high risk perception, as their outsourcer had tried to upgrade them to EHP7 a number of times to get them ready for HANA, and it was a slow and painful process. It took them more than 6 months just to get a stable development system.

So here are the numbers:

1. Just by migrating from on-prem Risk/AnyDB to the cloud with Windows/AnyDB, this company saved about 40% in their monthly bill!!! This includes all infrastructure services (compute, network, storage, security, service desk) and managed services (OS, DB and SAP Basis management, including ITSM process handling and overall service management).
2. This project was executed in 2 phases: phase 1 was just migration from Risk/AnyDB to Windows/AnyDB; phase 2 was upgrade with DMO from Windows/AnyDB to LINUX/HANA.
3. The 1st phase of the migration project will have a payback time of less than 6 months (meaning, 6 months of savings on the monthly charges paid the migration project);
4. The project for the technical upgrade with DMO to migrate to HANA was also funded by 6 months of savings!
5. The total monthly costs of running on the cloud with Linux/HANA were 17% higher than running with Windows/AnyDB. Which means that still with some increased costs against running on the cloud on LINUX/HANA vs Windows/AnyDB, it was still a lot lower than the costs of the IT Outsourcing contract they had for the last 10 years (we are talking more than 20% savings every month).
6. This customer evaluates that the business benefits of adopting HANA for their business, will complete overshadow the costs of the infrastructure and managed services, by multiple dozens multipliers.

Have to say that some of these numbers are “projected savings”, as with the Virtustream patented microVM consumption model, we expect the real numbers to look even better.

When talking about the difference in cost between running those systems on the cloud with Windows/AnyDB versus LINUX/HANA, we wanted also to understand where the difference came from, and so here is a bit more detail:

• Compute: costs more 116% on HANA vs AnyDB, and represents 29% of the total monthly HANA costs;
• It is a fact that SAP Business Suit applications on HANA consumes more CPU and Memory than on AnyDB. This is a factor of much of the application code and data structures not having been yet optimized for HANA. So, the benefits we see in terms of data footprint reduction when implementing Simple Finance, isn't yet available for many of the other business areas (to understand more about this, read: http://sapinfrastructureintegration.blogspot.com.es/2015/04/will-sap-s4-hana-really-require-less.html).
• Note that this simulation does not account for the impact that Virtustream’s patented microVM based consumption model will have, which we expect to make these numbers look better, as the customer can for example shutdown non-productive systems through the night or weekends when they are not used, and they will not pay this item for those period. These optimizations have not been factored in. 
• Again here, the fact that we have more OS images due to the separation of the HANA DB and APP Servers on different OS instances also has some influence.
• OS Licensing: costs more 1305% on LINUX vs Windows, an represents 2% of the total monthly HANA costs;
• This is due to 1 factors: 1) the fact that Linux support costs a lot more than Windows subscription licensing, and 2) the fact that when running on HANA we have more OS images as we do not run any systems as “central servers” (DB+ASCS+PAS on the same server). The good thing here is that the weight of this cost on the overall costs is very low.
• Backup: costs less 70% on LINUX/HANA vs Windows/AnyDB, and represents 2% of the total monthly HANA costs;
• I believe here factors like HANA not having indexes, and the compression play a big part. We’ll need more time in production to be able to take more supported conclusions.
• Storage: costs more 3% on LINUX/HANA vs Windows/AnyDB, and represents 9% of the total monthly HANA costs;
• We believe the storage costs could be lower, but as customers get concerned with SAP’s recommendations for x-time multipliers on Memory to disk (to know more about this, read http://scn.sap.com/docs/DOC-62595), they end-up over-provisioning capacity they will never use. I expect that as I get more customers “from proposal to production”, with more experience to provide stronger advice to customers to start with less storage, as in the cloud, they can grow as they go.
• Managed Services (OS, DB, SAP Basis): costs more 3% on LINUX/HANA vs Windows/AnyDB, and represents 37% of the total monthly HANA costs.
• The difference is relatively small and is due to the fact that the customer when running Windows/AnyDB had some central servers, so less OS images to manage, and we are implementing all systems separating the SAP HANA DB into a different server from the ABAP Application Servers.

NOTE: these values are specific to this customer scenario, and are dependent on the mix of systems in the landscape, the size of those systems, etc. So these numbers intend only to show – on this specific example – where the higher cost of operating HANA actually came from. Also, on these vales are included all software and services necessary to operate a full infrastructure service, and not only the respective hardware components. On your specific reality these numbers may be different depending on your source system, target system, landscape size, data volume size, etc.

               Conclusions and looking into the future

So, even if there were no business benefits from running their business on HANA, the savings of migrating to HANA on the cloud would have paid the migration project in less than a year.

When factoring in the business benefits of HANA, there is simply no argument against.

And to this you have to add the strategic and operational benefits brought by the adoption of the
Virtustream cloud:

• Better performance
• Better responsiveness of the services team
• Lower cost to experiment, by being able to spin up a sandbox system to test new functionalities for a couple of days and then shut it down, only paying for the resources actually consumed on that period
• Better cost predictability and transparency

It is indeed a great pleasure to have seen this project develop, watch this customer unleash creativity, and getting their IT team already focused on the next step, and how else can they help their business be more efficient, more innovative and more competitive.

And the next step on this customer’s roadmap, after a small stabilization period for the systems we've just migrated, is to start evaluating IoT scenarios, as they see a lot of potential in streaming data from their manufacturing machines and cross analyze it in real time against their orders data, inventory data, commodity costs, etc.

So, I’m already thinking on how to enable them to start experimenting with HANA Vora, and build a lab environment to enable them to bring that innovation to their business faster… Stay tuned for news on this soon!

So, this is it. The conclusion is simple: running SAP HANA in the Cloud is a lot cheaper that running SAP applications on Risk/AnyDB. Running your SAP applications on HANA in the Virtustream cloud will also provide you a way better service than the one outsourcers have been providing. So, why wait? Get started on your SAP HANA adoption plans!

Would love to hear your own experiences, your challenges in building a business case for SAP HANA adoption, and know how the numbers worked on your specific scenario.

Looking forward to your feedback!

SAP HANA Scale-up vs Scale-out learnings for SoH and S/4

It’s been quite some time since I’ve blogged, but as you all know, changing jobs always requires some adjustment time, and the traction that the Virtustream value proposition is observing in the European market, has simply made my agenda overflow! The good thing about is that my learning has only grown due to the massive number of customer engagements I’ve had just in a 2 months period.

And on these interactions, the question on whether the best option when deploying applications based on SAP HANA is to scale-up or scale-out keeps popping up much frequently than I would ever expect. Despite SAP’s recommendations in this regards, as the reality sometimes outpaces current rules, this becomes a key discussion point when deciding whether to move forward with SAP HANA adoption plans or not.

In this blog post I’m coming to share my most recent learning and experiences on this topic, as well as my reasoning behind the pros and cons of each of the scenarios, while sustaining my arguments with the currently available information from SAP.

My goal here is not to provide any clear guidance, but rather to show you my reasoning process and arguments, so that you can have your own discussion in a more informed way, and leveraging knowledge on what others are going through.

In the end, each customer scenario is different, and so there is no “one rule that fits all”.

               Setting the Scene

Time goes by, and this topic keeps popping up in many of the customer engagements I participate in. I’ve already written about this long ago, but the fact that I keep getting faced with absurd situations got me going to write about it more extensively.

And the question is: “For my specific scenario, should I implement SAP HANA in a scale-up or scale-out model”?

On my experiences, there were multiple angles to this question, and depending on the angle, the response may be different.

One of the aspects that disturbs me the most is how so many “technical architects” I come across ignore the variables in the picture above, and come up with solutions, that when we factor in the increased openness SAP HANA is going through, and the reasonable expectations on the expansion of available options over a 3 year period, still come with designs that on my mind simply do not make any business sense. After all IT architecture is not just a “techies” discipline! Being technically grounded, the IT architects should be key pieces in driving business and IT alignment, and so making choices and imposing scenarios that completely ignore the business side of IT, simply leads to bad solutions.

To get background on SAP HANA scalability, you can read SAP’s scalability whitepaper at: http://scn.sap.com/docs/DOC-60340

Will not repeat that here, so if you are new to the HANA scalability topic, I would recommend you to read this whitepaper before reading this blog post.

While I’ll focus mostly on the technical aspects of this discussion, on the back of my head is always: on one side the fact that IT has as primary goal to serve and support business needs; and on the other facts like the dramatic increase in RAM capacity on servers just over the last two years with the evolution of Intel CPUs from Westmere, to Ivy-Bridge and now Haswell, which has come to enable organizations today to dramatically simplify architectures thanks to the increased capacity these latest generations of technology have come to enable.

To get started, let remember current SAP recommendations in regards to SAP HANA scale-up vs scale-out:

• Scale-up first as much as you can before considering scale-out;
• Scaling out is only generally available for analytic workloads (BW on HANA or DataMart scenarios);
• For transactional workloads, like SAP Business Suite on HANA (ERP, CRM, SRM, SCM, etc), today only scale-up is generally available.

And we also all have heard SAP’s vision for the future of the enterprise, with a single version of truth, a single HANA database hosting all business applications, enabling real-time reporting on operational data.

So, why am I coming back with this question, isn’t it clear already?

               Cost and Reality Check disrupt current rules on SAP HANA scalability

Well, there are 2 variables that disrupt all the above stated:

• Cost
• Reality

Cost, because customers won’t buy something just because someone tells them they should. I’ve written a lot about that as well, and don’t want to repeat myself here, but IT exists to serve business purposes, so the cost of IT cannot be higher than the value gained by the services IT provides, and this is the simple reason many organizations live with “good enough” solutions.

How many times, being you a buyer or a seller in IT, have seen customers complaining systems don’t perform, and buyers going there trying to sell more gear to solve that, and those projects never got budget? In many of those situations, as IT stakeholders went up the chain up to the CIO level, or even the CFO, LoB or CEO levels to get funding for those projects, what you hear from them is that “we can live with what we have, and cannot afford more costs there”.

And this brings to the sellers the challenge of needing to build a TCO business case, to show how the benefits sold will pay off the investment and provide savings on the long run, which in many cases is not easy.

So, balancing costs with the business benefits is a fundamental aspect of any IT adoption decision, which in turn drives a search for the best balance between architecture choices and “good enough” business results, than enable the lowest possible cost each specific business requirements.

And Reality, because the reality I’m seeing in customers is not compatible with the simple rules I’ve stated above as communicated by SAP.

On the “reality check” part, let me just give you to examples from real customer cases I’ve been involved just over the last 2 months.

On one case, we were talking about an analytics scenario, with native HANA applications, so no ABAP there. The initial sizing was 12TB, with an expected growth over 3 years to 40 TB of in-memory data, and 800 TB of disk based data.

We need to remember that today the biggest box available for SAP HANA today is the SGI UV300 server which can hold up to 24 TB of RAM (in 32GB DIMMs). Also important to note that when talking about “disk based data”, on this scenario it would be SAP HANA Dynamic Tiering, which today only allows for single-server implementations of the “Extended Store”, which ends up limiting the capacity of the extended store to about 100 TB.

So, we are talking about a system that breaks the barriers of “supported”. And I say “supported” because in many cases the limitations are not a factor of “technology limitation” but rather a factor either of existing experience, knowledge or confidence in such extreme scenarios.

We all know that SAP is an organization conservative by nature, and when they state something as “generally available” it means that it has been extensively tested, is well documented, and basically any customer should be able to do it with no major surprises. The other side of this is when we face a context of such accelerated pace of change, where customer scenarios keep pushing the boundaries of “generally available”, and that’s where “controlled availability” or “restricted availability” come in. For these extreme cases, if you limit yourself to the options of “generally available”, you may just be killing your organization’s adoption plans. To which I would say, its better to push the boundaries a bit, than just give-up.

So, on many of these extreme scenarios, scaling-up is not an option, and you must consider scale-out “by design” and from the start.

The other case was a customer evaluating the migration of a SAP ERP with IS-U to HANA. The sizing pointed out to the need of 25 to 35 TB of In-memory capacity for go-live (I’m still working on qualifying this further and understand whether the sizing assumptions were correct, as it would not be the first time I would see some “gross mistakes” in this area).

So, here as well, we are outside the limits of the currently possible.

We might discuss whether SGI would be able to load their 32 socket servers with 64GB DIMMs and then scale to 48 TB of in-memory capacity.

Being a fan of SGI technology, putting myself in the shoes of a customer (which I once was), once you go beyond 16 sockets, you are limited to a single vendor, as no one else can provide boxes that big, and if you are getting started with this sizes, you can only expect that it will continuously outgrow the limits of technology. So I would defend a “multi-vendor limit”, as for example when considering 16 sockets you have multiple vendors (Like SGI and Bull), which provides you bargain power as you are not locked in to a single vendor, which as a consequence makes prices more reasonable. 

Also, defining this limit, enables you “in-box” growth paths, as if SAP confirms that you have a memory bottleneck but your CPU utilization is rather low, you may be allowed just to grow in memory, braking the current rules of memory to CPU ratios (note that this only applies to growth scenarios, and not initial sizing – for more info on this read for example SAP Note 1903576 and it’s attachment).

So, why not start with scale-out by design?

Oh, yes. For the first use case you could as it’s an analytics use case, but for this one you can’t as SAP HANA Scale-out is not generally available for transactional use cases.

               Scale-out transactional workloads (Suite on HANA) is possible!!!

But do you know that SAP HANA Scale-out is released under “controlled availability” for transactional use cases? And that there is an installation guide for “SAP Business Suite powered by SAP HANA Scale-out” in attach to SAP Note 1781986? (Check-out my list of relevant SAP notes on this topic at: http://sapinfrastructureintegration.blogspot.com/p/important-sap-notes.html)

So, scaling out transactional workloads on HANA is not a discussion of technical possibilities.

It is possible, there are already a few customers doing it and there is SAP documentation on it.

Being in controlled availability means it is not a solution so spread out that anyone can do it without supervision or support, so being in controlled availability means that SAP must approve you to enter the program in order to have support, enabling them to validate that your scenario and arguments make sense, and at the same time, commit to support you and enable you to then operate the solution. 

And Suite on HANA has been in controlled availability for a long time as you can see from the “very old” SAP slide above (anything older than 1 year, in the HANA world is already too old! :) ).

               Some of the proposed solutions I’ve seen make no sense…

Before going further on my perspective on scale-out, let me tell you the “indication” that was given to both these customers, with which I struggle to agree.

On the analytics use case, the customer was told to break his “analysis groups” in independent “universes”, and then load each group in a different HANA system, and then have multiple scale-up smaller single nodes, instead of a single large scale-out cluster.

On the transactional use case, a similar solution to the one above was suggested: break your business units in groups and put one business unit in each separate ERP, with a smaller single HANA scale-up system.

Heck, what the hell?!?!?! So, what about all that talk of the single system, single source of truth SAP has been telling over the last couple of years?!?!?!?! Am I missing something here?

In both these scenarios, the implication will be a proliferation of smaller systems, and the build of an “integration system”. For example in the transactional use case, we’ll go back to the scenario of 1 ERP for each company and then a consolidation ERP for the holding. Wasn’t this one of the things SAP was promising to end with the adoption of SAP HANA?

Which one is better or worse? To integrate the data at the database level through a scale-out architecture, or to integrate it at a functional level by creating interfaces across the multiple instances?

I see this in the following perspective: once data exceeds the capacity of a single server you’ll need to distribute it across multiple servers anyway. If you divide the universe of data across disparate independent systems, you’ll then need to take care of the integration at the functional level. If you are a company managing a portfolio of business with a dynamic acquisition and divestiture activity, you’ll have for sure a consolidation system, and have set your functional knowledge already to manage the integration at this level, so I would understand that you break your “massive single instance into more and smaller independent systems”.

But if you really need the integrated view across all those universes of data, I would say it would be easier to manage a scale-out cluster (being for analytic workloads or transactional workloads), than breaking the data across smaller single systems and then keep leaving with the “data latency” problem due to the ETL process and then the delayed reporting process we heard so much Hasso Platner talking about as a problem that having all data in HANA would solve.

In the end, when dealing with very large systems, complexity will exist. So as a CIO or senior leader on an IT organization you need to evaluate where you'd rather have the complexity:

• At the functional level, and have your development and functional teams dealing with the integration of data from multiple systems, with interface development, agregation systems, etc... well, all the things you've had across the last 2 decades on SAP;
•  At the technical level, and have your infrastructure architects and HANA DBAs deal with it, through expensive SQL statement analysis and infrastructure optimization.

If your business scenario indicates that you should have a single very large instance, then whether you have the integration at a functional level or the technical level will be a factor of:

• If you integrate at the functional level, whether you can leave with the "reporting latency" introduced by the need to move data around and aggregate it , and whether you trust more the capacity of your functional and development teams to manage interfaces and data integration;
• If you integrate at the technical level, whether you trust your DBA and infrastructure teams ability to properly design, architect, build and operate such an environment.

Considering that I'm now working with Virtustream, the fact that we as a service provider are able to take care of all the complexity involved at the infrastructure and HANA DBA levels, I would say that if the reason for you to consider a functional integration is the lack of capacity or skills in your platform teams, you should consider a cloud approach to this, where Virtustream would provide you an SLA at the SAP Basis (platform) level.

But let me continue this blog post ignoring that I'm working for a cloud provider specialized on SAP HANA, and continue the reasoning as if I was involved in deciding on-premise implementations.

               Will the HANA reality be of many small systems??? In some cases, yes.

Being a bit of a devil’s lawyer here, right? ;-)

But we need to make our minds clear: either we believe in the benefits of “single pond” where we build a single source of truth, or we understand that there are still reasons for customers to have multiple systems like happened in the past. So, on my mind, the message in the PowerPoints is not matching the reality I’m seeing in the field.

As a sort of disclosure, I believe there are plenty of reasons for organizations to maintain multiple separate databases for each of their business systems, and I’ve written on how VMware based virtualization is a great match here, as it brings massive operational simplification and efficiencies. The example I gave at a previous blog post was a corporation that manages a portfolio of business, and keeps buying and selling companies, where keeping separate independent ERP systems for each business is crucial to simplify the divestiture when the time comes. We all know that splitting the data within a single SAP ERP is the key pain in separating a business group (and I've been more than once in such projects, and it was not fast, cheap or simple at all).

But I also understand that there are some special cases where having a single database for the whole business brings massive benefits, and we need to tackle both these scenarios with an open mind, and no bias toward one or the other, in order to truly be able to make the best choice for each customer organization. For example, companies with vertical or horizontal integration, where the different companies businesses are interrelated and can potencially cross sell to join customers between each other will certainly benefit from having an integrated real-time global view across businesses, which would point out to having a single global instance for all business units.

So, you’ll see me in one customer, depending on his particular scenario advising to keep all their smaller systems and run them all on VMware, while on another with a different business scenario, you may see me advising for a single global instance with all data in a single database.

In the end it, the “best solution” really depends on each company business scenario.

 

               Data Temperatures with Dynamic Tiering and SAP HANA Vora

Have to say here that I’m ignoring all the topics about data temperatures. But that would take us on a completely different direction, so I’ll chose to ignore the implications of data temperatures here, although I must say, it is a very relevant angle to the problem I’m describing.

In fact, having a approach to “data lifecycle management” from the start, and incorporating a discussion with your businesses on the “business value of data” right from the early stages of your project evaluation, may lead you to some massive reduction of your required in-memory footprint, and consequence reduction in the overall costs of the solution. I've approached this topic for example in my blog post "The right Architecture for S/4 HANA and IoT".

Have to note though, that the “split between current data and historical data” on SAP ERP on HANA is still not a reality, and the “Data Temperature” discussion when in face of transactional systems, today is still much more limited in options than when talking about analytic use cases like SAP BW on HANA.

Key barrier to scaling-out transactional workloads on HANA is operations experience

But, enough with problem analysis and considerations! Let me then share a bit my perspective in regards to scaling out “transactional workloads” on SAP HANA.

I believe the key reasons to avoid scaling out for transactional scenarios is the lack of knowledge on the market on how to implement and manage these systems.

The technology is there, is available and works.

But if people implementing and operating these systems don’t understand the “angle of scaling out transactional workloads”, you may easily end up with an application that, because of the performance impact a bad scale-out architecture, implementation and operation implications, may end up lagging a lot on expectations performance wise.

Note that I haven’t said, “won’t work”. I said “lagging on expectations”.

And I’m making this note because I believe it’s all about expectation management.

               One customer example on proper expectations management with SAP HANA adoption

And let me take the example on the picture bellow to make the case.

In this example a customer was moving BPC from Oracle to HANA. BTW, this customer was EMC Corporation (read the full story at: http://www.emc.com/collateral/analyst-reports/emc-it-breaks-grounds-sap-hana.pdf ). I was an EMC employee when this happened, and was privileged to be in close contact with the EMC IT SAP Architecture and Basis guys driving this when the project went live.

Once they moved BPC “as is” to HANA, immediately one of their critical financial processes passed from 53 minutes to 7 minutes. Then, after they applied EHP1 for BPC which brings the HANA optimized code for that process, it passed to 17 seconds. EMC IT here also tested the impact of virtualization and did tests both on physical servers and virtual servers and verified that running that process on physical would take less 1,5 second.

So, when talking with the business users, their expectation was to be able to run that process faster. When they saw 7 minutes they were very happy. So when they saw 17 seconds, they were exhilarated! When asked whether 1,5 second would make any difference for them, when confronted with the difference of cost on both scenarios, virtualization being so much cheaper became a no-brainer option.

But when talking with the technical teams, their concern was all about how much is the difference between running the process on physical vs virtual, which for the business is simply an irrelevant discussion.

I’m telling this story because this happened in November 2013, way before SAP supported SAP HANA in production on VMware. And I was privileged to observe closely this story as it developed.

I wrote a lot about the benefits of virtualization then, and that it was fundamental to do a proper expectation management. I believe we are in face of a similar “expectations management” discussion in regards to scaling out transactional workloads on SAP HANA.

               Managing SoH Scale-out is not more complex that managing Oracle DBs

So, on my perspective managing a SAP HANA scale-out cluster is in no way more complex than managing an Oracle database.

It’s just a matter of understanding the conditions and tools.

Starting with the Oracle example. I remember a colleague that came from a Microsoft background managing SQL Server, and started on a project where all systems were Oracle.

To make him understand that he needed to do expensive SQL Statement analysis, that he needed to work with development teams to build new indexes, needed to determine whether a table or index would benefit from having dedicated table space and needed to define whether a certain tablespace would benefit from a dedicated LUN, or when was time to do a table reorganization, or to how do create the rollback segments so that you wouldn’t have processes breaking… you should have seen his face… He was so lost and frustrated!!!

Now imagine that he was the SAP Basis admin on an organization that had just implemented SAP ERP, the system was growing very fast, and he only had his SQL Server background. The result would have been a disaster, because he didn’t understand the need for all these optimization processes (as for example with SAP ERP on SQL Server you don’t determine the table placement at the disk level, SQL Server automatically stripes all data across all available datafiles), neither knew where to look for problems, or even if he knew where to look, he wouldn’t know how to interpret the numbers, or what actions to take to solve the problems. Having had SAP ADM315, ADM505 and ADM506 training courses would have helped him a lot.

On my perspective, the discussion on scaling-out transactional workload on SAP HANA is of a similar nature to the story I just told of the SQL Server DBA going to an Oracle landscape.

Similar nature, but much simpler! As I believe the complexity involved in managing SAP HANA in a scale-out cluster is not nearly as complex as it was to manage an Oracle 8 database!!! Remember the “PCTFREE”, “PCTUSED” parameters on Oracle??? Those were tough days when compared with SAP HANA System Administration. When I remember the project I worked in 1998 of a R/3 system with a 500GB database and 1200 concurrent users… Getting the system running smoothly was no easy job! (those doing "Basis" back then know how huge such a system was at that time...)

               Understanding latency implications on SAP HANA performance

So, diving into SAP HANA scale-out challenges for transactional workloads.

First of all is all about an “expectation management” problem. And this is due to the effect of network latency on the processing of data in the cluster.

When you look at the picture above, and if you consider that SAP HANA is optimized to process data in RAM, you see that the latency for data transmission between RAM and the CPU on the same NUMA node is of about 60 nanoseconds, while if you transmit data across the network between two servers you’ll always be looking at latencies from micro to milliseconds.

This means that if you need to move data between nodes across the network, that is a very “expensive” operation on SAP HANA performance wise when compared with processing data in the RAM of the same server. This is because, SAP HANA having a shared nothing cluster architecture which means data only exists in one cluster node (I’ve also written about this a long time ago, so please read that one for some history and context), and when in a scale-out cluster, data is stripped across multiple nodes.

The challenge on a transactional system is then that as you have a lot of joins between tables across many different business areas, and you also have “business objects” like invoices, materials and others that are stored in many different tables, you’ll be facing a very high probability of a large number of processes requiring cross-node communication.

The consequence here is that once reporting against data that are on tables across different server nodes, you’ll need to move data over the network to gather it all together and calculate results. And that “costs” a lot more time than if all tables were in the RAM of a single server. Will this mean that those operations will be slower than on “anyDB”? Well, the reality may be similar to the scenario I’ve described once EMC decided to virtualize BPC on HANA.

The technical guys may be very concerned with how many millisecconds more it will take on scale-out vs scale-up, while the business guys will be thrilled with the improvements observed in terms of increased business visibility, and reduced lead-time to consolidated information, and might consider that the reports taking a bit longer to generate is a small price to pay, especially when in context of their starting scenario on legacy ERP on AnyDB.

The same problem happens when you write data, as if you write an object that will be stored in multiple tables, and those tables are distributed across different server nodes, you’ll need to do the write in each of the nodes, and the commit is only given once all nodes communicate that data has been written, so writing may become slower.

Then, my point that this is no different from managing an Oracle database.

As in Oracle (in fact as in any database), once you start dealing with very large systems, you’ll need to look at parameters and variables that for smaller scenarios you just ignore, as the standard/default configurations are good enough for you.

When faced with bigger systems, once you learn where to look and how to manage this additional level of detail, then it becomes routine and a no-brainer.

The same must happen with HANA. But the knowledge in the market is just not there yet.

SAP should develop and make available a course similar to the old “ADM315” (or any course for SAP Basis kind of people just focusing on performance analysis and optimization) focusing on HANA, just covering the monitoring tools today available to evaluate performance in a scale-out cluster, and well as the tools and mechanisms to address potential performance problems.

There is already some training on this area, must if fully focused on developers, and so, it would be useful to have something really focused on SAP Basis people.

For example, today there is the possibility to mirror tables across HANA nodes. Why is this relevant? Imagine that you have a very large table partitioned across multiple nodes, on which you run a report that implies a join with a configuration table that is just 100 lines long. By mirroring that table across nodes, you avoid cross node joins, and all joins are done locally in each node, so you truly taking benefit of the massive parallel scale-out architecture of SAP HANA.

Also the tools on HANA (like the “Plan Visualizer”) to analyze where time is spent when a specific statement is executed, are mind blowing when compared with I knew in Netweaver, and provide you a level of detail that enable you to clearly identify whether the delay is caused by network, disk or any other reason.

It is then my belief that the key barrier to further scale-out adoption is then lack of knowledge, with all the risk implications of it for business critical applications.

And on my perspective, although I may accept that scaling-up is the short term solution to avoid the risk, considering the increased number of customer cases I’m seeing with very large datasets needing massive amounts of memory, and considering that most things about Data Temperatures are not ready yet for transactional workloads (Dynamic Tiering, NLS or Vora do not solve this problem now), documenting and transmitting knowledge on SAP HANA Scale-out performance analysis and optimization would be the way to go for these customers (at least for the customer examples I mentioned here).

               Ok, then when to scale-up or to scale-out?

Couldn’t finish this blog, without adding some comments on the variables I identified at the beginning.

And that is, when is it a good option to scale-up and to scale-out.

So, let's look at what many say about scale-out:

• In some cases (with certain server vendors), from a CAPEX perspective 2 x 4 socket box is a lot less expensive than 1 x 8 socket (as an example);
• If you have to do HA, looking for example for a 18 TB system, instead of having 6+6+6+(6 for HA), you would have 2x24TB;
• If you have a very dynamic change environment, where you buy and sell companies, launch and divest business lines, having a standard building block in the datacenter enables you to easily reallocate these smaller boxes, while in the big box scenario, it will always be there, as you can’t even slice it with VMware.
• Looking also from a service provider perspective (and the same for very large customers with 50+ productive SAP Systems), scale-out provides less cost of vacancy as its easier to play around with the multiple smaller boxes.
• You could argue that scale-out drives higher OPEX:
• More OS images to manage and patch:
• If you implement automation tools, doing one or many should be the same, and for these 50+ systems customers, they will always be managing a ton of OS images, so more or less 5 OS images…
• Need to do continuous table distribution analysis and replacement:
• In the Oracle world everyone looks as a normal activity to look at table partitioning optimization in BW, and do “hot-spot analysis”, reorganizing tables across tablespaces or putting tables in dedicated tablespaces. Table redistribution on HANA would look exactly like that. If you make it a part of your core knowledge, it becomes a no brainer.
• Future S/4 will be based on Multi-tenant Database containers. So we are consolidating multiple applications on the same SAP HANA System to then break it up between multiple tenants? Why would we then put them all on a single box? Rather split them across a scale-out node. The only caveat here would be the network connection between the nodes, as if the tenants would have a lot of cross tenant communication; it would definitely be faster in a single scale-up node.
• Another topic that is gaining increased traction is “data aging” or “data tiering”, meaning most likely in the future you’ll have less and less data footprint. So, you would buy a ton of iron to solve a problem that will see its impact reduced across the next years. I would rather “scale-out” and then repurpose. Add to this the fact that many of the large scale-out sizing exercises already account for 3 years of growth… so who knows what will happen in 3 years.
• Imagine as well that you have a fully virtualized datacenter, and that you’ll need a 9 TB system. Why not to have a 3x3TB scale-out system on VMware? I know this is not available today, but I believe we’ll reach it at some point. Or that you have a 4 TB system that has outgrown the vShpere6 maximums? Do you move from virtual to physical and buy a huge box, or just scale-out?
• Imagine customers that have massive growth rates on their SAP Systems (know one with a 90 TB ERP…). Keep scaling up will lead you at one point in time to a wall. Scale-out keeps your scaling options open, while enabling to invest as you grow.
• Also from a bargain perspective, if you buy many smaller servers, your bargain power as a customer is higher, and you avoid vendor lock in…

All of these are examples of arguments about scale-out I've heard, and have to say, some of them are very strong.

But let’s look at scaling-up:

• A strong argument is definitely consolidating multiple AnyDB systems on a single SAP HANA system. But again, this will be based on MDC, so does it really make sense to put all of these in a single box?
• Another strong argument is that there are always development resources in the code development teams that suck, and produce bad code, that isn't properly checked for quality, especially when in "RUN mode", so doing maintenance on existing systems with the pressure from business to do it faster and cheaper. In a single scale-up system, this problem would not be as evident as in a scale-out.
• You may also say that its easier to buy more iron than to upgrade a SAP System. So, all the things in regards to MDC and Data Tiering will still take some years to be real and mature, and you need to make a decision today. Then a scale-up would be a good fit.
• Also if your system is expected to be rather stable, and you do not foresee a very dynamic change environment affecting the SAP Systems, you may argue that on the long run, a scale-up system may bring OPEX savings that compensate for the larger CAPEX.

You would say that the arguments above are highly biased towards scale-out, right?

There are 2 recent experiences I've had, which I believe add another perspective to this discussion:

• There are server vendors with modular architectures (for example Bull and SGI) that enable you to scale-up as you go, and rearrange the building blocks as you need.
• For example Bull servers are composed of 2 socket modules that are aggregated together. So a 16 socket server is the aggregation of 8 x 2 socket modules.
• In the example of SGI, the servers grow in 4 socket modules, and then a 16 socket server is the aggregation of 4 x 4 socket modules.
• On both you can add additional modules to grow, or reutilize the modules by breaking down the server in case you no longer need such a large box.
• I’ve also learned that “if you negotiate properly”, the costs of these servers can be equivalent (meaning not massively more expensive, and in some cases maybe even cheaper!!!) to similar capacity from other vendors in smaller servers.
• Then you have service providers like Virtustream, that have the possibility upon growth to reallocate compute capacity between tenants and so provide a risk free evolution path for these very large scenarios. Meaning, you don’t need to figure all this out by yourself, as Virtustream will do this analysis with you and provide you with the right solution in a purely OPEX model while taking care of SAP HANA systems administration for you, and so eliminating all this complexity and risk. This would enable you to just chose between scale-up or scale-out based on your business requirements, and not the architecture and systems administration constraints that would entail.

Conclusions: final questions and… what about Cloud?

So, today I would say, that my personal preference is to scale-up on all sizes up to 16 sockets (as there are multiple server vendor alternatives in the market), and to scale-out beyond that.

Why I say this? It’s not as simple as stated above, as I would need to factor in many considerations.

Questions I would ask to provide a better advice to customers would be:

• What is your growth forecast for the next 3 years: after 3 years technology is obsolete, and being SAP HANA running on x86 servers, you just need to do an homogeneous system copy, which being in a TDI scenario – with external storage – may mean just attaching the storage to a new server, with a minimum downtime and risk.
• Do you have any extreme performance requirements: number of transactions per minute may indicate how much being in a scale-out can become a problem performance wise or not. There are a lot of customers that have a lot of data, but not that extreme volume of transactions, which will mean that SAP will allow them to scale beyond current CPU/memory ratios to increase just memory - always when confirmed that your CPU utilization is really low.
• Do you need high availability, and which are your SLAs: when in a scale-out scenario, you have less stand-by capacity than in a scale-up scenario. So, considering the failover time of a node with SAP HANA Host Auto-failover in a scale-out scenario with external storage, if this is acceptable for the customer, may enable him to save some money. This may change once SAP enables SAP HANA System Replication with the standby node active for reporting, but we can't always postpone our decisions based on futurology, and this option is not yet available today.
• How resilient you need your data to be: this goes to RPO in a scenario of disaster. Many organizations put in place a DR scenario because they simply cannot afford to operate without SAP systems. But facts are showing that with today’s datacenter standards, it is almost impossible for a Tier4 datacenter to go out of service. This makes many customers to take an approach of disaster avoidance, and build high availability across different buildings in a Tier4 datacenter provider, instead of having asynchronous replication to a remote location. Have to say that this varies a lot by region, and by industry. For example in regions more subject to extreme natural disasters, is more likely for remote data replication to be required, while in western Europe is increasingly common for organizations to assume a disaster avoidance scenario and just do synchronous replication across metro distances. This will have implications on the mechanisms to put in place and the associated costs.
• And as a final aspect, I’ll look at what operation automation mechanisms have to be put in place: for example, there are customers making a standard doing more frequent data refresh on Q&A systems. Doing it for example on a weekly basis, implies a high degree of automation involved in this, and has implications on infrastructure architecture. It is a lot simpler and faster to automate this for a single scale-up node, than for a scale-out cluster.

And the obvious conclusion from this journey through my thoughts and learnings on this topic is obvious: the right choice depends!

As always, hope that this helps you build up your mind on what is right for your organization, and be aware of the possibilities.

Anyway, remember that today you can just avoid all this complexity and risk by putting all your SAP systems in an “Enterprise Class Cloud”, by leveraging the uniqueness of the offer of companies like Virtustream which have pushed Security and Compliance to a level that truly makes it safe for organizations to run their mission critical systems on. That also opens you the possibility of working with architects like me on evaluating your business scenario, and assisting you in making the right decisions for your business.

As a final word, have to say I’m feeling truly privileged and overwhelmed with the amount of talent and innovation at Virtustream and how it is leading the emergence of a new “Cloud Generation”, which I would call “Enterprise Class Cloud” or "Cloud 2.0"!

Stay tuned, as I’ll be writing very soon on what I’ve learned so far about Virtustream, what sets its Cloud offer apart, and in what ways it breaks barriers to adoption of SAP HANA, and overcomes long-time concerns of organizations when faced with the scenario of moving enterprise mission critical applications like SAP Business Suite to the cloud.