Large ERP Systems on HANA? Balancing performance and availability.

Just came from a meeting with a large customer.

The topic: migrating to HANA!

Found that SAP is working with this customer to migrate a 70 TB ERP and a 90 TB BW both to HANA.
Will it work? Of course it will!

So here are the unanswered questions:

• But isn't Suite on HANA only available on Single Node?
• What about datacenter practices for aggressive uptime and data resilience demands? What kind of compromises or balances will have to be made between data resiliency and system performance?

Let me give you my perspective on this.

There is a restricted customer program to address the needs of Business Suite customers migrating to HANA where the target database will not fit today's largest servers which make available 6 TB of RAM.

This program presents lots of conditions and restrictions, and I believe will evolve a lot over the coming years.

Some of the details can be found on the following SAP Notes:

• https://service.sap.com/sap/support/notes/1774566 Business Suite on HANA - Restrictions
• https://service.sap.com/sap/support/notes/1825774 Business Suite Powered by HANA - Multi-Node Support
• https://service.sap.com/sap/support/notes/1950470 Hardware Pre-requisites for Business Suite on HANA Scale-Out

          Suite on HANA Scale-Out

So, from these notes you can see that scale-out support for Business Suite on HANA is coming "sloooowly".
Sometimes we see news and we wonder whether this is really happening. Well I've just met two customers on this journey, one in Europe and another one in Asia. So, definitely there are customers today moving very large mission critical ERP systems to HANA.

By curiosity, the cases I saw were customers exchanging DB2 for HANA. Some of that SAP on Power/AIX/DB2 going away to HANA on x86... interesting aspect to think about...

This means that if your sizing points out to a system larger than 6 TB, escalate this to your SAP account team and get your company approved for this restricted program of support for Suite on HANA Scale-Out.

There are some considerations in regards to suite on HANA configs:

• Both on BW and Suite on HANA, as still with HANA SP8 row tables are all loaded on the primary node, the size of the row store will be critical in two aspects:
• the startup time of HANA in the case of a failure, as the row store is fully loaded before user connections are accepted to HANA;
• the size of the server, so even with servers with 6 TB of RAM, as only about half of it can be used for persistent storage, we are talking about a maximum of between 2 and 3 TB of row store, which means for lots of customers I know will imply to implement some house keeping practices they never had over the last 10 or 20 years...
• There are two operations that will imply cross node communication, and that will become critical in terms of performance:
• Cross node joins, so trying to group as better as possible in the same node tables that are most often joined will be critical, as well as it will be critical for system admins to monitor this and evaluate the need for table redistribution in case it starts to become a problem;
• Multi-step commits for those cases where a commit implies writing to objects on multiple nodes, which will imply that each node needs to make its own commit and then communicate it so that the commit is only given as done once all the nodes have committed their part (the problem of working with scale-out shared nothing architectures). So minimizing these occurrences will be critical for write performance.

So, two recommendations become very clear:

1. Be very tough from the beginning with limiting the amount of row tables on custom developments as well as ensuring aggressive housekeeping on them;
2. Build your cluster from the start with a very "powerful and performing" cross HANA nodes network connection, as this will easily become the weakest point in the performance chain.

          Can I run Suite on HANA Scale-out with nodes of less than 6 TB?

When I talk about suite on HANA scale-out, there are also lots of cases where customers would like to do it with smaller boxes, mainly for a cost reason, as 2 x 4 socket servers are usually a lot cheaper than a single 8 socket server.

If you read the SAP notes I've mentioned above, you can easily find out that SAP only considers a customer for the Suite on HANA Scale-out program if their sizing is larger than the largest box currently available. And lots of customers do not like to hear that.

But this is the current status.

I believe the future will make this evolve in one of two ways:

1. Either SAP will with time and experience open up further the suite on Hana scale-out program, which will for sure imply greater requirements in terms of network connection between the nodes;
2. Or all server vendors in the market will invest further in 8 sockets servers, driving down their cost in the market, making 8 socket x86 the new standard (which will depend a lot on Intel investing further in chipsets and other components targeted to 8 sockets machines).

Which is the most likely? There are lots of variables influencing, and it can go both ways.
Nevertheless, today option 2 seems the most likely.

But what restrictions do I expect to come out from SAP with further openness for Suite on HANA Scale-Out?

• Higher requirements in terms of the network interconnect between the HANA nodes, so like we have seen on Oracle's Hexadata, it would not surprise me for SAP to demand 40 GB connection between the HANA nodes as the internode communication will be very critical (maybe Infiniband might become a requirement);
• The other side that might represent a bottleneck, would be the log writes for 4K and 16K blocks, for which I would expect as well tougher requirements on these in terms of write latency and throughput. I believe technologies like EMC's recently acquired DSSD might become the solution for this challenge.

In summary, if your going to spend money on Infiniband switches, wouldn't this make a 2 x 4 socket config more expensive than 1 x 8 socket config? Let me leave the question like this for us to think about it...


          The balancing act between performance and data resilience

But there is an aspect I do not see often discussed, and which is very important: the laws of physics!

Remember that, even with fiber optics, information always travels at the most at the speed of light. So if you have high resiliency requirements, for example, by having data replicated to a very distant location, it will still take time for data to reach that location.

And although global communications are improving in terms of bandwidth, while lowering costs, having large network connections over large distances still costs a lot of money.

So, with HANA we are talking about the system writing in the grades of 1.000 MB / second / per node. This means that if we have 10 nodes, we are talking about 10.000 MB / second being transmitted over a very large distance.

Why do I bring this point?

Because I've seen some very passionate discussions on how to make SAP HANA the best performing as it can be on one room, and on the other I saw people taking care a service level agreements for RPO, RTO and data resilience. I happened to enter both rooms, and it was like to different realities!

One will impact the other! If you want very high performance, you'll need to go to asynchronous methods to get your data to a second location.

And depending on variables like:

• affordable cost of communications;
• change rate of your system > volume of changed data to be transmitted to the secondary site;

      It may happen that your data loss (RPO) may climb above your requirements.

Or if you want not to lose any data, and have synchronous replication, then either you are very limited on the distance you can replicate to, or will observe a major impact on the write performance of your HANA cluster.

My point here is that, when we reach certain volumes of data and change rates, the discussion is no longer a discussion of technologies, but of limitations of physics.

So, my advice here would be to: adopt the technologies you know better, and then measure the impact this will have in terms of application availability and discuss that with your business.


          Conclusion

I've written a blog on why in certain circumstances hardware based solutions can be better for HANA implementations. My arguments there are not that hardware based solutions are always better, but that there are multiple circumstances where they may prove to be a better fit.

When we come to discussions about very large systems, we enter the world of disaster avoidance, and not disaster recovery, since the data gravity (volume of data in a single location) simply starts to impose limitations.

Today you can run very large systems on HANA, but what I've seen is that many times, teams involved in these projects put beforehand the limitation in front of the customer that he can only use certain software based solutions, and through those passionate discussions, forget the limits of physics and the cost implications of those options.

The comment I wanted to make then was: "Sorry guys, but you still have a lot to learn about IT operations...".

But instead I've tried (as I always do), to get these guys to a new level of understanding that at certain points there is no single better technology option.

There will be multiple ways to solve a certain customer challenge, and that they should focus on addressing the business challenges of the customer through all those great benefits HANA brings to the table, and trust more on their partners who have been working on operations for years to then ensure the technical solution at the datacenter level meets the requirements.

Here I have to say, that there are still lots of missing pieces:

• Still no TDI allowed for Suite-on-HANA scale-out, which I believe is an urgent need to meet these very large customer scenarios;
• SAP is not involving their technology partners properly and early enough in the process, providing an incomplete perspective on the possibilities to their customers.

I believe these are the natural evidences of a maturity journey being walked, and hopefully in some time, all these aspects or concerns have become clearer for the key SAP and customer stakeholders and we'll have better and more robust solutions to achieve a better balance between data resilience and performance, leveraging both HANA's software features as well as the capabilities SAP's technology partners can bring to the table.