There has been a lot of buzz about the new Nehalem processor and chipset family from Intel and how to optimize memory configuration for speed. I had quite a lengthy discussion with some colleagues at work on how to best organize the RAM to gain the most performance from the Nehalem’s 5500 chipset. I wanted to share some of what I found while researching.

The Nehalem uses an integrated memory controller on each CPU with 3 channels and 3 DIMM slots on each channel. This gives most dual CPU servers 2 memory controllers (one per CPU) with 6 channels and 3 DIMM slots per channel (total of 18). This gives many, many MANY options on how to configure servers. My discussion today with my colleagues was about the HP BL490G6.

Here is a great graphic from HP that gives us a look into the way the Nehalem is organized, it includes parts of the BL490 G6 (click to enlarge):

Click to zoom.

With the Nehalem, I like to stick to non-fully populated memory because of the benefit from the increased memory bandwidth when all channels are not in use.  When using all 18 DIMM slots the bus speed is downgraded to 800mhz compared to the fastest at 1333Mhz with only a single slot per channel occupied.  Having the bus speed at downgrade to 800 Mhz from 1333 Mhz bus is almost 40% less throughput overall which is a considerable jump.

Now I want to talk about all of this in relation to virtualization with VMware.  In this scenario I am going to be talking about using 8GB Dual Ranked Registered ECC DIMMS and the 95w versions of the Nehalem*, but the concepts can be applied to smaller or larger DIMMs.  The generally accepted rule of thumb is to have 2 vCPU per core (not a limit, just a good point to start from when planning). So, assuming that I am going to run 1vcpu per VM (statistically most VMs only require a single vCPU), then that is a starting point of 16 machines on each blade. With 2gb of RAM each (again, an average point to start from), 32gb of RAM would be required (however, in an ideal situation I would want to double that so physical hosts are running at 50% in case of a disaster situation causing a host failure).

One issue with using 64gb (Scenario 1) is that it would give us non-uniform memory distribution across memory channels** so the next logical move would be to 80GB of memory (Scenario 2), which would be 10×8gb memory modules – still allowing for expansion and giving us the extra bus speed over 800mhz.  This again presents an issue; 80GB would create a non-uniform memory distribution situation across CPUs/CPU Cores (80GB / 2cpu / 3 channel comes to 12 which cannot be divided by 8GB DIMM modules evenly across channels), which is less than ideal. In the end (Scenario 3), 8GB of memory per channel per CPU is 48GB, 24 per CPU, 6gb per core and will use the fastest bus speed. This is probably more than adequate for most designs because the next move there would be to 96gb which is probably overkill for most environments.  The 50% overhead margin will be decreased by adding more blades to the cluster, thus the load from a single node failure can be spread out more evenly.

Here are a few tables summarizing that last paragraph so it makes more sense:

Click to zoom.

Scenario 3 and 4 illustrate how to get to 96 GB in the interest of cost savings and speed respectively.  Either of these scenarios I would consider acceptable – I just want to stay away from the 800MHZ speeds due to the huge performance loss.  There really isn’t a wrong way to go about implementing servers based on Nehalem, it just requires some forethought to achieve the best results for each organizations goals.  Both Intel and VMware (as well as most server vendors) have best practice white papers published on this subject so be sure to check with them as well.  I would be interested in hearing about anyone’s experiences with Nehalem and also anything that I may have missed. Leave a comment or drop me a note!

*-HP does offer the BL490c with 1333MHZ bus using 6 DIMMS, but this was not the origional design of the chipset. The 95w processors are required to achieve the 1333MHZ bus speed.
** – It is desirable to have uniform distribution across memory channels and CPU cores because this will then require less calculation on the part of the hypervisor to place work loads.  This is also much more efficient for the hardware as a whole.

One approach is to build fewer, higher powered hosts. This is known as “scale up” architecture.  With this architecture there are fewer hosts to manage and a higher virtual machine density.  One obvious disadvantage to this is that any single host failure will have a greater impact on the overall infrastructure.  Another approach is to build many lower powered hosts. This is known as “scale out” architecture.  With this architecture there are more hosts to manage. However, virtual machine density will be small, so a host failure will have less of an impact on the overall infrastructure.

In my opinion, there needs to be a combination of both. There is a delicate balance between scale up and scale out. Too much scale up and there is a greater chance of a failure taking down a critical service. Too much scale out and the density of virtual machines becomes so low that virtualization almost becomes meaningless (aside from the benifits of high availability).  I believe this balance will be different in every environment. It is the most important piece of information to determine in any project, yet it is often very difficult and can have long standing effects.

So the question becomes: scale up or scale out, or a combination of both?

One concept that the instructor keeps bringing up is to Know Thy App.  What this means is that any successful virtualization project begins with a thorough understanding of the applications that are being virtualized.  I never really gave it thought, but this is the most important part of being a VMware engineer: the understanding of many, many, many different applications.

An exceptional systems engineer knows three or four applications, and probably only one or two of them inside and out (obviously there are exceptions to this).  An exceptional VM engineer really needs to know every application that would come in contact with his environment in extreme detail.  For example, when an application developer comes and claims that virtualization is making their application slow, the VM engineer needs to have the knowledge to analyze their application on all levels and find the issue.  The solution may or may not be at the fault of the virtual environment (my guess is it it’s not, but I’m obviously biased).

So, KNOW THY APP.