VMmark Scores Investigated: should VMmark be part of your hardware decisions?

The VMmark Scoring Chaos

When the new Xeon based on the "Nehalem" architecture launched, there was a lot - and I am being polite - of confusion about the VMmark scores. Scores ranged from 14.22 to 23.55 for dual CPU servers based on the same Xeon X5570 2.93GHz. Look at the published and "non published" VMmark scores we found from various sources:

The new Xeon 5570 is between 55% and 157% faster, depending on your source. Let's try to make sense out of this. First, take the lowest score, 14.22.

Intel's own benchmarking department was very courageous to publish this score of 14.22. Intel allowed us to talk to Tom Adelmeyer and Doshi A. Kshitij, both experienced engineers. Tom is one of the creators of vConsolidate and Doshi is a principal engineer who wrote some excellent academic papers; his specialty is hardware virtualization. The answer I got was surprising: the 14.22 score was a simple "out-of-the box" VMmark run that did not make use of Hardware Assisted Paging (or EPT if you like) running on top of an ESX 3.5 update 4 beta. Remember only the full ESX 3.5 update 4 contains full support for Intel's EPT technology. Update: My mistake. ESX 3.5 update 4 has full support for Intel's X55xx CPU, but not for EPT technology. That is only available in ESX 4.0 and later.  My thanks goes to Scott Drummonds (VMware) to point this out.

So the 14.22 score is not comparable to the 11.28 score of the AMD Opteron "Shanghai" as the latter is a fully optimized result. Intel's Xeon X5570 can obviously do better, but how much better? A score obtained in February in the same Intel labs, which was never published, was 17.9. It is in the right ballpark for a VMmark run that is clearly better optimized and most likely running with EPT enabled. Update: We believe this was run on early version of ESX 4.0.

On Intel's own site you can find this PDF, which in the very small print mentions a score of 19.51, obtained somewhere in march on a VMware ESX Build 140815 with DDR3-800 (Update: This seems to be a sort of "Release Candidate" of ESX 4.0). As running 13 tiles requires a lot of memory, Intel outfitted the Nehalem server with 18x4GB DDR3 DIMMs. Since this means there are three DIMMs per channel, the clock speed of the DDR3 is throttled back to 800MHz.

At the launch date of the Xeon X5570, scores above 23 were reached with VMware builds 148592, 148783, and 150817. Since the launch of VMware's vSphere 4.0, these numbers have been replaced by ESX 4.0. One of the reasons that these configurations obtain higher scores is the fact they run with two DIMMs per channel, so the DDR3 DIMMs run at 1066MHz. That is good for a boost of 5-6%, which has been confirmed by both Intel's and AMD's VMmark experts. The second and most important reason is that ESX 4.0 is used. As VMware states in some whitepapers, CPU scheduling has improved in the new ESX 4.0, especially for the Nehalem architecture with its SMT (Hyper-Threading).

Intel's server benchmarking department did a favor to the IT community by releasing the 14.22 "out-of-the-box, unoptimized, and no EPT" score. This gave us a realistic worst-case score. The benchmark showed that the newest Nehalem Xeon could outperform its competitor in even the worst conditions, adding credibility to Intel's claims. In contrast, the benchmark claims at Intel's product page are pretty shady.

A 161% performance boost over the previous generation is called an "exceptional gain", but the claim is completely overshadowed by the flawed comparison. It is simply a bad practice to compare a score obtained on - at that time - unreleased brand-new software (VMware ESX build 148592, similar to ESX 4.0) with a benchmark run on older but available software (ESX 3.5 Update 3). While there is little doubt that any server based on the Xeon X55xx is a superior consolidation platform, it is unfortunate that Intel did not inform its customers properly. Especially if you consider that a fair comparison with the Xeon 54xx and Xeon 55xx both on ESX 4.0 would probably also deliver "exceptional gains".