VMmark Scores Investigated: should VMmark be part of your hardware decisions?

ESX 4.0: Nehalem Enhanced

There are strong indications that the Xeon 54xx will not gain much from running VMmark on ESX 4.0. The "CPU part" of performance improvements in the new ESX 4.0, the foundation of VMware's vSphere, seems to have focused on the new Nehalem Xeon. That is hardly a surprise, as the previous Xeon platforms have quite a few disadvantages as a high performance virtualization platform: limited bandwidth, no hardware assisted paging, and power hungry FB-DIMMs.

The new Nehalem Xeon and Shanghai Opteron are clearly the virtualization platforms of the future with hardware-assisted paging, loads of bandwidth, and all kinds of power saving features. Nevertheless, ESX 4.0 really focused on the newest Xeon 55xx series. Even the newest Opterons gain very little from running on ESX 4.0 instead of ESX 3.5:

The quad Opteron 8389SE at 3.1GHz performs about 9% faster on ESX 4.0 compared to the Opteron 8384 at 2.7GHz on ESX 3.5 update 3. In other words, the performance gain seems to be solely coming from the 15% higher clock and not some hypervisor improvements. That is pretty bad news for AMD. Comparing the 11.28 VMmark of the Opteron 2384 obtained on ESX 3.5 update 3 to the ones of the Xeon x5570 on ESX 4.0 (>22) is unfair, but it does not seem like that score will improve a lot. Even worse, the newest dual Xeon servers outperform the best quad Opteron platforms when both are running the newest hypervisor from VMware.

AMD told us they expect that the VMmark scores of the Opterons will be 5 to 10% higher on ESX 4.0. Although the Quad socket numbers do not reflect this, it is a plausible claim. ESX 4.0 takes into account the processor cache architecture to optimize CPU usage and has a finer grained locking mechanism. These improvements should benefit all CPUs.