VMmark Scores Investigated: should VMmark be part of your hardware decisions?

Back to Reality

While Intel's claims on the Xeon 55xx product page are based on a flawed comparison, the newest VMmark data suggests that the "Nehalem Xeon" is indeed more than twice as fast than the older Xeons and (almost) twice as fast as the newest Opterons when running on ESX 4.0. We expect the Dual Opteron "Shanghai" 8389 at 2.9GHz to achieve a score between 12 and 13.5 on ESX 4.0, while the typical score for the Xeon X5570 is around 20-23.5 depending on the clock speed of the DDR3 modules.

That tells the ICT professional that the Xeon X5570 is a CPU with the potential to run extremely high amounts of VMs, but not much more. The real world value of VMmark is highly debatable as it is showing its age:

• How many of us are running more than 50 to 100 VMs, which need on average only 1GB per VM? It sounds like desktop virtualization, but it is supposed to represent server virtualization.
  
• Would there be any real world java application that shows the same performance profile as SPECjbb? No I/O, no shared data between the different threads, and "SPECjbb-only" JVM optimizations?
  
• How close is SysBench, which perform all its transactions on a monolithic table, to a real OLTP database running on top of a hypervisor?

The problem is that VMmark is only one data point, which hardly reflects any real world scenarios IT professionals currently use. As a result, VMmark is yet another industry benchmark where the experts of the large OEMs create unrealistically high scores with expensive SAN configurations. It's still interesting but hardly relevant for the real world. It is time for a new data point. The more virtualization scenarios tested the better. Just give us a few more days….