VMmark Scores Investigated: should VMmark be part of your hardware decisions?
by Johan De Gelas on May 8, 2009 12:00 AM EST- Posted in
- IT Computing
The VMmark Scoring Chaos
When the new Xeon based on the "Nehalem" architecture launched, there was a lot - and I am being polite - of confusion about the VMmark scores. Scores ranged from 14.22 to 23.55 for dual CPU servers based on the same Xeon X5570 2.93GHz. Look at the published and "non published" VMmark scores we found from various sources:
The new Xeon 5570 is between 55% and 157% faster, depending on your source. Let's try to make sense out of this. First, take the lowest score, 14.22.
Intel's own benchmarking department was very courageous to publish this score of 14.22. Intel allowed us to talk to Tom Adelmeyer and Doshi A. Kshitij, both experienced engineers. Tom is one of the creators of vConsolidate and Doshi is a principal engineer who wrote some excellent academic papers; his specialty is hardware virtualization. The answer I got was surprising: the 14.22 score was a simple "out-of-the box" VMmark run that did not make use of Hardware Assisted Paging (or EPT if you like) running on top of an ESX 3.5 update 4 beta. Remember only the full ESX 3.5 update 4 contains full support for Intel's EPT technology. Update: My mistake. ESX 3.5 update 4 has full support for Intel's X55xx CPU, but not for EPT technology. That is only available in ESX 4.0 and later. My thanks goes to Scott Drummonds (VMware) to point this out.
So the 14.22 score is not comparable to the 11.28 score of the AMD Opteron "Shanghai" as the latter is a fully optimized result. Intel's Xeon X5570 can obviously do better, but how much better? A score obtained in February in the same Intel labs, which was never published, was 17.9. It is in the right ballpark for a VMmark run that is clearly better optimized and most likely running with EPT enabled. Update: We believe this was run on early version of ESX 4.0.
On Intel's own site you can find this PDF, which in the very small print mentions a score of 19.51, obtained somewhere in march on a VMware ESX Build 140815 with DDR3-800 (Update: This seems to be a sort of "Release Candidate" of ESX 4.0). As running 13 tiles requires a lot of memory, Intel outfitted the Nehalem server with 18x4GB DDR3 DIMMs. Since this means there are three DIMMs per channel, the clock speed of the DDR3 is throttled back to 800MHz.
At the launch date of the Xeon X5570, scores above 23 were reached with VMware builds 148592, 148783, and 150817. Since the launch of VMware's vSphere 4.0, these numbers have been replaced by ESX 4.0. One of the reasons that these configurations obtain higher scores is the fact they run with two DIMMs per channel, so the DDR3 DIMMs run at 1066MHz. That is good for a boost of 5-6%, which has been confirmed by both Intel's and AMD's VMmark experts. The second and most important reason is that ESX 4.0 is used. As VMware states in some whitepapers, CPU scheduling has improved in the new ESX 4.0, especially for the Nehalem architecture with its SMT (Hyper-Threading).
Intel's server benchmarking department did a favor to the IT community by releasing the 14.22 "out-of-the-box, unoptimized, and no EPT" score. This gave us a realistic worst-case score. The benchmark showed that the newest Nehalem Xeon could outperform its competitor in even the worst conditions, adding credibility to Intel's claims. In contrast, the benchmark claims at Intel's product page are pretty shady.
A 161% performance boost over the previous generation is called an "exceptional gain", but the claim is completely overshadowed by the flawed comparison. It is simply a bad practice to compare a score obtained on - at that time - unreleased brand-new software (VMware ESX build 148592, similar to ESX 4.0) with a benchmark run on older but available software (ESX 3.5 Update 3). While there is little doubt that any server based on the Xeon X55xx is a superior consolidation platform, it is unfortunate that Intel did not inform its customers properly. Especially if you consider that a fair comparison with the Xeon 54xx and Xeon 55xx both on ESX 4.0 would probably also deliver "exceptional gains".
Facebook
Twitter
RSS
23 Comments
View All Comments
duploxxx - Sunday, May 10, 2009 - link
time for some OEM beta hardware :)has407 - Saturday, May 9, 2009 - link
With a single socket (assuming quad core), I'd think you should be able to do it with less memory, maybe 32-48GB? (unless your IO subsystem is slow) Even if that's beyond reach, a relative measure with a smaller number of tiles might be interesting (ok, it won't be strictly by-the-VMark-book).tshen83 - Friday, May 8, 2009 - link
So,I take it that you want to discredit VMmark as being a relevant benchmark for virtualization?You know VMmark isn't the only benchmark that says Nehalem is twice as efficient in performance/watt than Shanghais right?
In the last paragraph, you said "give us a few more days". To do what? To selectively choose a few benchmarks that show that Shanghai is a better CPU for virtualization workloads? Good luck with that.
Sometimes I want to find a ruler and measure just how deep you stuck your head up AMD's rear end. Sometimes I also wonder why a Belgian is so freaking adamant about AMD. Anand got too cheap I guess to outsource an important job to Belgium I guess.
7Enigma - Monday, May 11, 2009 - link
What I find so funny about this post is that since the Nehelem launch the author has chronically been labeled pro-Intel in the comments section of the majority of his articles. Just goes to show you; you can please all of the people some of the time, some of the people all of the time, and in the case of Intel/AMD, none of the people all of the time. :)whatthehey - Friday, May 8, 2009 - link
Or perhaps he just has some information on a new virtualization benchmark suite, which may or may not show Shanghai in a better light.I think it's pretty easy to conclude that the two year old design of VMmark is aging and not as relevant as when it first came out (if it was even truly relevant then). So let's wait a few more days, eh?
Sometimes I want to pull out a ruler to measure just how far up Intel's ass tshen83 has shoved his head so that he can't even consider any viewpoint that doesn't state that Intel is unequivocally the best. Seriously, look at any AMD or Intel article, and he's there espousing the virtues of Intel and trashing everything AMD does. It's not all black and white, dude... except when you get paid by Intel to do what you do, of course.
Viditor - Monday, May 11, 2009 - link
My own guess is that tshen83 has become a 100% Intel ass, the 2 things have merged in this reality...:)I would like to see how things compare on the larger boxes though. There are an awful lot of 4 and 8 way VM machines going out there right now...
noxipoo - Friday, May 8, 2009 - link
Plenty of companies have old servers that doesn't need much. NT4 to 2000 servers can easily range to those numbers at most corporations.
JohanAnandtech - Saturday, May 9, 2009 - link
Possible, but still an exception. Windows 2000 and NT4 servers have become a minority, probably less than 5% of the installed base.And you are probably not too concerned about CPU performance when consolidating those servers.
duploxxx - Friday, May 8, 2009 - link
Finally some good article to breach this new vmmark scores.Although it is clear that the new Nehalem based system is better then current shanghai this is mostly due to the 3 mem controllers (which in the end provides more mem/cpu) and faster memory. The HT feature is the main VMmark whoop score cause here, it is already stated by many Vmware performance representatives that people should take care about the HT core as a real core in production, if you do this the performance will get bad just as previous HT, although the ESX sw is no much more aware of this feature (esx 3.5u4 and esx4), but is seems like the vmmark is not able to see the difference since there is not enough load on the system.
All other features are now equal while shanghai switching time was way better then harpertown the nehalem is more or less equal, also the ept/npt or rapid V or whatever you want to call it is now implemented.
so a final vmmark performance score you stated around 16-17 sounds very reasonable.
the performance enhancements in esx4 are not really for HT rather the core coherency features like vmware wants to call this, iommu which will be first introduced by amd istanbul and most important the paravirtualized scsi driver and off course more cpu/vm and a lot of memory, scheduling improvement.....etc....
Perhaps you should contact there are aware of this VMmark real world difference and are working on a new version.
lopri - Friday, May 8, 2009 - link
Yet again from Johan. Johan never disappoints! I have just had a quick read, but I will take a thorough read later. Thank you much and I'd like the follow-up articles very much, too.