This is indeed an interesting comment!

Publication of benchmarks would be very useful. Real, explicitly detailed benchmarks that can be used as a basis for direct comparison and easily reproduced are key in my mind.

I have done some very interesting benchmarks of ESX versus Xen but as far as I understand VMWare ESX forbids publication of benchmarks.

It is quite possible that if I was allowed to publish benchmarks then someone might see a flaw in my testing or a sub-optimal configuration that I might have chosen - and I'm not suggesting that only ESX has performance issues.

An environment freely accepting of benchmark publications could rapidly see the dissemination of best practices documentation and recommendations as flaws in the various benchmarks are exposed. This should lead to improved user experience and product satisfaction, no doubt things VMWare would appreciate.

If Diane is seriously interested in showing customers real-world relevant performance via benchmarks then the first step should be to allow open publication of benchmarks by anyone. Please correct me if I am wrong on ESX having an anti-benchmarking clause, I don't have a copy of license agreement handy at the moment.

If Diane is talking about the useless TPC-C style benchmarks where vendors throw a thousand disks at a solution to make sure it's faster than the other guy and at the same time forbidding the free and open publication of benchmarks, no thanks.