Performance benchmarks

Every development project needs a formal performance engineering process - one that emphasizes on early performance testing and benchmarking.

For performance benchmarks, it is recommended to do a shallow and wide implementation of a few critical use-cases and then run the load tests against the target hardware. These test results would help in some basic capacity planning.

But what if you have to do some initial rough capacity planning to allocate budgets and do not have the time to do a formal benchmarking exercise. It is here that standard performance benchmarks help. These standard performance benchmarks take a sample transactional use-case (e.g. Order Processing System) and run this workload on various platforms to gather statistics. There are 2 standards that are quite popular -

1. TPC (Transaction Processing Performance Council) - (TPC) is a non-profit organization founded to define transaction processing and database benchmarks and to disseminate objective, verifiable TPC performance data to the industry. TPC-C is the benchmark for OLTP workloads. 


2. SPECjEnterprise2010 - SPECjEnterprise2010 is an industry-standard benchmark designed to measure the performance of application servers conforming to the Java EE 5.0 or later specifications.

Interesting results of the performance benchmarks on various hardware can be found here:
http://www.tpc.org/tpcc/results/tpcc_perf_results.asp
http://www.spec.org/jEnterprise2010/results/jEnterprise2010.html

For the past few years, the Java Day Trader application and its .NET equivalent StockTrader application have been used by vendors to compare the performance of Java vs .NET on their respective platforms. Jotting down some links that point to some interesting debatable data :)

http://www.ibm.com/developerworks/opensource/library/os-perfbenchmk/index.html

http://blogs.msdn.com/b/wenlong/archive/2007/08/10/trade-benchmark-net-3-0-vs-ibm-websphere-6-1.aspx

http://msdn.microsoft.com/en-us/netframework/bb499684.aspx

https://cwiki.apache.org/GMOxDOC22/daytrader-a-more-complex-application.html