Benchmark design is an art and a science.

• The art of building a synthetic workload that captures the most performance critical elements of a real-life application.
• The science of simplifying the workload to bring ease in the execution of the benchmark and clarity in the analysis of the results.

Combining the art and the science is key to maintaining a high degree of relevance to the real-life application being modeled.

The best fit is the right solution.

Original benchmark designs are at one end of the spectrum. They are best used when no standard benchmark exists in the targeted application area.

The definition of a completely original benchmark is a project that requires significant resources and expertise.

Creating an accurate performance measurement framework requires answers to questions such as:

• Can the workload be both simple and relevant?
• Which execution contraints should be included?
• How to avoid introducing an artificial bias?

Semi-Standard benchmarks represent the middle ground in the spectrum. This approach combines the best of both custom and standard designs.

With a semi-standard benchmark, most of the design is done through the adoption of an existing standard specification. Elements of the standard benchmark are then adapted, augmented or eliminated to better fit the target application.

Adapting an existing benchmark requires substantially less time and effort than defining a completely original benchmark. Yet, experience has shown that the quality of the outcome is generally equal if not superior.

Standard benchmarks are at the other end of the spectrum. They completely eliminate the time and cost of benchmark designs.

Standards can provide the highest degree of impartiality between all competing vendors and technologies.

Since the use of a standard benchmark implies that the specific needs of the target application have not influenced the benchmark design, the winning technology may not be the most suitable.

Standard = One size fits all