Choosing a Benchmark Design

There essentially exist three types of performance benchmarks: standard, semi-standard, and custom.

Custom benchmarks, at one end of the spectrum, are likely to be the most faithful to the target application. Yet, choosing this option has many pitfalls. The definition of a fully custom benchmark often turns out to be too lengthy and too costly while having a high risk of producing a biased test that favors one vendor over the others.

At the other end of the spectrum are standard benchmarks. The choice of a standard benchmark completely eliminates the time and cost of the custom design while providing the highest degree of impartiality toward all vendors. But, the use of a standard also means that the specific needs of the target application were not examined and that the winning technology may not be the most adapted.

Between those two opposite poles on the benchmark spectrum is a solution that combines the best of both worlds and avoids most of the pitfalls. This solution is called a semi-standard benchmark.

Strengths of a Semi-Standard Benchmark

With a semi-standard benchmark, most of the benchmark design and definition is eliminated by the adoption of an existing standard benchmark. The work is focused on an analysis on the specificity of the target application, which gets translated into a set of elements of the standard benchmark that need to be adapted, augmented or eliminated.

Adapting an existing benchmark requires substantially less time and effort than the complete definition of a fully custom benchmark. Yet, experience has shown that the quality of the outcome is generally equal if not superior.

The option of a semi-standard benchmark was chosen by such data warehouse pioneer as Boeing and MCI as the premier tool in their technology selection process. By using the an adapted version of the TPC-D standard they were successful in combining the specificity of a custom approach with the reliability of a standard.

back to projects