The TPC provides a specification document that contains a functional description of what should be implemented. The TPC does not provide a full implementation kit for all of its benchmarks. When a partial kit or no kit is provided, the vendor is responsible for designing and building an optimal implementation of the TPC's functional specifications.

At the center of any TPC benchmark implementation is a DBMS. A large portion of the implementation is dependent on the DBMS. As a result, most DBMS vendors have developed their own benchmark implementation kit. These kits vary in completeness and ease of use. Some are better adapted to selected operating systems.

The first step toward implementing a TPC benchmark is to produce or obtain a full implementation kit for the selected benchmark.

top of page

The first deployment of any TPC benchmark kit on a target test platform will typically produce mediocre performance results. To bring the performance results to a competitive level, much tuning will likely be required.

There is no guideline on how much tuning is sufficient. It depends on many factors involving the hardware and the software components of the test platform as well as the performance goals that have been established.

At one end of the spectrum, the target hardware platform has no significant flaws and is well adapted to the TPC benchmark kit and to the underlying DBMS. A few weeks of tuning can be sufficient to produce the desired results.

On the other end of the spectrum, the first few executions of the benchmark may uncover major flaws in the hardware design. Some components such as the I/O bus or the CPU caching architecture may have to be re-engineered. The port of the DBMS on the hardware platform may be deficient and may have to be re-optimized. Several months of work may be necessary before an acceptable performance result can be produced.

top of page

The TPC requires that all published results be independently verified by a TPC Certified Auditor. The first step of this verification is the audit of the benchmark implementation.

This step is also referred to as the "pre-audit" as it takes place prior to the audit of the actual performance result. During the pre-audit, all the components of the implementation are verified for conformance with the TPC benchmark specification.

In addition, all the procedures and scripts that will be used to execute the actual performance measurements and collect the results are reviewed and approved by the auditor. The benchmark is now ready to be executed and to produce compliant performance results.

top of page

Once all major bugs and performance deficiencies have been removed and once the benchmark implementation has been audited, the final tuning and the formal measurements can take place.

The final tuning usually consists of finalizing the layout of the database on the available storage subsystem and fine-tuning the various DBMS and OS parameters. Last minute bugs may also be discovered and may have to be corrected at this stage.

Because the implementation has already been audited, no major changes can be made to the benchmark kit without having to repeat the pre-audit.

Once the target performance has been reached, a fresh new database in built and the formal measurements begin. The audited procedures are followed and the audited scripts are executed. The results of all the measurement tests are captured into predefined audit files, ready for the final audit.

top of page

The results from all the measurement tests must be audited by a TPC Certified Auditor. This is usually done by sending the entire set of final audit files to the auditor. This process is referred to as a "remote audit" as it does not require the physical presence of an auditor during the actual test.

The auditor verifies the results from the measurement tests by examining the audit files. In addition, the auditor verifies the configuration used in the test and checks that the proposed pricing of the tested configuration complies with the benchmark requirements. When necessary, the auditor has the option of remotely connecting to the test bed for additional hands-on verifications.

top of page

Only fully audited results can be filed with the TPC. The TPC's disclosure requirements include the production and submittal of a Full Disclosure Report (FDR).

An FDR must be produced for each benchmark result. This report must be sent to the TPC prior to any public use of the result. It must include the details of the entire implementation, price quotes for third party components used in the configuration and an attestation letter from a TPC Certified Auditor.

Each FDR, once filed with the TPC, enters a 60-day review period during which it can be challenged for non-compliance with the benchmark specifications. A few FDRs are challenged on rare occasions. Such challenges may result in a finding of non-compliance and a withdrawal of the result.

top of page