|
Producing TPC Results
Following is a summary description of the general process for
producing, auditing and publishing TPC results. A formal description
of this process can be found in the Policies and Guidelines of
the TPC.
The TPC's role is focused on the standardization of benchmark specifications
and the publication of results produced by vendors in conformance with
those specifications.
System vendors seeking the publication of
TPC performance results for their platforms are responsible for
accomplishing the following:
- Implementing the TPC benchmark on their
platform
- Tuning the implementation of the benchmark
- Auditing the implementation of the benchmark
- Executing the TPC benchmark on their configuration
- Auditing the benchmark results
- Disclosing the results to the TPC for publication
Implementing the Benchmark
The TPC provides a specification document that contains a functional
description of what should be implemented. No actual code or implementation
kit is provided by the TPC. Each vendor is responsible for designing
and building an optimal implementation of the TPC's functional
specifications. (In the case of TPC-D some code is provided by
the TPC to help in the generation of raw data.)
At the center of any TPC benchmark implementation is a database
management software (DBMS). A large portion of that implementation
is dependent on the DBMS. As a result, most DBMS vendors have
developed their own benchmark implementation kit. These kits vary
in completeness and ease of use. Some are better adapted to selected
operating systems.
The first step toward implementing a TPC benchmark is to select
a DBMS vendor. Aside from strategic relationships, one of the
criteria for selecting that vendor should be how complete, adapted
and easy to use their TPC implementation kit is and how willing
they are to support the benchmarking effort.
Once a DBMS has been selected and an implementation kit obtained
from the DBMS vendor, the implementation can be deployed on the
target hardware platform.
top of page
Tuning the Implementation
The first deployment of any TPC benchmark kit on a target hardware
platform will typically produce mediocre performance results.
To bring the performance results to a competitive level, much
tuning is required.
There is no guideline on how much tuning is enough. It all depends
on many factors involving the hardware and the software components
of the test platform as well as the performance goals that have
been established.
At one end of the spectrum, the target hardware platform has no
significant flaw and is well adapted to the TPC benchmark kit
and the underlying DBMS. A few weeks of tuning usually produces
the desired results.
On the other end of the spectrum, the first few executions of
the benchmark may uncover major flaws in the hardware design.
Some components such as the I/O bus or the CPU caching architecture
may have to be re-engineered. The port of the DBMS on the hardware
platform may be deficient and may have to be re-optimized. Several
months of work may be necessary before an acceptable performance
result can be produced.
top of page
Auditing the Implementation
The TPC requires that all published results be independently verified
by a TPC Certified Auditor. The first step of this verification
is the audit of the benchmark implementation.
This step is also referred to as the "pre-audit" as it takes place
prior to the audit of the actual performance result. During the
pre-audit, all the components of the implementation are verified
for conformance with the TPC benchmark specification.
In addition, all the procedures and scripts that will be used
to execute the actual performance measurements are reviewed and
approved by the auditor. The benchmark is now ready to be executed
and compliant performance results can be produced.
top of page
Executing the Benchmark
Once all major bugs and performance deficiencies have been removed
and once the benchmark implementation has been audited, the final
tuning and the formal measurements can take place.
The final tuning usually consists in finalizing the layout of
the database on the available disks and fine-tuning the various
DBMS and OS parameters. Last minutes bugs may also be discovered
and may have to be corrected at this stage.
Because the implementation has already been audited, no major
changes can be made at this point without having to repeat the
pre-audit.
Once the target performance has been reached, a fresh new database
in built and the formal measurements begins. The audited procedures
are followed and the audited scripts are executed. The results
of all the measurement tests are captured into predefined audit
files, ready for the final audit.
top of page
Auditing the Benchmark Results
The results from all the measurement tests must be audited by
a TPC Certified Auditor. This is usually done by sending the entire
set of final audit files to the auditor. This process is referred
to as a "remote audit" as it does not require the physical presence
of an auditor during the actual test.
The auditor verifies the results from the measurement tests by
examining the audit files. In addition, the auditor verifies the
configuration used in the test and checks that the pricing of
the 5-year cost of ownership complies with the benchmark requirements.
When necessary, the auditor has the option of remotely connecting
to the tests bed for additional hands-on verifications.
top of page
Disclosing the Results to the TPC
Only fully audited results can be disclosed to the TPC. The TPC's
disclosure requirements include the production and submittal of
a Full Disclosure Report (FDR).
An FDR must be produced for each benchmark result. This report
must be sent to the TPC prior to any public use of the result.
It must include the details of the entire implementation, price
quotes for third party components used in the configuration and
an attestation letter from a TPC Certified Auditor.
Each FDR, once filed with the TPC, enters a 60-day review period
during which it can be challenged for non-compliance with the
benchmark specifications. While a few FDRs are challenged from
time to time, it should be mentioned that since mandatory audits
have been established no such challenge has resulted in a benchmark
result being officially declared non-compliant.
top of page
|
