The SPC provides a specification document that contains a functional description of the benchmark, along with a benchmark toolkit. The test sponsor is responsible for deploying the toolkit on the test platform.

The target of all SPC benchmarks is a storage subsystem. A large portion of the toolkit deployment is focused on taking advantage of the strengths of the storage subsystem. The test platform also includes a host system that presents a load to the storage subsystem.

top of page

The first deployment of any SPC benchmark toolkit on a target test platform will typically produce mediocre performance results. To bring the performance results to a competitive level, much tuning will likely be required.

There is no guideline on how much tuning is sufficient. It depends on many factors involving the configuration of host system and storage subsystem as well as the performance goals that have been established.

At one end of the spectrum, the test platform has no significant flaws and is well adapted to the SPC benchmark toolkit and to the underlying I/O load. A few weeks of tuning can be sufficient to produces the desired results.

On the other end of the spectrum, the first few executions of the benchmark may uncover major flaws in the hardware design. Some components such as the I/O bus or the caching architecture may have to be re-engineered. Several months of work may be necessary before an acceptable performance result can be produced.

top of page

The SPC requires that all published results be independently verified by an SPC Certified Auditor. The first step of this verification is the audit of the toolkit deployment and the definition of an audit protocol.

This step is also referred to as the "protocol definition". It takes place prior to the audit of the actual performance result. During the protocol definition, all the aspects of the toolkit deployment are verified for conformance with the SPC benchmark specification.

In addition, all the procedures and scripts that will be used to execute the actual performance measurements and collect the results are reviewed and approved by the auditor. The benchmark is now ready to be executed and to produce compliant performance results.

top of page

Once all major bugs and performance deficiencies have been removed and once the benchmark toolkit deployment has been audited, the final tuning and the formal measurements can take place.

The final tuning usually consists of finalizing the layout of the data store on the available storage subsystem and fine-tuning the various storage subsystem and OS parameters. Last minute bugs may also be discovered and may have to be corrected at this stage.

Because the toolkit deployment has already been audited, no major changes can be made to the test platform without having to repeat the protocol definition.

Once the target performance has been reached, the formal measurement can begin. The audited procedures are followed and the audited scripts are executed following the audit protocol. The results of all the measurement tests are captured into predefined audit files, ready for the final audit.

top of page

The results from all the measurement tests must be audited by an SPC Certified Auditor. This is usually done by sending the entire set of final audit files to the auditor. This process is referred to as a "remote audit" as it does not require the physical presence of an auditor during the actual test.

The auditor verifies the results from the measurement tests by examining the audit files. In addition, the auditor verifies the configuration used in the test and checks that the proposed pricing of the tested configuration complies with the benchmark requirements. When necessary, the auditor has the option of remotely connecting to the test bed for additional hands-on verifications. In some cases, the auditor may elect to perform part of the result audit at the test site.

top of page

Only fully audited results can be filled with the SPC. The SPC's disclosure requirements include the production and submittal of a Full Disclosure Report (FDR).

An FDR must be produced for each benchmark result. It is generally produced by the auditor and reviewed by the test sponsor. The FDR must be sent to the SPC prior to any use of the result in public material or presentation. It must include the details of the entire implementation, price quotes for third party components used in the configuration and an attestation letter from a SPC Certified Auditor.

Each FDR, once filed with the SPC, enters a 60-day review period during which it can be challenged for non-compliance with the benchmark specifications. A few FDRs are challenged on rare occasions. Such challenges may result in a finding of non-compliance and a withdrawal of the result.

top of page