Comparability Criteria Selection
When insufficient data exists to power a statistical equivalency test such as TOST, the TI method may be considered an alternative. In general, a useful technique to consider the adequacy of each statistical approach is to perform a statistical performance assessment (SPA). An SPA permits an early assessment of each technique in order to correctly show comparability when there is no difference across means and correctly show noncomparability when there is a difference across means.
As an illustration of the TOST and TI approaches, consider the following scenario: 12 values are sampled from the pre-change process, and four values are sampled from the post-change process.
This data will be used to establish comparability in means. To calculate the SPA, we shall further assume the following:
1. Pre- and post-change processes are normally distributed,
2. Pre- and post-change processes have equal variances,
3. The TOST goalpost is two times the standard deviation from the pre-change process, and
4. A 90/99% tolerance interval will be calculated using pre-change process data.
It is possible to calculate the probability of meeting the comparability criteria by calculating the statistical power for a TOST approach, and the probability of all four post-change values falling inside the TI. The results are shown in Table 2 and the Figure.
For example, if there is a one standard deviation difference across the pre- and post-change means, there is roughly a 50% chance of (incorrectly) concluding that the means are equal using TOST, but more than a 99% chance using the TI approach. As shown in the Figure, the TI approach tends to conclude comparability more frequently than the TOST approach, regardless of the actual difference across the two means.
Using an SPA, all stakeholders in the comparability assessment can be made aware of the implications associated with a statistical approach. A reasonable comparability strategy may then be decided upon before collecting and analyzing data.