A Note on Confidence Interval for the Power of the One Sample 𝑡 Test

Abstract

In introductory statistics texts, the power of the test of a one-sample mean when the variance is known is widely discussed. However, when the variance is unknown, the power of the Student's 𝑡-test is seldom mentioned. In this note, a general methodology for obtaining inference concerning a scalar parameter of interest of any exponential family model is proposed. The method is then applied to the one-sample mean problem with unknown variance to obtain a (1−𝛾)100% confidence interval for the power of the Student's 𝑡-test that detects the difference (𝜇−𝜇₀). The calculations require only the density and the cumulative distribution functions of the standard normal distribution. In addition, the methodology presented can also be applied to determine the required sample size when the effect size and the power of a size 𝛼 test of mean are given.

1. Introduction

Let (𝑥₁,…,𝑥_𝑛) be a random sample from a normal distribution with mean 𝜇 and variance 𝜎². As presented in any introductory statistics text, such as Mandenhall et al. [1, page 425], a (1−𝛾)100% confidence interval for 𝜎² is𝐿_𝜎²,𝑈_𝜎2=(𝑛−1)𝑠²𝜒^{2𝑛−1,1−𝛾/2},(𝑛−1)𝑠²𝜒^{2𝑛−1,𝛾/2},(1.1) where 𝑥=∑𝑥_𝑖/𝑛, 𝑠²=∑(𝑥_𝑖−𝑥)²/(𝑛−1), and 𝜒^2𝜈,𝛿 is the 100𝛿th percentile of the 𝜒² distribution with 𝜈 degrees of freedom. Moreover, for testing𝐻₀∶𝜇=𝜇₀versus𝐻_𝑎∶𝜇=𝜇₀+𝑘𝜎𝑘>0,(1.2) the null hypothesis will be rejected at significance level 𝛼 if𝑥−𝜇₀𝑠/√𝑛>𝑡^{𝑛−1,1−𝛼},(1.3) where 𝑡_𝜈,𝛿 is the 100𝛿th percentile of the Student's 𝑡 distribution with 𝜈 degrees of freedom. Although the power of this test is rarely discussed in introductory statistics texts, Lehmann [2] proved that the probability of committing Type II error of a size 𝛼 test with the hypotheses stated in (1.2) is𝛽=𝐺_{√𝑛−1,𝑘𝑛}𝑡_{𝑛−1,1−𝛼},(1.4) where 𝑘=(𝜇−𝜇₀)/𝜎 is the effect size and 𝐺_𝜈,𝜆(⋅) is the cumulative distribution function of the noncentral 𝑡 distribution with 𝜈 degrees of freedom and noncentrality 𝜆. Note that the calculation of 𝛽 involves the unknown 𝜎. A naive point estimate of 𝛽 is𝛽̂=𝐺_𝑛−1,𝑘̂_√𝑛𝑡_{𝑛−1,1−𝛼},(1.5) where 𝑘̂=(𝑥−𝜇₀)/𝑠. Thus, the corresponding point estimate of the power of the size 𝛼 test that detects the difference (𝜇−𝜇₀) is 1−𝛽̂.

In Section 2, a general methodology is proposed for obtaining inference concerning a scalar parameter of interest of an exponential family model. Applying the general methodology to the one-sample mean problem with unknown variance, a (1−𝛾)100% confidence interval for 1−𝛽 is derived. This interval estimate will depend only on the evaluation of the density and the cumulative distribution functions of the standard normal distribution. The methodology can also be used to determine the required sample size when the effect size and the power of a size 𝛼 test are fixed. Numerical examples are presented in Section 3 to illustrate the accuracy of the proposed method. Finally, some concluding remarks are given in Section 4.

2. Confidence Interval for the Power of the Test and Sample Size Calculation

From (1.1), for a given (𝜇−𝜇₀) value, a (1−𝛾)100% confidence interval for (𝜇−𝜇₀)√𝑛/𝜎 is𝜇−𝜇⁰√𝑛√𝑈_𝜎2,𝜇−𝜇⁰√𝑛√𝐿_𝜎2.(2.1) Hence, from (1.4), the corresponding confidence interval for 𝛽 is𝐿_𝛽,𝑈_𝛽=𝐺_{𝑛−1,(𝜇−𝜇0)√√𝑛/𝑈𝜎2}𝑡_{𝑛−1,1−𝛼},𝐺_{𝑛−1,(𝜇−𝜇0)√√𝑛/𝐿𝜎2}𝑡_{𝑛−1,1−𝛼}.(2.2) Finally, a (1−𝛾)100% confidence interval for the power of a size 𝛼 test that detects the difference (𝜇−𝜇₀) is1−𝑈_𝛽,1−𝐿_𝛽.(2.3)

Evaluating (2.3) requires the cumulative distribution function of the noncentral 𝑡 distribution, which is generally not discussed in introductory statistics texts. In statistics literature, various approximations of 𝐺_𝜈,𝜆(⋅) have been proposed. For the rest of this section, a simple and accurate approximation of 𝐺_𝜈,𝜆(⋅) will be derived.

Let 𝑋₁,…,𝑋_𝑛 be identically independently normally distributed random variables with mean 𝜇 and variance 𝜎². It is well known that 𝑋=∑𝑋_𝑖/𝑛 and (𝑛−1)𝑆²/𝜎²=∑(𝑋_𝑖−𝑋)²/𝜎² are independently distributed as normal with mean 𝜇 and variance 𝜎²/𝑛 and 𝜒² with (𝑛−1) degrees of freedom, respectively. Let 𝑇^∗=√𝑛𝑋−𝜇/𝜎−√𝑛𝑧_𝑝𝑆/𝜎=√𝑛𝑆𝑋−𝜇−𝑧_𝑝𝜎,(2.4) where 𝑧_𝑝 denotes the 100𝑝th percentile of the standard normal distribution, then 𝑇^∗ follows a noncentral 𝑡 distribution with (𝑛−1) degrees of freedom and noncentrality −√𝑛𝑧_𝑝.

Now, consider a sample (𝑥₁,…,𝑥_𝑛) from a normal distribution with mean 𝜇 and variance 𝜎². Let the parameter of interest be𝜓=√𝑛𝑠𝑥−𝜇−𝑧_𝑝𝜎,(2.5) where 𝑥=∑𝑥_𝑖/𝑛 and 𝑠²=∑(𝑥_𝑖−𝑥)²/(𝑛−1), then the log-likelihood function can be written asℓ(𝜃)=ℓ𝜓,𝜎²=−𝑛2log𝜎²−12𝜎²𝑠²𝑛−1+𝜓²+𝛿𝑠𝜓√𝜎²,(2.6) where 𝛿=−√𝑛𝑧_𝑝. Denote that𝑝(𝜓)=𝑃𝑇^∗≤𝜓=𝐺_{𝑡𝑛−1,𝛿}(𝜓).(2.7)

The overall maximum likelihood estimate (MLE) of 𝜃, 𝜃̂=(𝜓,𝜎²)^=(𝛿√(𝑛−1)/𝑛,((𝑛−1)/𝑛)𝑠²)^ is obtained by solving (𝜕ℓ(𝜃)/𝜕𝜃)|_𝜃=𝜃̂=0, and the determinant of the observed information matrix evaluated at the overall mle is||𝑗_𝜃𝜃′𝜃̂||=||||𝜕²ℓ(𝜃)𝜕𝜃𝜕𝜃′||||_𝜃=𝜃̂=2(𝑛−𝑛⁴1)³𝑠⁴.(2.8) The constrained mle of 𝜃 at a fixed 𝜓, 𝜃̂_𝜓=(𝜓,𝜎^2𝜓)=(𝜓,𝑠²𝐴²/4𝑛²), where𝐴=𝛿²𝜓²+4𝑛𝑛−1+𝜓²−𝛿𝜓,(2.9) is obtained by solving (𝜕ℓ(𝜃)/𝜕𝜎²)|_𝜃=𝜃̂_𝜓=0. Moreover, the determinant of the observed nuisance information matrix evaluated at the constrained mle is||𝑗_𝜎2𝜎2𝜃̂_𝜓||=||||𝜕²ℓ(𝜃)𝜕𝜎²𝜕𝜎²||||_𝜃=𝜃̂_𝜓=8𝑛⁵𝑠⁴𝐴⁵(𝐴+𝛿𝜓).(2.10) Hence, the signed log-likelihood ratio statistic is𝑟=𝑟(𝜓)=sgn𝜓−𝜓2[ℓ(𝜃̂)−ℓ(𝜃̂_𝜓)]^1/2=sgn𝜓−𝜓−𝑛log4𝑛(𝑛−1)𝐴²+𝛿²−2𝑛𝛿𝜓𝐴^1/2.(2.11)

It is well known that 𝑟 is asymptotically distributed as the standard normal distribution with rate of convergence 𝑂(𝑛^−1/2). Hence, 𝑝(𝜓) can be approximated by Φ(𝑟) where Φ(⋅) is the cumulative distribution function of the standard normal distribution. It is important to note that 𝑟 is reparameterization invariant.

In statistics literatures, various likelihood-based small sample asymptotic methods have been proposed. In particular, if the model is a canonical exponential family model and the canonical parameter is 𝜃=(𝜓,𝜆′)′, Lugannani and Rice [3] derive𝑝(𝜓)=1−Φ(𝑟)−𝜙(𝑟)1𝑟−1𝑞,(2.12) where 𝜙(⋅) is the density function of the standard normal distribution, 𝑟 is defined in (2.11), and 𝑞 takes the form𝑞=𝑞(𝜓)=𝜓−𝜓||𝑗^𝜃𝜃′𝜃̂||||𝑗_𝜆𝜆′𝜃̂_𝜓||^1/2.(2.13) This approximation has a rate of convergence 𝑂(𝑛^−3/2). It is important to note that 𝑟 is reparameterization invariant whereas 𝑞 is not.

For a general exponential family model with canonical parameter 𝜑=𝜑(𝜃) and a scalar parameter 𝜓=𝜓(𝜃), to obtain inference concerning 𝜓 based on the Lugannani and Rice (1980) [3] method, 𝑟 remains unchanged as in (2.11) because it is reparameterization invariant, but 𝑞 has to be re-expressed in the canonical parameter scale, 𝜙 scale. To achieve this, let 𝜑_𝜃(𝜃) and 𝜑_𝜆(𝜃) be the derivatives of 𝜑(𝜃) with respect to 𝜃 and 𝜆, respectively. Denote 𝜑^𝜓(𝜃) to be the row of 𝜑^𝜃−1(𝜃) that corresponds to 𝜓, and ‖𝜑^𝜓(𝜃)‖² is the square length of the vector 𝜑^𝜓(𝜃). Let 𝜒(𝜃) be a rotated coordinate of 𝜑(𝜃) that agrees with 𝜓(𝜃) at 𝜃̂_𝜓. Then𝜒(𝜃)=𝜑^𝜓𝜃̂_𝜓‖‖𝜑^𝜓𝜃̂_𝜓‖‖𝜑(𝜃)(2.14) can be viewed operationally as the scalar parameter of interest in 𝜑(𝜃) scale.

Since ℓ(𝜃)=ℓ(𝜑(𝜃)), by the chain rule in differentiation, we have||𝑗_𝜑𝜑′𝜃̂||=||𝑗_𝜃𝜃^′𝜃̂||||𝜑_𝜃𝜃̂||⁻²,||𝑗_(𝜆𝜆^′₎𝜃̂_𝜓||=||𝑗_𝜆𝜆^′𝜃̂_𝜓||||𝜑^𝜆𝜃̂_𝜓𝜑_𝜆𝜃̂_𝜓||⁻¹.(2.15) Hence, an estimated variance for |𝜒(𝜃̂)−𝜒(𝜃̂_𝜓)| in 𝜑(𝜃) scale is |𝑗_(𝜆𝜆^′₎(𝜃̂_𝜓)|/|𝑗_𝜑𝜑(𝜃̂)|. Thus, 𝑞=𝑞(𝜓), as defined in (2.13) and expressed in 𝜑(𝜃) scale, is𝑞=𝑞(𝜓)=sgn𝜓−𝜓||𝜒𝜃̂−𝜒𝜃̂_𝜓||||𝑗^𝜑𝜑′𝜃̂||||𝑗_{(𝜆𝜆′)}𝜃̂_𝜓||^1/2.(2.16) Therefore, 𝑝(𝜓) can be obtained from (2.12) with 𝑟 and 𝑞 being defined in (2.11) and (2.17), respectively.

Note that the model being considered is an exponential family model with canonical parameter𝜑(𝜃)=1𝜎²,1𝜎²𝑥−𝑠√𝑛𝜓−𝑧^𝑝√𝜎²^.(2.17) From (2.17), we have𝜑_𝜃(𝜃)=⎛⎜⎜⎜⎝0−1𝜎⁴−𝑠√𝑛𝜎²−𝑥𝜎⁴+𝑠𝜓√𝑛𝜎⁴−𝛿2√𝑛𝜎²^3/2||𝜑_𝜃⎞⎟⎟⎟⎠,𝜃̂||=−𝑠√𝑛𝜎⁶,||𝜑^𝜎₂𝜃̂_𝜓𝜑_𝜎²𝜃̂_𝜓||=1𝜎^8𝜓1+𝐵²,(2.18) where𝐵=−𝑥+𝑠𝜓√𝑛−𝛿𝜎^2𝜓2√𝑛.(2.19) Moreover, by obtaining the inverse of 𝜑_𝜃(𝜃), we have𝜑^𝜓(𝜃)=√𝑛𝜎²𝑠(−𝐵,−1).(2.20) Hence, from (2.14), we can obtain||𝜒𝜃̂−𝜒𝜃̂_𝜓||=1√1+𝐵²|||||−√𝑛𝜓(𝑛−1)𝑠+𝛿𝐴²+4𝛿𝑛(𝑛−1)4√𝑛(𝑛−1)𝑠𝐴|||||.(2.21) Thus, from (2.16), we have𝑞=𝑞(𝜓)=sgn𝜓−𝜓||𝛿𝐴²−4𝑛𝜓𝐴+4𝛿𝑛(𝑛−1)||√𝐴+𝛿𝜓√𝑛(𝑛−1)𝐴^5/2.(2.22) Finally, 𝑝(𝜓)=𝐺_{𝑛−1,𝛿}(𝜓) can be approximated from (2.12) with rate of convergence 𝑂(𝑛^−3/2).

By reindexing all the necessary equations, we have𝐺_{√𝑛−1,𝑘𝑛}𝑡_{𝑛−1,1−𝛼}=1−Φ(𝑟)−𝜙(𝑟)1𝑟−1𝑞,(2.23) where 𝜙(⋅) and Φ(⋅) are the density and cumulative distribution functions of the standard normal distribution, and𝑟=sgn𝑘√𝑛−1−𝑡_{𝑛−1,1−𝛼}−𝑛log4𝑛(𝑛−1)𝐴²+𝑛𝑘²−2𝑘𝑛^3/2𝑡_{𝑛−1,1−𝛼}𝐴^1/2𝑞=sgn𝑘√,𝑛−1−𝑡_{𝑛−1,1−𝛼}𝑛(𝑛−1)𝐴^5/2||𝑘√𝑛𝐴²−4𝑛𝐴𝑡_{𝑛−1,1−𝛼}+4𝑘𝑛^3/2(𝑛−1)||𝐴+𝑘𝑛^1/2𝑡_{𝑛−1,1−𝛼}𝐴=−𝑘√,𝑛𝑡_{𝑛−1,1−𝛼}.+𝑘²𝑛𝑡^{2𝑛−1,1−𝛼}+4𝑛𝑛−1+𝑡_{𝑛−1,1−𝛼}.(2.24) Finally, with a predetermined effect size 𝑘 and power of a size 𝛼 test, the sample size can be obtained by iterations.

Note that DiCiccio and Martin [4] derived an asymptotic approximation of marginal tail probabilities for a real-valued function of a random vector where the function has continuous gradient that does not vanish at the mode of the joint density of the random vector. Applied to the noncentral 𝑡 distribution problem, the results are identical. Nevertheless, the approach of DiCiccio and Martin [4] is quite different from the proposed method. More specifically, DiCiccio and Martin [4] worked directly from the log density and treated the parameters as fixed whereas the proposed method works from the log-likelihood function where the data are observed.

3. Numerical Example

Figure 1 plots the power function of a one-sample 𝑡 test against the effect size 𝑘 for 𝑛=2,3 and 𝛼=0.05,0.01. The exact method is obtained from the built-in cumulative distribution function of the noncentral 𝑡 distribution in 𝑅. From the plot, it is clear that the signed log-likelihood ratio does not provide satisfactory results. The proposed method and the built-in function of 𝑅 are very close even when the sample size is 2. It is interesting to note that the built-in function of 𝑅 has a discontinuity point in the 𝑛=2,𝛼=0.01 case.

(a) n=2, α = 0.05

(b) n=3, α = 0.05

(c) n=2, α = 0.01

(d) n=3, α = 0.01

Figure 1 

Now, consider the data set recorded in Mandenhall et al. [1, page 103] 0.46,0.61,0.52,0.48,0.57,0.54.(3.1)

For testing the hypothesis𝐻₀∶𝜇=0.5versus𝐻_𝑎∶𝜇=𝜇₁>0.5,(3.2) the power function of a size 0.05 test and the corresponding 95% confidence bands are plotted in Figure 2. From Figure 2, the approximated power at 𝜇₁=0.52 is 0.5764. Furthermore, the 95% confidence interval for the power of the above test when 𝜇₁=0.52 is (0.1856,0.8992). At first, the confidence interval seems too wide. However, by examining (2.3), the result is not too surprising because (2.3) depends on (1.1). Since 𝜒² distribution is a skewed distribution, by defining the confidence interval of 𝜎² to have equal tail coverage, (1.1) is a wide interval and hence (2.3) is a wide interval.

Figure 2 

Power function and the 95% confidence band.

Finally, to illustrate the determination of the sample size, let the effect size be 0.8, and at 𝛼=0.025, let the power be at least 0.9, then the proposed method gives 𝑛=19 with power 0.909.

4. Summary and Conclusion

The (1−𝛾)100% confidence interval for the power of the size 𝛼 Student's 𝑡-test detecting the difference (𝜇−𝜇₀) is presented. The major advantages of the presented confidence interval are that it depends only on the evaluations of the density and cumulative distribution functions of the standard normal distribution and that it is extremely accurate. The 𝑅 source code is available from the author upon request.

As a final note, the proposed method can be applied to any distribution that belongs to the exponential family model with known canonical parameters. Although the method depends on the correct specification of the underlying distribution, Fraser et al. [5] examined a special case when the error distribution of the regression model is misspecified and the likelihood-based method still gives results that are more accurate than the existing Central Limit Theorem-based approximations.