Constrained Parameters in Applications: Review of Issues and Approaches

Abstract

This review article provides an introduction to statistical issues that arise when some statistical model parameters are constrained. This often happens in applications, in particular in testing for variance components (e.g., genomics) and construction of one-sided confidence intervals (e.g., environmental risk analysis). Heuristic explanations are provided, and a number of general and recent statistical results that appeared in statistical literature are summarized for use in applications. Simulation results are shown for illustration of consequences of ignoring parameters on the boundary. Special attention is paid to likelihood ratio tests, but other approaches to confidence interval construction, such as Wald, bootstrap, and Bayesian are also briefly discussed. This paper presents examples from the risk assessment field and genomics, but all conclusions apply to whenever one-sided testing is conducted. Recommendations are provided for dealing with parameters on the boundary for a range of situations.

1. Introduction

Most standard results in statistical literature (e.g., [1]) are derived assuming that the true values of parameters are interior to the parametric space. It is well known that, under some very general conditions, the usual asymptotic theory of estimation based on the maximum likelihood estimates (MLEs) and the usual asymptotic theory of tests based on the likelihood ratio tests (LRTs) are valid and provide useful tools for meaningful statistical inference in large samples. Typically, we conclude the asymptotic normality of the MLE and the asymptotic chi-square distribution of −2 * ln (LRT) with suitable degrees of freedom (df) under a null hypothesis. This is what is generally covered in statistical textbooks.

However, the asymptotic distribution of the LRT in situations when regularity conditions do not hold, in particular when parameters of interest and/or nuisance parameters are on the boundary of a parameter space, is important in many applications, for example, in variance components testing in many natural science applications; however, statistical textbooks do not cover this subject. In the field of genetics and biology, there is indeed considerable amount of interest in statistical issues when some parameters may lie on the boundaries: in almost 1000 references to pioneering paper [2] on statistical results when parameters are on the boundary, about half come from publications in genetics and biology fields.

This review paper is organized as follows. Section 2 first introduces some examples from various applications that provide motivation. Next, it provides heuristic reasoning to demonstrate complicated statistical issues arising in boundary parameter situations using simple examples. Section 3 surveys literature on several approaches to parameters on the boundary and summarizes published results, paying most attention to the LRT. Finally, discussion section summarizes advice for scientists working in applications.

2. Examples and Heuristics

2.1. Statistical Issues due to Boundaries: Examples and Motivation

We start with basic examples of normal distribution that we will carry through the next section.

Example 1. Interior and boundary points of parametric spacefor parameters of interest and nuisance parameters.
Let 𝑋₁,𝑋₂,…𝑋_𝑛∼𝑁(𝜇,𝜎²),(a)𝜎² is known. 𝐻₀∶𝜇=0 versus 𝐻₁∶𝜇≠0 results in the parametric space (−∞,+∞), 0 is an interior point, and 𝜇 is parameter of interest,(b)𝜎² is known. 𝐻₀∶𝜇=0 versus 𝐻₁∶𝜇>0 results in the parametric space [0,+∞), 0 is a boundary point, and 𝜇 is parameter of interest,(c)𝜇 is unknown. 𝐻₀∶𝜎²=1 versus 𝐻₁∶𝜎²≠1 results in the parametric space [0,+∞) and 1 being an interior point. 𝜎² is parameter of interest and 𝜇 is called a nuisance parameter.

We continue with an application from the risk assessment field.

Example 2 (Constraints in multistage model). Toxicological experience and principles indicate that the response will generally be bounded and nondecreasing in the dose level [3]. Monotonicity may require constraints on the parameters of a dose response model, or constraints also may be needed to ensure that the probability lies within [0;1]. Any of these constraints on a parameter leads to a parametric space that includes boundary. Quantal response models represent the probability 𝑃(𝑑;𝜃) of a quantal response, like presence or absence of a particular type of cancer, in relation to dose (𝑑) in a model with parameter vector 𝜃. The experimental data consist of counts of animals exposed to a chemical, the numbers exhibiting the response, and the dose levels (e.g., 𝑛_𝑖, 𝑥_𝑖, 𝑑_𝑖, 𝑖=0,1,…,𝑚) from a bioassay conducted with mice or rats, typically in 2–5 dose groups and the control group, each having 10–50 animals. Commonly used model for modeling bioassay data is called multistage 𝑃(𝑑;𝜃)=1−exp−^𝑘_𝑗=0𝜃_𝑗∗𝑑^𝑗,𝜃^𝑗≥0.(1) All parameters of the multistage model are nonnegative, resulting in parameter space [0, +∞), with true or estimated values of parameters may be 0 or be on the boundary, and 𝑘 is called the order of the multistage model, for example, model with 𝑘=3 is called cubic. Maximum likelihood estimation for the parameters 𝜃 employs the binomial likelihood 𝐿(𝜃∣X)=^𝑚_𝑖=0𝐵𝑛_𝑖,𝑥_𝑖𝑃𝑑_𝑖;𝜃^𝑥𝑖1−𝑃𝑑_𝑖;𝜃^{𝑛𝑖−𝑥𝑖},(2) where 𝐵(𝑛_𝑖,𝑥_𝑖) is the binomial coefficient. The benchmark dose method [4, 5] consists of estimating a lower one-sided confidence limit for the dose associated with a specified increase in adverse response (i.e., increased risk) above the background level.

In the context of dose-response modeling, the statistical problems arising when true values of parameters of a dose-response model may be on the boundary were acknowledged long ago [5] but have not been resolved clearly for the practitioners of dose-response modeling. Molenberghs and Verbeke [6] provide a nice discussion of these issues in the risk assessment context.

Our final example is variance component testing in genetics.

Example 3 (ACE model in twin studies). The ACE model used in twin studies consists of the three components: A (additive genetics), C (common environment), and E (unique environment). Given the ACE model, researchers can test which of the three variance components are present. The likelihood ratio statistic is generally used as the basis for these tests [7]. The statistical test of AE against ACE alternative is the test of the hypothesis that there is no influence of environmental factors common among twins. This corresponds to the statistical test that variance component corresponding to C is 0. The test of E against ACE is the test of no familial correlation, that is, of the hypothesis that there is no influence of genetic or shared environmental factors. This corresponds to the statistical test of the null hypothesis that variance components corresponding to A and C are both 0. As variance is always nonnegative, the null hypothesis that variance component(s) is 0 is on the boundary of that parameter space.

What follows from the examples above is that most common causes of boundaries are one-sided testing and a dual problem of one-sided confidence interval construction. Suppose that we are testing the null hypothesis for a point 𝜃₀. The two-sided test 𝐻₀∶𝜃=𝜃₀ versus 𝐻₁∶𝜃≠𝜃₀, and corresponding parameter space is [−∞,+∞]. However, for one-sided test: 𝐻₀∶𝜃=𝜃₀, versus 𝐻₁∶𝜃>𝜃₀ the corresponding parameter space is [𝜃₀,+∞], and 𝜃₀ is necessarily a boundary point! Therefore, in one-sided testing, under null hypothesis, the parameter of interest is always on the boundary of the parametric space, and standard results, like one-sided t-test, are not applicable [6].

2.2. Heuristics

2.2.1. Maximum Likelihood Estimation

Example 1 (continued). Unrestricted and restricted maximum likelihood estimate (MLE) of mean 𝜇 of 𝑋₁,…,𝑋_𝑛∼Normal (𝜇,1), when true 𝜇=0.(a)𝜇-unrestricted. In this case, 𝜇 is estimated by 𝑋. 𝑋 is normal (𝜇,1/𝑛) (Figure 1(a));(b)𝜇≥0. In this case, 𝜇 is estimated by ^{𝑋,if𝑋≥0,0,}_if𝑋<0, which is a mixture of point mass at 0 and half-normal (Figure 1(b)).

(a)

(b)

Figure 1 

Distribution of unrestricted (a) and restricted (b) MLE in Example 1.

Example 2 (continued). Figure 2 shows joint distribution of the parameters of cubic multistage model when no true parameters are on the boundary (top panel) and when a true quadratic term = 0 (bottom panel). When no parameters are on the boundary, the joint distribution of parameters is multivariate normal, as expected. When the quadratic parameter is on the boundary, it is half-normal, but the other parameters are affected and are further from normality.

(a)

(b)

Figure 2 

Monte Carlo simulated joint distribution of parameters of a cubic multistage model with no true parameters on the boundary (a) and quadratic term on the boundary (b) in Example 2.

In the simple univariate example above, the distribution of restricted MLE is easily described. However, in a general multivariate case, the joint distribution of MLE is generally not multivariate Normal when just a single parameter has true value on the boundary (Section 3).

2.2.2. Likelihood Ratio Tests

The following example is continuation of Example 1(b).

Example 4. Likelihood ratio test (LRT) for a single normal random variable 𝑋∼𝑁(𝜇,1)(a)Two-sided test 𝐻₀∶𝜇=0 versus 𝐻_𝐴𝜇≠0. The Likelihood Ratio (LR) is (2𝜋)^−1/2exp−(𝑋−0)²/2(2𝜋)^−1/2sup_{𝜇∈(−∞;+∞)}exp−(𝑋−𝜇)²/2=exp−𝑋2²(3) as supremum in the denominator is achieved when 𝜇 is equal to 𝑋. The distribution of −2 * log(LR) is a familiar 𝜒² with 1 degree of freedom.(b)One-sided test 𝐻₀∶𝜇=0 versus 𝐻₁∶𝜇>0. The likelihood ratio is (2𝜋)^−1/2exp−(𝑋−0)²/2(2𝜋)=^−1/2supexp_{𝜇∈[0;+∞)}e−x𝑋p₂−(𝑋−𝜇)²/2/2=⎧⎪⎨⎪⎩⎧⎪⎨⎪⎩1ee1x,x,pp−−𝑋𝑋²₂/2,iiifff𝑋𝑋𝑋≥<<000,2,if𝑋≥0.(4)

In this situation, supremum in the denominator depends on whether 𝑋 is positive or negative, and distribution of −2 * log(LR) is a mixture of point mass at 0 and 𝜒² with 1df.

Example 4 demonstrates that when the LRT is used for one-sided tests, the distribution is not 𝜒² with 1 degree of freedom anymore and can be fairly complicated. Various cases are described in the next section.

3. Results

3.1. Likelihood Ratio Tests

A pioneering theoretical article [2], following mainly on earlier work [8–11], proposed a general approach to deriving distribution of the LRT when some parameters of interest and/or nuisance parameters may be on the boundary of a parameter space. They showed that, under fairly general regularity conditions, the distribution of the LRT depends only on number of parameters of interest and nuisance parameters on the boundary and off the boundary. They also explicitly derived the LRT for several particular situations but were not completely correct for some of these. Sinha et al. [12] and Kopylev and Sinha [13] solved some of the cases in [2] and derived asymptotic distribution of the LRT for some other important special cases. All these results are summarized below.(i)When one or several parameters of interest not on the boundary and some nuisance parameters are on the boundary, the asymptotic distribution of the LRT remains 𝜒² with corresponding degrees of freedom [12].(ii) When one parameter of interest is on the boundary and no nuisance parameters are on the boundary (e.g., Example 3 above with testing AE versus ACE), the asymptotic distribution of the LRT is a 50 : 50 mixture of point mass at 0 and 𝜒² with 1df [2].(iii) When one parameter of interest is on the boundary and one parameter of interest not on the boundary and no nuisance parameters are on the boundary, the asymptotic distribution of the LRT is 50 : 50 mixture of 𝜒² with 1df and 𝜒² with 2 df [2].(iv) When two parameters of interest are on the boundary and no nuisance parameters are on the boundary (e.g., Example 3 above), the asymptotic distribution of the LRT is a mixture of point mass at 0, 𝜒² with 1 df and 𝜒² with 2 df with mixing parameters 𝑝, 1/2 and 1/2−𝑝, where 𝑝=cos⁻¹(𝜌)/2𝜋, 𝜌=𝐼₁₂/sqrt(𝐼₁₁∗𝐼₂₂), and 𝐼 is an information matrix (e.g., [13] for derivation of 𝜌). Unfortunately, this example in [2] has probabilities of the mixture components inverted, most likely a misprint. A lucky erroneous inversion of the sign in calculations [7] allowed them to make the correct recommendation.(v) When one parameter of interest is on the boundary and one nuisance parameter is on the boundary, the asymptotic distribution of the LRT is a mixture of point mass at 0, 𝜒² with 1df and 𝜒² with 2df with mixing parameters 1/2−𝑝, 1/2 and 𝑝, where 𝑝=sin⁻¹(𝜌)/2𝜋, 𝜌=𝐼₁₂/sqrt(𝐼₁₁∗𝐼₂₂)≥0, and 𝐼 is an information matrix. When 𝜌<0, asymptotic distribution is not a mixture of chi-squares but is explicitly derived [13].(vi) When one parameter of interest and two nuisance parameters are on the boundary or two parameters of interest and one nuisance parameter are on the boundary, Kopylev and Sinha [13] derive a straightforward way for simulation of the asymptotic distribution of the LRT in these cases.

3.1.1. Simulations

Table 1 (adapted from [13]) demonstrates that, in case of one parameter of interest and two nuisance parameters on the boundary, distribution of the LRT can be very different from 𝜒² with 1 df and use of 𝜒² with 1 df can be anticonservative.

Table 1 

Upper percentiles of the asymptotic distribution of the LRT in case of one parameter of interest and two nuisance parameters on the boundary, calculated by simulations. Correlations are between parameter of interest and nuisance parameter 1, parameter of interest and nuisance parameter 2, and nuisance parameters 1 and 2. The last row shows corresponding percentiles of the chi-square with one degree of freedom—the asymptotic distribution of the LRT when complications due to boundary parameters are ignored. Percentiles exceeding corresponding percentiles of 𝜒² (1), that is, situations when use of 𝜒² (1) is anticonservative, are bolded.

3.2. Wald

The correct joint asymptotic distribution of the MLE in case of some parameters on the boundary is described in [2], following earlier work [8]. It turns out that the asymptotic distribution of the MLE is not multivariate normal distribution anymore (cf. Example 1). Sinha et al. [12] adapted general statistical theory [2] for the multistage model and showed that asymptotic joint distribution of the MLE can be easily simulated starting from the standard multivariate Normal distribution. They also demonstrated how to construct confidence intervals following [2] and investigated the coverage of Wald-like confidence intervals for the multistage model. For sample sizes common for bioassays, the two-sided coverage of Wald-like intervals was comparable with that of the profile likelihood method for a wide range of scenarios. However, one-sided coverage by Wald-like intervals was found to be different from the nominal coverage [12].

3.3. Bootstrap

Bootstrap approaches have been suggested by many authors for use in the field of risk assessment. Bailer and Smith [14] investigated coverage of one-sided confidence intervals for the risk based upon parametric and nonparametric approaches to bootstrap, for a number of scenarios including true values of parameters on or near the boundary. They found that both parametric and nonparametric bootstraps may be anticonservative for some scenarios and very conservative for others.

3.4. Bayesian

There is only limited literature on Bayesian approaches to boundary parameter problems, especially when nuisance parameters on the boundary are involved. Gelfand et al. [15] recommended use of Bayesian methods when a parameter is constrained. However, this approach had been shown not to work well when distribution of the parameter has a nonzero mass concentrated on a boundary (e.g., [16]). For situations when there is a mass on the boundary, publications [16, 17] suggest a Bayesian approach.

4. Discussion

In the field of science, there is indeed considerable amount of interest in statistical issues when some parameters may lie on the boundaries. However, a number of publications suggest, if one is willing to use a conservative test, to ignore the possibility of parameters on the boundary as these publications claim that using the usual chi-square distribution with appropriate degrees of freedom (as when no parameters are on the boundary) will result in a conservative test (e.g., [18, 19] in the field of genetics, and [20] in the field of psychology). Nevertheless, results in [13] demonstrate that when one or more nuisance parameters are on the boundary, the suggested strategy of ignoring parameters on the boundary may result in anticonservative tests for many different correlation structures between parameters that are on the boundary, as seen in Table 1 illustrating what happens with two nuisance parameters on the boundary.

Therefore, correct asymptotic distribution of the likelihood ratio test must be used whenever a parameter of interest on the boundary, for example, one-sided tests, is encountered, especially if nuisance parameters on the boundary are also present. Asymptotic results for many situations that are routine in applications are summarized in Section 3.1. However, in case a different situation is encountered in applications, the theory [2] and explanations and clarifications [12] can be used to derive correct asymptotic distribution of the LRT for that particular situation. In some cases, that asymptotic distribution may be explicit, but it is likely that in most cases the asymptotic distribution will have to be simulated.

From other approaches to confidence interval construction described in Section 3, the Bayesian approach (e.g., [16, 17]) is the most interesting, but more research is required especially when there are constrained nuisance parameters. Another approach useful in applications would be to try to use a different model without boundary parameters or, if possible, reparameterize existing model so that to avoid situations with parameters on the boundary.

Acknowledgments

The author wishes to thank Professor Bimal Sinha of the Department of Mathematics and Statistics at University of Maryland Baltimore County and John Fox of Office of Research and Development of US EPA for their advice and encouragement on this project. The views expressed in this paper are those of the author and do not necessarily reflect the views or policies of the US Environmental Protection Agency.