Advances in Decision Sciences

Advances in Decision Sciences / 2008 / Article

Research Article | Open Access

Volume 2008 |Article ID 463781 | https://doi.org/10.1155/2008/463781

D. J. Best, J. C. W. Rayner, O. Thas, "Tests of Fit for the Logarithmic Distribution", Advances in Decision Sciences, vol. 2008, Article ID 463781, 8 pages, 2008. https://doi.org/10.1155/2008/463781

Tests of Fit for the Logarithmic Distribution

Academic Editor: Chin Lai
Received21 Sep 2007
Accepted25 Feb 2008
Published19 Mar 2008

Abstract

Smooth tests for the logarithmic distribution are compared with three tests: the first is a test due to Epps and is based on a probability generating function, the second is the Anderson-Darling test, and the third is due to Klar and is based on the empirical integrated distribution function. These tests all have substantially better power than the traditional Pearson-Fisher 𝑋2 test of fit for the logarithmic. These traditional chi-squared tests are the only logarithmic tests of fit commonly applied by ecologists and other scientists.

1. Introduction

Species diversity data can sometimes be modeled by a zero-truncated negative binomial distribution with index parameter near zero. Fisher et al. [1] examined the limit as the index parameter of this distribution approached zero and so derived the logarithmic distribution. A random variable X has this distribution if and only if 𝑃(𝑋=π‘₯)=𝑝π‘₯=𝛾𝛽π‘₯π‘₯,π‘₯=1,2,3,…(1.1) in which 0 < Ξ² < 1 and 𝛾=βˆ’1/ln(1βˆ’π›½). The logarithmic or log-series distribution is often applied to species diversity data.

As an example of species diversity data which the logarithmic distribution may fit, consider the following data on insect catches from the Sierra Tarahuma, Mexico, reported by Aldrete [2]. Ten species were caught precisely once, three species were caught precisely twice, and so on according to Table 1. The expected line in Table 1 shows the expected counts on fitting a logarithmic distribution. For these data, the alpha index (𝐴𝐼) of diversity is 9.01, where ̂̂𝛽𝐴𝐼=𝑛(1βˆ’π›½)/ in which n is the total number of insects and ̂𝛽 is the maximum likelihood estimator of Ξ². The AI quoted here is defined in Krebs [3, 12.13]; for a discussion of the index of diversity see Krebs [3, Section 12.4.1]. Note that larger AI implies more diversity while smaller AI implies less.


Times caught12345681011121316256995At least 13
No. of species10342221111112116
Expected8.784.282.782.031.581.281.070.910.620.55β€”β€”β€”β€”β€”7.63

It would seem sensible to test the data for consistency with the logarithmic distribution before quoting an AI value. However, the only statistic that appears to be commonly used by ecologists as a test of fit for the logarithmic distribution is the so-called chi-squared test, which, as Krebs [3, Section 12.4.1] notes, may not always have good power.

β€œThe goodness-of-fit of the logarithmic series … can be tested by the usual chi-squared goodness-of-fit test … this means low power …. Thus in most cases the decision to use the logarithmic series … must be made on ecological grounds, rather than statistical goodness-of-fit criteria.” [3, page 429].

In this paper, we will examine a number of statistical tests which are considerably more powerful than the traditional Pearson-Fisher 𝑋2 test. These include tests of fit based on components of Neyman’s smooth test statistic, the Anderson-Darling test discussed by Lockhart et al. [4], an empirical integrated distribution function test given by Klar [5], and a test due to Epps [6] based on a probability generating function (pgf). We suggest that these could be used to help make a decision as to whether or not to use the logarithmic series based on statistical as well as ecological criteria. In particular, the dispersion statistic, 𝑉2, defined subsequently, should be useful for identifying the not infrequent case of data for which the abundance species are more abundant than predicted by the logarithmic series.

Our second example is included for its somewhat curious interest and is not involved with conventional species diversity. Collins and Hand [7] have counted the number of times, in the period 1983 to 2000, that a Hollywood film won one Oscar, two Oscars, three Oscars, and so on, giving the data in Table 2. The film with 11 Oscars was β€œTitanic.”


Oscars1234567891011At least 9
No. of films1112914961412013
Expected103361795321β€”β€”β€”2

2. Tests of Fit for the Logarithmic

A discussion of smooth tests of fit and their components, particularly when testing for the logarithmic, is given in Appendix A. These tests may be derived as a routine application of Rayner and Best [8, Theorem 6.1.1]. The first-order component 𝑉1 is identically zero when Ξ² is estimated by maximum likelihood, or, equivalently, by method of moments. The test based on the component 𝑉2 suggests whether or not the data are consistent with the logarithmic variance while the test based on 𝑉3 suggests whether or not the data are consistent with logarithmic moments up to the third. To find p-values for these tests, it is suggested that the parametric bootstrap is to be used as convergence to the asymptotic standard normal distribution is very slow. See GΓΌrtler and Henze [9] and Appendix B for details of the parametric bootstrap in a goodness of fit context.

In Section 3, we give powers for the Anderson-Darling test based on the statistic 𝐴2=βˆžξ“π‘—=1𝑍2π‘—π‘π‘—β„Žπ‘—ξ‚€1βˆ’β„Žπ‘—ξ‚(2.1) in which 𝑍𝑗=βˆ‘π‘—π‘₯=1(𝑂π‘₯βˆ’π‘›π‘π‘₯),β„Žπ‘—=βˆ‘π‘—π‘₯=1̂𝑝π‘₯, and Ox is the number of observations equal to x. Summation is halted when x is the minimum such that Ox = 0 and βˆ‘βˆžπ‘—=π‘₯̂𝑝𝑗<10βˆ’3/𝑛. We also give powers for a test given by Klar [5], based on the empirical integrated distribution function with test statistic 𝑇𝑛=βˆšπ‘›|sup1β‰€π‘˜β‰€π‘€π‘˜ξ“π‘—=1𝑍𝑗|(2.2) in which M is the largest observation. Finally, for comparison purposes, we quote powers of the pgf and 𝑋2 tests given by Epps [6].

3. Power Comparisons

Random deviates from the logarithmic (L), positive Poisson (P+), and positive geometric (G+) distributions were generated using IMSL [10] routines RNLGR, RNPOI, and RNGEO. Random zeta deviates (Z) and random Yule deviates (Y) were found using algorithms of Devroye [11, pages 551 and 553]. Table 3 gives powers for the same alternatives as used by Epps [6], but with the addition of two Yule alternatives. For convenience, we reproduce the powers given by Epps for his pgf and 𝑋2 tests. The powers we give for 𝐴2, Tn, 𝑉22, 𝑉23, and 𝑉22 + 𝑉23 were found using parametric bootstrap with 1000 simulations both for the inner and the outer loops. Note that the calculation of 𝑉3 can involve large numbers, and calculation of the pgf and 𝐴2 statistics can involve small numbers. Care with rounding error may be needed. The statistics Tn and 𝑉22 are less prone to rounding error. Klar [5] notes that the smooth tests, the 𝑋2 test, and the pgf test are not consistent against all alternatives.


AlternativeV^22V^32V^22 + V^32PGFX2TnA2

P+(1.05)0.390.540.510.560.430.500.57
P+(1.2)0.490.680.630.720.580.610.73
P+(1.3)0.570.780.690.790.630.680.79
G+(0.25)0.450.070.050.780.510.520.77
G+(0.33)0.320.370.300.610.390.380.58
G+(0.4)0.210.390.330.460.300.300.48
Z(1.0)0.850.370.540.840.080.730.72
Z(1.3)0.740.630.720.720.160.680.69
Z(2.0)0.500.380.430.440.170.480.42
Y(2.75)0.550.380.50β€”β€”0.490.43
Y(3.0)0.500.350.45β€”β€”0.460.35

From Table 3 our powers for Tn are a little greater than those of Klar [5], and we observe that the power for the Z(1.0) alternative is 0.73, somewhat larger than the 0.40 reported by Klar [5]. Also from Table 3, we see that the 𝑋2 test is not generally competitive with the other tests.

The test based on 𝑉22 + 𝑉23 performs reasonably well. The test based on the Tn statistic has power a little less than that for the pgf- and 𝐴2-based tests. An advantage of the test based on Tn is that Klar [5] showed it is consistent.

The test based on the dispersion statistic 𝑉22 has good power for the zeta and Yule alternatives, while the 𝐴2 and pgf tests generally have competitive powers for all alternatives. Clearly, the test based on 𝑉22 will not have good power for alternatives with similar dispersion to the logarithmic distribution. If the test based on 𝑉22 is not significant but that based on 𝑉23 is, this suggests a skewness departure from the logarithmic distribution. However, if the test based on 𝑉22 is significant, then this suggests that the test based on 𝑉23 may be significant due to either a dispersion or a skewness departure of the data from the logarithmic distribution. Notice that we say the test based on 𝑉23 suggests how the data deviate from the logarithmic. We do not claim that the data actually do deviate in this manner. See the comments of Henze and Klar [12].

On the basis of Table 3 powers, we suggest that the tests based on 𝐴2 and Klar’s Tn are considered as tests of fit for the logarithmic distribution. These tests have good power and are consistent. We recommend that the tests based on Tn and 𝐴2 are augmented by the use of 𝑉22 and 𝑉23 in a data analytic fashion.

4. Examples

In the following parametric bootstrap, p-values for the tests based on 𝑉22, 𝑉23, 𝐴2, and Tn are given. These use 1000 random samples of the logarithmic distribution with parameter ̂𝛽 as given below. We give 𝑉2 and 𝑉3 values because they may suggest how the data deviate from the logarithmic. We give the 𝐴2 and Tn values because the tests based on these statistics are consistent and have good power.

4.1. Insect Data

From the data in Table 1, we find ̂𝑉𝛽=0.9743,𝐴𝐼=9.0104,2=0.4879 with p-value 0.52, 𝑉3=βˆ’0.8791 with p-value 0.19, 𝐴2=0.2613 with p-value 0.82, and 𝑇𝑛=204.8192 with p-value 0.33. It appears that the logarithmic distribution is a good fit. In agreement with this, the Pearson-Fisher statistic takes the value 4.56 on 11 degrees of freedom when data greater than 12 have been combined.

4.2. Oscars Data

We find that 𝑉2𝑉=0.49,3=βˆ’1.27, and 𝑋2=7.40 on 7 degrees of freedom if the classes greater than or equal to 9 are combined. The corresponding p-values are 0.58, 0.08, and 0.62. It appears that a logarithmic distribution with ̂𝛽 = 0.7044 fits the data reasonably well. However, the p-value for 𝑉3 suggests that the data may not be quite as skewed as would be expected for the logarithmic distribution. Collins and Hand [7] suggest a Yule distribution fits the data well. In addition, we note that 𝐴2=0.9727 with p-value 0.12 and 𝑇𝑛=120.0203 with p-value 0.26.

5. Conclusion

In this paper, we have examined a number of statistical tests which are considerably more powerful than the traditional Pearson-Fisher 𝑋2 test. We suggest that these could be used to help make a decision as to whether or not to use the logarithmic series based on statistical as well as ecological criteria. A test of fit could be done before quoting the index of diversity. In particular, the dispersion statistic, 𝑉2, should be useful for identifying the not infrequent case of data for which the abundance species are more abundant than predicted by the logarithmic series.

Appendices

A. The Smooth Tests and Their Components

For distributions from exponential families the smooth tests can be derived as score statistics for testing H0: ΞΈ = 0 against K : ΞΈβ‰  0 for observations 𝑋1,…,𝑋𝑛 from the model πΆξ€·ξ€Έπœƒ,𝛽exp{π‘ž+π‘˜ξ“π‘–=π‘ž+1πœƒπ‘–π‘”π‘–ξ€·ξ€Έξ€·ξ€Έπ‘₯;𝛽}𝑓π‘₯;𝛽(A.1) in which

(i)𝑓(π‘₯;𝛽) is a probability density function that depends on a π‘žΓ—1 vector of nuisance parameters Ξ² and for which we test;(ii){𝑔𝑖(π‘₯;𝛽)} is a complete orthonormal set on 𝑓(π‘₯;𝛽);(iii)𝐢(πœƒ;𝛽) is a normalizing constant. For details see Rayner and Best [8].

The score test statistic has a particularly appealing form ξπ‘†π‘˜=𝑉2π‘ž+1𝑉+β‹―+2π‘ž+π‘˜,(A.2) where ξπ‘‰π‘Ÿ=1βˆšπ‘›π‘›ξ“π‘—=1π‘”π‘Ÿξ‚€π‘₯𝑗;̂𝛽,π‘Ÿ=π‘ž+1,…,π‘ž+π‘˜.(A.3) Here, ̂𝛽 is the maximum likelihood estimator of Ξ² assuming that 𝐻0 is true.

To define ξπ‘‰π‘Ÿ, central moments of 𝑓(π‘₯;𝛽) up to order 2r are required. For example, to directly define components up to 𝑉3 to test for the logarithmic, we note that the equation to estimate Ξ² is 𝑉1≑ 0, as discussed below. To define 𝑉2 and 𝑉3 requires 𝑔2(π‘₯;𝛽) and 𝑔3(π‘₯;𝛽), which in turn require central logarithmic moments up to order six. These are given by πœ‡=𝛾𝛽,πœ‡(1βˆ’π›½)2=𝛾𝛽(1βˆ’π›Ύπ›½)(1βˆ’π›½)2,πœ‡3=𝛾𝛽1+π›½βˆ’3𝛾𝛽+2𝛾2𝛽2(1βˆ’π›½)3,πœ‡4=𝛾𝛽1+4π›½βˆ’4𝛾𝛽+𝛽2βˆ’4𝛾𝛽2+6𝛾2𝛽2βˆ’3𝛾3𝛽3(1βˆ’π›½)4,πœ‡5=𝛾𝛽1+11𝛽+11𝛽2βˆ’5π›Ύπ›½βˆ’20𝛾𝛽2+10𝛾2𝛽2+𝛽3βˆ’5𝛾𝛽3+10𝛾2𝛽3βˆ’10𝛾3𝛽3+4𝛾4𝛽4(1βˆ’π›½)5,πœ‡6ξ‚€=𝛾𝛽1+26π›½βˆ’6𝛾𝛽+66𝛽2βˆ’66𝛾𝛽2+15𝛾2𝛽2+26𝛽3βˆ’66𝛾𝛽3+60𝛾2𝛽3βˆ’20𝛾3𝛽3+𝛽4βˆ’6𝛾𝛽4+15𝛾2𝛽4βˆ’20𝛾3𝛽4+15𝛾4𝛽4βˆ’5𝛾5𝛽5/(1βˆ’π›½)6.(A.4) To calculate further orthonormal polynomials directly, we could use the result that for the logarithmic, π‘˜π‘Ÿ+1=π›½πœ•π‘˜π‘Ÿ/πœ•π›½ generates cumulants and hence central moments, but it is more efficient to use recurrence as described in Rayner et al. [13]. Proceeding directly, the first six central moments can be used to calculate 𝑔1(π‘₯;𝛽)=(π‘₯βˆ’πœ‡)βˆšπœ‡2,𝑔2(π‘₯;𝛽)=π‘₯βˆ’πœ‡2βˆ’πœ‡3ξ€·ξ€Έπ‘₯βˆ’πœ‡/πœ‡2βˆ’πœ‡2ξ‚‡ξ”πœ‡4βˆ’πœ‡23/πœ‡2βˆ’πœ‡22,𝑔3ξ€·ξ€Έ(π‘₯;𝛽)=π‘₯βˆ’πœ‡3ξ€·ξ€Έβˆ’π‘Žπ‘₯βˆ’πœ‡2ξ€·ξ€Έβˆ’π‘π‘₯βˆ’πœ‡βˆ’π‘ξ‚™πœ‡6βˆ’2π‘Žπœ‡5+ξ‚€π‘Ž2ξ‚πœ‡βˆ’2𝑏4ξ€·ξ€Έπœ‡+2π‘Žπ‘βˆ’π‘3+𝑏2ξ‚πœ‡+2π‘Žπ‘2+𝑐2(A.5) in which ξ€·πœ‡π‘Ž=5βˆ’πœ‡3πœ‡4/πœ‡2βˆ’πœ‡2πœ‡3𝑑,ξ‚€πœ‡π‘=24/πœ‡2βˆ’πœ‡2πœ‡4βˆ’πœ‡3πœ‡5/πœ‡2+πœ‡23𝑑,𝑐=2πœ‡3πœ‡4βˆ’πœ‡33/πœ‡2βˆ’πœ‡2πœ‡5𝑑,𝑑=πœ‡4βˆ’πœ‡23πœ‡2βˆ’πœ‡22.(A.6) These formulas give the first three orthonormal polynomials for any univariate distribution.

The components ξπ‘‰π‘Ÿ can be called smooth components as they are analogous to the components of the smooth test for uniformity introduced by Neyman [14]. His smooth components also used orthonormal polynomials. When testing for distributions from exponential families these components are asymptotically independent and asymptotically have the standard normal distribution.

For the logarithmic distribution, the maximum likelihood and method of moments estimators ̂𝛽 of Ξ² coincide, given by 𝑉1≑ 0 or ̂𝛽𝑋=βˆ’ξ‚€Μ‚π›½ξ‚ξ‚€Μ‚π›½ξ‚=Μ‚1βˆ’ln1βˆ’π›½Μ‚π›Ύξ‚€Μ‚π›½ξ‚1βˆ’=ξπœ‡.(A.7) To solve this equation, the Newton-Raphson algorithm can be used. An initial estimate of ̂𝛽 and other details helpful in the solution are given in Birch [15]. Note also that for the logarithmic, 𝑉2 is proportional to (π‘š2βˆ’ξπœ‡2) where π‘š2=βˆ‘π‘›π‘—=1(π‘₯π‘—βˆ’π‘₯)2/𝑛, so the test based on 𝑉2 tests for the dispersion of the logarithmic distribution. Similarly, if π‘š3=βˆ‘π‘›π‘—=1(π‘₯π‘—βˆ’π‘₯)3/𝑛, then the numerator of 𝑉3 is of the form 𝑛(π‘š3βˆ’π‘Žπ‘š2βˆ’π‘), so the test based on 𝑉3 assesses whether the data are consistent with moments of the logarithmic up to the third.

B. P-Values via the Parametric Bootstrap

GΓΌrtler and Henze [9, page 223] suggest that p-values can be obtained using an analogue of the parametric bootstrap. If Wn denotes a test statistic, calculate π‘€π‘›βˆΆ=π‘Šπ‘›(π‘₯1,π‘₯2,…,π‘₯𝑛) where π‘₯1,π‘₯2,…,π‘₯𝑛 denote, as usual, the data. Find an estimate ̂𝛽 from the data and conditional on this estimate, generate 𝐡=10000 say pseudorandom samples of size n, each having the logarithmic (̂𝛽) distribution. For 𝑗=1,…,𝐡 compute the value π‘Šβˆ—π‘›,𝑗 on each random sample. The parametric bootstrap p-value is then the proportion of the π‘Šβˆ—π‘›,𝑗 that are at least the observed wn, namely, βˆ‘π΅π‘—=1𝐼(π‘Šβˆ—π‘›,𝑗β‰₯𝑀𝑛)/𝐡.

The above requires random logarithmic (Ξ²) values. Devroye [11, page 547] outlines an algorithm for generating random logarithmic deviates. Alternatively, the routine RNLGR from IMSL [10] can be used. To obtain p-values for two-tailed tests proceed as above and find the p-value, say P. Then if 𝑃≀0.5, the two-tailed p-value is 2P, while if 𝑃>0.5, the two-tailed p-value is 2(1βˆ’π‘ƒ).

Acknowledgment

Research of Olivier Thas was supported by IAP research network Grant no. P6/03 of the Belgian Government (Belgian Science Policy).

References

  1. R. A. Fisher, A. S. Corbet, and C. B. Williams, β€œThe relation between the number of species and the number of individuals in a random sample of an animal population,” Journal of Animal Ecology, vol. 12, no. 1, pp. 42–58, 1943. View at: Publisher Site | Google Scholar
  2. A. N. G. Aldrete, β€œPsocoptera (insecta) from the Sierra tarahumara, Chihuahua, Mexico,” Anales del Instituto de BiologΓ­a, Universidad Nacional AutΓ³noma de MΓ©xico, Serie ZoologΓ­a, vol. 73, no. 2, pp. 145–156, 2002. View at: Google Scholar
  3. C. J. Krebs, Ecological Methodology, Addison Wesley Longman, New York, NY, USA, 1998.
  4. R. A. Lockhart, J. J. Spinelli, and M. A. Stephens, β€œCramΓ©r-von Mises statistics for discrete distributions with unknown parameters,” Canadian Journal of Statistics, vol. 35, no. 1, pp. 125–133, 2007. View at: Google Scholar | Zentralblatt MATH | MathSciNet
  5. B. Klar, β€œGoodness-of-fit tests for discrete models based on the integrated distribution function,” Metrika, vol. 49, no. 1, pp. 53–69, 1999. View at: Google Scholar | Zentralblatt MATH | MathSciNet
  6. T. W. Epps, β€œA test of fit for lattice distributions,” Communications in Statistics: Theory and Methods, vol. 24, no. 6, pp. 1455–1479, 1995. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  7. A. Collins and C. Hand, β€œVote clustering in tournaments: what can Oscar tell us?” Creativity Research Journal, vol. 18, no. 4, pp. 427–434, 2006. View at: Publisher Site | Google Scholar
  8. J. C. W. Rayner and D. J. Best, Smooth Tests of Goodness of Fit, The Clarendon Press, Oxford University Press, New York, NY, USA, 1989. View at: Zentralblatt MATH | MathSciNet
  9. N. GΓΌrtler and N. Henze, β€œRecent and classical goodness-of-fit tests for the Poisson distribution,” Journal of Statistical Planning and Inference, vol. 90, no. 2, pp. 207–225, 2000. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  10. IMSL, β€œUsers' Manual,” IMSL, Houston, Tex, USA, 1995. View at: Google Scholar
  11. L. Devroye, Non-Uniform Random Variate Generation, Springer, New York, NY, USA, 1986. View at: Zentralblatt MATH | MathSciNet
  12. N. Henze and B. Klar, β€œProperly rescaled components of smooth tests of fit are diagnostic,” Australian & New Zealand Journal of Statistics, vol. 38, no. 1, pp. 61–74, 1996. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  13. J. C. W. Rayner, O. Thas, and B. De Boeck, β€œA generalised Emerson recurrence relation,” to appear in Australian & New Zealand Journal of Statistics. View at: Google Scholar
  14. J. Neyman, β€œβ€œSmooth test” for goodness of fit,” Skandinavisk Aktuarietidskrift, vol. 20, pp. 149–199, 1937. View at: Google Scholar | Zentralblatt MATH
  15. M. W. Birch, β€œ194 Note: an algorithm for the logarithmic series distribution,” Biometrics, vol. 19, no. 4, pp. 651–652, 1963. View at: Publisher Site | Google Scholar | Zentralblatt MATH

Copyright Β© 2008 D. J. Best et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


More related articles

 PDF Download Citation Citation
 Download other formatsMore
 Order printed copiesOrder
Views1886
Downloads358
Citations

Related articles

We are committed to sharing findings related to COVID-19 as quickly as possible. We will be providing unlimited waivers of publication charges for accepted research articles as well as case reports and case series related to COVID-19. Review articles are excluded from this waiver policy. Sign up here as a reviewer to help fast-track new submissions.