-
Notifications
You must be signed in to change notification settings - Fork 7
count data Poisson Rates Exact and Asymptotic Tests
(Note: References below may not be the most appropriate. I did not do an extensive search and might not have gotten the most important articles.)
This relates to several issues in statsmodels github. This parallels similar hypothesis tests, confidence intervals and power calculations for proportions.
Main argument: Asymptotic tests can be quite liberal in small samples especially Wald test. Score tests can also be liberal in some cases. Using an exact small sample, "exact" distributions will improve or maintain the size of hypothesis tests and the coverage of confidence intervals.
Caveat: These exact or approximately exact statistics fully depend on the distributional assumption. There is no misspecification robustness build in.
Target: One, Two or k sample tests for comparing means or related properties of the underlying distribution.
Based on asymptotic normality we have the usual Wald, Score and Likelihood Ratio tests. Additional tests use transformation to obtain a statistic that is closer to normally distributed.
Reference: Gu et al. 2008 have 6 tests for comparing two independent Poisson rates, or more precisely the ratio of two Poisson rates. score and wald on difference, score and wald with log transformation, a test with a variance stabilizing square root transformation and the Wald test.
Status: I have 3 of the 6: score, wald and sqrt-transformed.
Poisson as a member of the LEF family with natural link/parameter allows for conditioning that removes the nuisance parameter. The test conditions on the sum of the observed counts of the two samples. The resulting statistics is has univariate Binomial distribution and can be calculated with an exact Binomial test. As an exact test it always maintains the size but is often very conservative has low power. Midpoint p-values get closer to nominal size but can have small violations of the size. (Related Agresti and some authors argue in favor of average size, and mid-p p-values are becoming popular for this.)
Gu et al. has formulas for one sided tests.
Status: conditional exact and mid-p tests are implemented in script
In the following we have two possible definitions of "exact". We are using the exact small sample distribution given our distributional and sampling assumptions, but not all versions here are guaranteed to maintain the size of the test or the coverage of a confidence interval. Violations are usually small if they occur.
When we do not condition out the nuisance parameter, then we have several choices for how to treat it in calculating the p-values of a test or the confidence intervals. The choices are to use a estimate for the nuisance parameters or to find the largest (sup) p-value over a set of values for the nuisance parameters. Additionally, there are several options for the test statistic that is used in the p-value. The main choices for this are using a score, Wald, likelihood ratio or conditional exact test statistic which could be a p-value.
The following orders by the treatment of the nuisance parameter. Within each we can have different versions depending on the test statistic that is used to order the sample space in less or more outlying than the value in the observed sample.
E-Tests
In E-Tests we replace the nuisance parameter by an estimate and calculate the probability of more outlying samples using the assumed data generating distribution.
References: Krishnamoorthy and Thomson 2004 are the main reference for using the E-test for the two sample Poisson test. They use the Wald test statistic. Gu et all 2008 have formulas and compare E-tests based on Wald, Score ... E-tests in their simulations
M-test
The standard way to make a unconditional exact test to always maintain the size is to maximize the p-value over the entire parameter space of the nuisance parameter, i.e. a sup test. This is more difficult in the Poisson case than in the binomial case because the nuisance parameter is not in a compact space.
The only paper I have read is Liu Hsueh 2015 "Exact tests of the superiority under the Poisson distribution" which show that the search for the largest p-value can be confined to a point or to a compact interval in the superiority, one sided test (both Null and Alternative hypotheses are inequalities).
The more common approach in the case of Poisson is to use a restricted parameter space that is derived from a confidence interval based on the parameter estimates, which is described next.
Confidence interval p-values - Berger, Boos
Searching the full parameter space for the largest p-value might be impossible or it includes values that are implausible based on the current sample. The Berger and Boos approach restricts the parameter space to one based on confidence intervals with a small alpha, for example 1e-4. An added correction factor makes hypothesis test in this case exact in the sense of always maintaining the size.
Shan 2015 "Exact unconditional testing procedures for comparing two independent Poisson rates" uses this.
E+M or E+B
Lloyd introduce a method that combines the E and the M estimates in sequence. First a p-value is calculated base on the M-test, then the maximum of these p-values is calculated either for the entire space of the nuisance parameter or a Berger-Boos confidence interval restriction to it.
Shan 2015 uses the E+B test for the two sample Poisson test. The approach is computationally intensive, has good properties, but improves only little over the BB test using the score statistic. This is similar to results (AFAIR) for E+M and E+B tests for two sample tests of proportions.
Gu et al 2008 has the formulas for the power calculations of the tests that they include. NCSS/PASS is using those.
(I haven't looked at any details yet.)
Confidence intervals can be obtained following the same approaches as for the hypothesis tests. One-sided confidence intervals immediately follow from the one sided tests. Two sided confidence intervals are more ambiguous because of the asymmetry in the exact distributions.
Fay in R journal (year ?) provides an overview of R procedures that provide two-sided confidence intervals that are consistent or inconsistent with the corresponding hypothesis tests.
(I haven't looked in details at confidence intervals for Poisson testing yet.)
Allmost all the references above are for one-sided tests. Some take a null value different from inequality into account, some do not. Tests based on asymptotic distribution and conditional exact tests have standard handling of the alternatives, and non-zero null values. Unconditional tests using the exact distribution might be more difficult or ambiguous in how to link one-sided to two-sided tests and confidence intervals.
(so far)
- Krishnamoorthy and Thomson 2004
- Gu et al 2008
- Shan 2015
- Liu Hsueh 2015
- Fay ... ?
- NCSS/PASS documentation chapter 437
- ???
- some class web sites