Streaming Breusch-Pagan test for large datasets¶
Source: R/performance.R ยท Help: ?performBPTestStreaming
Computes the classical Breusch-Pagan statistic chunk by chunk so that large
datasets can be evaluated without materialising the full auxiliary regression in
memory. The streamed result is algebraically identical to
performBPTest.
Usage¶
Arguments¶
model- A fitted stats::lm object describing the mean structure whose residual variance is to be assessed.
data- The data frame
modelwas fitted to, orNULL, the default, read asperformWhiteTestStreaming()reads it: the response, design, weights and offset come from the fit itself, so a model that transforms its variables, has anoffset()term, or takes itsweightsoroffsetfrom a column of the data needs nodata(since 0.16.0), and data that are given are matched to the observations of the fit. chunk_size- Positive integer giving the number of observations processed per streaming chunk. Smaller values reduce peak memory usage at the expense of additional iteration overhead.
progress- Logical flag controlling whether a textual progress bar is displayed
while chunks are processed. Defaults to
interactive().
Value¶
A htest object mirroring performBPTest and reporting the
chi-squared statistic, degrees of freedom, p-value, and metadata describing the
chunked computation.
Details¶
Each chunk of the regressors, centred at their means, is reduced by a QR
decomposition stacked on the factor of the chunks before it, as in
performWhiteTestStreaming(), since 0.17.0. Since 0.18.0 the factor
leaves out what is left of a regressor that is a combination of those before it
in the rows so far, rounding error, as its details say. Up to 0.16.0 the
cross-products of the regressors were accumulated: those of a regressor 1,000
standard deviations from zero looked collinear with the intercept, and the test
stopped ("Breusch-Pagan auxiliary regression became rank deficient"); and squared
residuals of about \(10^{-10}\) were taken for constant ("requires variation in
squared residuals"). It gives the value of performBPTest() on both.
Sparse cross-products from Matrix are no longer used.
The degrees of freedom are the rank of the auxiliary regression less its
intercept, as in performBPTest() (see its Degrees of freedom).
Since 0.18.0 the test gives them, and the value of performBPTest(), on
a model without an intercept whose columns span the constant, such as the
dummies of every level of a factor in lm(y ~ 0 + group + x1): 8.653 on
4 degrees of freedom, where it stopped ("Breusch-Pagan auxiliary
regression became rank deficient").
Since 0.18.0 the test refuses an auxiliary regression that fits the squared
residuals exactly, as on a paired design, by the rule and with the message and
class of performBPTest() (see its Exact fit), where it
gave 63.25 on 30 degrees of freedom on 30 subjects of two observations. It
refuses it when the design forces it, decided from the squared pseudo-residuals
it carries in the same pass, as performBPTest() decides it; a fit
the data make exact gives a value.
Since 0.16.0 the chunks are rows of the design of the fit, as in
performWhiteTestStreaming(), whose details say what changed.
Supported models¶
It accepts an lm and a glm with the Gaussian family and the identity link, unweighted
or with positive weights, a fitWLS() fit and, since 0.16.0, a rank-deficient fit, which it
tests on the estimable part of its design: it gives the value of the same model fitted without the
aliased columns. It refuses a glm of another family or link, a survey::svyglm() fit, a
fit with a zero weight, a coefficient that is not finite for another reason, data that
lack rows or variables of the fit, and, since 0.16.0, a multivariate lm, a fit changed after it
was fitted and a fit with model = FALSE whose data are gone, unless data gives its
response and fitted values. Which model each test accepts, and why, is set out in
test_results, under Supported models.
See also¶
performBPTest for the standard implementation.