Memory-efficient White test for large datasets

Source: R/performance.R · Help: ?performWhiteTestStreaming

Computes White's statistic via chunked cross-products rather than fitting the full auxiliary regression in memory. This streaming approach allows the test to scale to datasets that would otherwise exhaust available RAM.

Usage

performWhiteTestStreaming(
  model,
  data = NULL,
  chunk_size = 10000,
  cross_products = TRUE,
  max_interactions = 10,
  progress = interactive()
)

Arguments

model

A fitted stats::lm object representing the mean specification to be diagnosed.

data

A base::data.frame (or object coercible to one) containing the variables referenced by model. It must include the observations used to fit model and will be checked for missing values. When NULL, the default, the model frame of model is used. It holds the terms of the model rather than its variables (log(x) rather than x), so a model that transforms a variable needs data, as does a model with an offset() term, and so may a fit whose weights or offset argument is a column of the data, which the model frame holds as (weights) or (offset).

chunk_size

Positive integer specifying the number of observations per chunk. Smaller values reduce memory usage at the expense of additional iteration overhead.

cross_products

Logical scalar indicating whether to include all pairwise cross-products of the regressors in the auxiliary regression. Defaults to TRUE and should remain enabled unless dimensionality makes the regression unstable.

max_interactions

Single positive integer giving the maximum number of original predictors for which cross-products are generated. When the number of regressors exceeds this threshold, cross-products are dropped to avoid explosive growth in columns. Defaults to 10.

progress

Logical flag indicating whether a progress bar should be displayed while streaming the data. Defaults to interactive().

Value

A htest object containing the chi-squared statistic, p-value, and metadata about the chunked computation.

Details

The streaming implementation iteratively builds the cross-product matrices required for the auxiliary regression without materialising the full design matrix. Each chunk contributes to \(X'X\), \(X'y\), and summary statistics for the response. The final \(n R^2\) statistic is then computed exactly as in the standard White test, ensuring numerical equivalence while dramatically reducing peak memory usage. When Matrix is installed, sparse cross-products are leveraged automatically for large chunks.

References

White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroscedasticity. Econometrica, 48(4), 817–838.

See also

performWhiteTest() for the exact computation and performWhiteTestRobust() for enhanced reporting.

Examples

data(mtcars)
performWhiteTestStreaming(lm(mpg ~ wt + qsec, data = mtcars), mtcars, chunk_size = 16)