Benchmark heteroTests against reference implementations

Source: man/run_benchmark_suite.Rd · Help: ?run_benchmark_suite

run_benchmark_suite() evaluates a selection of heteroscedasticity diagnostics across varying sample sizes while measuring runtime, memory allocation, and statistical agreement with the lmtest and car packages. The companion generate_benchmark_report() helper aggregates the raw output into tidy summary tables and recommendations.

Usage

run_benchmark_suite(
  sample_sizes = c(100L, 500L, 1000L, 10000L, 100000L, 1000000L),
  tests = NULL,
  replicates = 3L,
  hetero_patterns = c("none", "linear", "group"),
  hetero_strength = 1,
  baseline_packages = c("lmtest", "car"),
  seed = 123L,
  n_predictors = 4L,
  profile_memory = TRUE,
  progress = interactive()
)

generate_benchmark_report(benchmark_results, accuracy_tolerance = 1e-4)

Arguments

sample_sizes

Integer vector of observation counts to benchmark. Values may range from a few hundred observations to one million.

tests

Optional character vector restricting the benchmark to specific diagnostics (currently "breusch_pagan", "koenker", and "ncv").

replicates

Scalar integer applied to all sample sizes or an integer vector matching sample_sizes to control the number of repetitions per configuration.

hetero_patterns

Character vector describing the heteroscedasticity patterns used when generating synthetic data. Supported values are "none", "linear", "group", and "exponential"; the vector is recycled across replications.

hetero_strength

Positive numeric multiplier controlling the severity of variance heterogeneity.

baseline_packages

Character vector of external packages used for comparisons. Missing dependencies are skipped and reported in the metadata.

seed

Integer seed passed to the data generator for reproducibility.

n_predictors

Number of regressors (excluding the intercept) used when simulating benchmark datasets.

profile_memory

Logical; when TRUE and the bench package is installed, the suite records peak memory allocations in megabytes.

progress

Logical indicating whether to display a textual progress bar during execution.

benchmark_results

List returned by run_benchmark_suite().

accuracy_tolerance

Non-negative numeric tolerance for declaring statistical agreement with reference implementations (based on absolute p-value differences).

Value

run_benchmark_suite() returns a list with four elements:

performance

Data frame containing runtime, memory, and inference metrics for every test, sample size, and replicate.

accuracy

Data frame recording absolute differences in p-values and test statistics relative to each baseline implementation.

summary

List of aggregated medians for quick inspection of speed, memory, and accuracy.

metadata

Metadata describing the benchmark configuration (sample sizes, baselines, seed, and availability of optional dependencies).

generate_benchmark_report() converts the raw results into aggregated speed, memory, accuracy, recommendation, and scalability tables alongside the original metadata.

Details

Datasets are generated from a linear regression model with a configurable number of predictors and variance patterns (none, linear, group, or exponential). Each diagnostic from heteroTests is executed alongside equivalent tests from lmtest and car when available. Memory profiling is enabled via bench when installed; otherwise only runtimes are collected. The output is suitable for regression testing (to detect performance regressions) and for building artefacts such as CSV benchmark reports.

Examples

if (requireNamespace("lmtest", quietly = TRUE) &&
    requireNamespace("car", quietly = TRUE)) {
  results <- run_benchmark_suite(
    sample_sizes = c(100L, 250L),
    replicates = 1L,
    profile_memory = FALSE,
    progress = FALSE
  )
  report <- generate_benchmark_report(results)
  head(report$recommendations)
}