Introduction to Nonparametric Tests

In statistical analysis, nonparametric tests are invaluable tools, especially when the assumptions of parametric tests (like normality or homoscedasticity) are violated. Among these, the Mann-Whitney U test, also known as the Wilcoxon rank-sum test, is one of the most widely used for comparing two independent groups. The test is often lauded for its simplicity and robustness, with many researchers mistakenly believing that it directly compares the medians of the two groups. However, this common interpretation overlooks the true nature of the test and can lead to incorrect conclusions, particularly when the data distributions differ in ways beyond central tendency.

This article delves into the limitations of the Mann-Whitney test, especially in the context of comparing medians, and contrasts it with quantile regression—a method that explicitly focuses on differences in specific quantiles, such as the median. Through simulations and real-world examples, we will explore scenarios where the Mann-Whitney test may lead to misleading results and discuss alternative methods that offer more reliable insights.

Quantile Regression vs. Mann-Whitney: What Do They Test?

Understanding Quantile Regression

Quantile regression is a powerful statistical technique that extends the concept of linear regression to quantiles of the response variable. While ordinary least squares (OLS) regression focuses on estimating the mean of the dependent variable, quantile regression allows for the estimation of various quantiles, such as the median (50th percentile), quartiles, or any other percentile of interest. This flexibility makes quantile regression particularly useful when the distribution of the dependent variable is skewed or when we are interested in the effects of covariates at different points in the distribution.

In the context of comparing two groups, quantile regression at the median provides a direct test of the difference in medians between the groups. This is particularly important when the median is a more representative measure of central tendency than the mean, especially in skewed distributions.

The Mann-Whitney Test: Beyond Medians

The Mann-Whitney U test is often described as a nonparametric alternative to the two-sample t-test, designed to compare the distributions of two independent groups. However, the test does not specifically compare medians. Instead, it tests the null hypothesis that one group is stochastically greater than the other. This means that it assesses whether the probability of a randomly chosen observation from one group being larger than a randomly chosen observation from the other group is equal to 0.5.

In simpler terms, the Mann-Whitney test evaluates the overall distribution of the data, including aspects like shape and spread, rather than focusing solely on the central tendency. As a result, the test can yield significant results even when the medians of the two groups are identical, especially if one group has a larger spread or is more skewed than the other.

Practical Example: Identical Medians, Different Variances

Consider a situation where we have two groups of data, each with an identical median but different variances. For example, group 1 might have a narrow distribution centered around the median, while group 2 has a broader distribution. If we apply the Mann-Whitney test to these data, it might indicate a significant difference between the groups, not because their medians differ, but because of the difference in spread.

# Generating the data
set.seed(1234)
v5 <- rlnorm(500, 0, 1)

# Scaling the data to increase dispersion while keeping medians constant
variance_factor <- 6
median_v5 <- median(v5)
scaled_dev <- (v5 - median_v5) * sqrt(variance_factor)
v6 <- (median_v5 + scaled_dev)

Running the Mann-Whitney test on these data:

wilcox.test(v5, v6, conf.int = TRUE)

The Mann-Whitney test may yield a significant p-value, suggesting a difference between the groups. However, this result is driven by the difference in variance rather than a difference in medians.

Simulation Study: When Medians Are Misleading

Simulation Setup

To better understand when and why the Mann-Whitney test might fail to provide accurate insights, we conducted a simulation study. In this study, we generated 200 pairs of datasets, each with identical medians but different variances. The goal was to evaluate how often the Mann-Whitney test would incorrectly suggest a significant difference in medians.

For each pair of datasets, we performed the Mann-Whitney test, quantile regression, and the Brunner-Munzel test—a more robust alternative that accounts for differences in variances.

set.seed(1234)
N <- 200
failed_comparisons <- 0

sapply(seq(N), function(i) {
   x1 <- rlnorm(500, 0, 1)
   variance_factor <- 6
   median_x1 <- median(x1)
   scaled_dev <- (x1 - median_x1) * sqrt(variance_factor)
   x2 <- (median_x1 + scaled_dev)
 
   diff_med <- median(x1) - median(x2)
   
   if(diff_med <= .Machine$double.eps) {
     wilcox.test(x1, x2)$p.value
   } else {
     print(paste("Ouch! Medians differ by:", diff_med, "which is more than my machine epsilon"))
     failed_comparisons <<- failed_comparisons + 1
   }
}) -> sim_p

# Summary of simulation results
sprintf("Fraction of rejections for same medians: %d / %d (%.2f%%)", 
         sum(sim_p <= 0.05),
         N-failed_comparisons,
         100*(sum(sim_p <= 0.05) / (N-failed_comparisons)))

Results and Discussion

The results of this simulation study were striking. Despite the medians being numerically identical, the Mann-Whitney test rejected the null hypothesis of no difference in 33.16% of the cases. This high rate of false positives is concerning, especially when considering that the only difference between the groups was in their variances.

On the other hand, quantile regression at the median consistently returned p-values close to 1, correctly indicating no significant difference in medians. The Brunner-Munzel test also provided more accurate results, though it was still somewhat influenced by the difference in variances.

These findings underscore the importance of understanding the assumptions and limitations of the statistical tests we use. While the Mann-Whitney test can be a valuable tool in many scenarios, it is not a reliable method for comparing medians when there are differences in the distributions’ spreads.

Advanced Considerations: Dispersion and Stochastic Dominance

The Role of Variance in Nonparametric Tests

The simulation study highlighted a critical issue: the sensitivity of the Mann-Whitney test to differences in variance. This sensitivity can lead to misleading conclusions, particularly in studies where the primary interest is in comparing central tendencies, such as medians.

The Brunner-Munzel test offers a more robust alternative in such cases. Unlike the Mann-Whitney test, it accounts for differences in variance between the groups, providing a more accurate assessment of whether one group stochastically dominates the other.

Practical Implications and Best Practices

When comparing two groups, especially when the distributions are not identical in shape or spread, it is crucial to choose the appropriate statistical test. If the goal is to compare medians, quantile regression should be the method of choice. If there is concern about differences in variance, the Brunner-Munzel test is a better alternative to the Mann-Whitney test.

Additionally, researchers should be cautious about relying solely on nonparametric tests without considering the underlying distributional characteristics of the data. Understanding the specific hypothesis each test evaluates is essential for making valid inferences.

Conclusion

The Mann-Whitney U test, while widely used, is often misunderstood as a test of medians. This misconception can lead to incorrect conclusions, particularly in cases where the distributions differ in variance or shape. Through a detailed examination of quantile regression and the Brunner-Munzel test, we have shown that these methods provide more accurate and reliable results when comparing central tendencies or dealing with unequal variances.

In statistical practice, the choice of test should be guided by the specific research question and the characteristics of the data. By understanding the limitations and proper applications of these tests, researchers can avoid common pitfalls and ensure that their conclusions are both valid and meaningful.

Appendix: Full R Code for the Simulations

Below is the full R code used in the simulations and analyses presented in this article. This code can be used to replicate the results and explore the nuances of nonparametric testing in your own datasets.

# Set seed for reproducibility
set.seed(1234)

# Generate data for two groups with identical medians but different variances
v5 <- rlnorm(500, 0, 1)
variance_factor <- 6
median_v5 <- median(v5)
scaled_dev <- (v5 - median_v5) * sqrt(variance_factor)
v6 <- (median_v5 + scaled_dev)

# Mann-Whitney U test
wilcox.test(v5, v6, conf.int = TRUE)

# Quantile regression
stacked <- stack(data.frame(v5, v6))
summary(rq(values ~ ind, data = stacked), se = "boot", R = 9999)

# Brunner-Munzel test
brunnermunzel::brunnermunzel.test(v5, v6)

# Simulation study: Assessing the performance of Mann-Whitney test
N <- 200
failed_comparisons <- 0

sim_p <- sapply(seq(N), function(i) {
   x1 <- rlnorm(500, 0, 1)
   variance_factor <- 6
   median_x1 <- median(x1)
   scaled_dev <- (x1 - median_x1) * sqrt(variance_factor)
   x2 <- (median_x1 + scaled_dev)
 
   diff_med <- median(x1) - median(x2)
   
   if (diff_med <= .Machine$double.eps) {
     wilcox.test(x1, x2)$p.value
   } else {
     print(paste("Ouch! Medians differ by:", diff_med, "which is more than my machine epsilon"))
     failed_comparisons <<- failed_comparisons + 1
   }
})

# Summary of simulation results
sprintf("Fraction of rejections for same medians: %d / %d (%.2f%%)", 
         sum(sim_p <= 0.05),
         N - failed_comparisons,
         100 * (sum(sim_p <= 0.05) / (N - failed_comparisons)))

This code serves as a foundation for understanding the nuances of nonparametric testing. By experimenting with this code and modifying it to suit your specific needs, you can gain deeper insights into the behavior of these statistical tests under different conditions.

Appendix: Full Ruby Code for the Simulations

Below is the full Ruby code used to replicate the analysis presented in this article. This code mirrors the R simulations and analyses, providing insights into nonparametric testing using Ruby.

Required Libraries

To run the following Ruby code, you’ll need the distribution, statsample, and descriptive_statistics gems. Install them using:

gem install distribution statsample descriptive_statistics

Code Implementation

require 'distribution'
require 'statsample'
require 'descriptive_statistics'

# Function to generate log-normal data
def generate_lognorm_data(size, mean, sd)
  Array.new(size) { Math.exp(Distribution::Normal.rng(mean, sd).call) }
end

# Function to scale data to increase dispersion while keeping medians constant
def scale_data(data, factor)
  median = data.median
  scaled_dev = data.map { |x| (x - median) * Math.sqrt(factor) }
  scaled_dev.map { |x| median + x }
end

# Mann-Whitney U test
def mann_whitney_u_test(data1, data2)
  Statsample::Test::UMannWhitney.new(data1.to_scale, data2.to_scale).probability
end

# Brunner-Munzel test (Note: This is a custom implementation)
def brunner_munzel_test(data1, data2)
  # Calculating ranks
  combined = data1 + data2
  ranks = combined.sort.map.with_index { |_, i| i + 1 }
  ranks1 = data1.map { |x| ranks[combined.index(x)] }
  ranks2 = data2.map { |x| ranks[combined.index(x)] }

  # Calculating test statistic
  n1 = data1.size
  n2 = data2.size
  mean_rank1 = ranks1.sum.to_f / n1
  mean_rank2 = ranks2.sum.to_f / n2

  var1 = ranks1.map { |r| (r - mean_rank1)**2 }.sum / (n1 - 1)
  var2 = ranks2.map { |r| (r - mean_rank2)**2 }.sum / (n2 - 1)

  bm_stat = (mean_rank1 - mean_rank2) / Math.sqrt(var1 / n1 + var2 / n2)
  p_value = 2 * (1 - Distribution::Normal.cdf(bm_stat.abs))

  p_value
end

# Simulation study
N = 200
failed_comparisons = 0
p_values = []

N.times do
  data1 = generate_lognorm_data(500, 0, 1)
  data2 = scale_data(data1, 6)

  diff_medians = data1.median - data2.median

  if diff_medians.abs <= Float::EPSILON
    p_values << mann_whitney_u_test(data1, data2)
  else
    puts "Ouch! Medians differ by: #{diff_medians}, which is more than the machine epsilon"
    failed_comparisons += 1
  end
end

fraction_rejections = p_values.count { |p| p <= 0.05 }
total_tests = N - failed_comparisons
percentage_rejections = 100.0 * fraction_rejections / total_tests

puts "Fraction of rejections for same medians: #{fraction_rejections} / #{total_tests} (#{percentage_rejections.round(2)}%)"

Notes on Implementation

  • Mann-Whitney U Test: This is implemented using the Statsample gem, which provides nonparametric tests in Ruby.
  • Brunner-Munzel Test: A custom implementation is provided due to the lack of a direct equivalent in common Ruby libraries.
  • Simulation Results: The code outputs the fraction of rejections when testing datasets with identical medians but different variances.

This Ruby implementation allows you to replicate and explore the findings discussed in the article using a different programming language, providing flexibility and adaptability to your analysis workflow.

Appendix: Full Python Code for the Simulations

Below is the full Python code used to replicate the analysis presented in this article. This code mirrors the R simulations and analyses, providing insights into nonparametric testing using Python.

Required Libraries

To run the following Python code, you’ll need the numpy, scipy, and statsmodels libraries. Install them using:

pip install numpy scipy statsmodels

Code Implementation

import numpy as np
from scipy.stats import mannwhitneyu, rankdata, norm
from statsmodels.regression.quantile_regression import QuantReg
import statsmodels.api as sm

# Function to generate log-normal data
def generate_lognorm_data(size, mean, sd):
    return np.random.lognormal(mean, sd, size)

# Function to scale data to increase dispersion while keeping medians constant
def scale_data(data, factor):
    median = np.median(data)
    scaled_dev = (data - median) * np.sqrt(factor)
    return median + scaled_dev

# Mann-Whitney U test
def mann_whitney_u_test(data1, data2):
    return mannwhitneyu(data1, data2, alternative='two-sided').pvalue

# Brunner-Munzel test (custom implementation)
def brunner_munzel_test(data1, data2):
    combined = np.concatenate([data1, data2])
    ranks = rankdata(combined)
    ranks1 = ranks[:len(data1)]
    ranks2 = ranks[len(data1):]

    n1 = len(data1)
    n2 = len(data2)
    mean_rank1 = np.mean(ranks1)
    mean_rank2 = np.mean(ranks2)

    var1 = np.var(ranks1, ddof=1)
    var2 = np.var(ranks2, ddof=1)

    bm_stat = (mean_rank1 - mean_rank2) / np.sqrt(var1 / n1 + var2 / n2)
    p_value = 2 * (1 - norm.cdf(abs(bm_stat)))

    return p_value

# Quantile regression
def quantile_regression(data1, data2):
    stacked_data = np.concatenate([data1, data2])
    indicators = np.concatenate([np.zeros(len(data1)), np.ones(len(data2))])
    model = QuantReg(stacked_data, sm.add_constant(indicators))
    result = model.fit(q=0.5)
    return result.pvalues[1]

# Simulation study
N = 200
failed_comparisons = 0
p_values = []

for _ in range(N):
    data1 = generate_lognorm_data(500, 0, 1)
    data2 = scale_data(data1, 6)

    diff_medians = np.median(data1) - np.median(data2)

    if abs(diff_medians) <= np.finfo(float).eps:
        p_values.append(mann_whitney_u_test(data1, data2))
    else:
        print(f"Ouch! Medians differ by: {diff_medians}, which is more than the machine epsilon")
        failed_comparisons += 1

fraction_rejections = sum(p <= 0.05 for p in p_values)
total_tests = N - failed_comparisons
percentage_rejections = 100.0 * fraction_rejections / total_tests

print(f"Fraction of rejections for same medians: {fraction_rejections} / {total_tests} ({percentage_rejections:.2f}%)")

Notes on Implementation

  • Mann-Whitney U Test: This test is implemented using the scipy.stats.mannwhitneyu function, which performs the Mann-Whitney U test.
  • Brunner-Munzel Test: A custom implementation is provided due to the lack of a direct equivalent in common Python libraries.
  • Quantile Regression: The quantile regression is performed using the statsmodels library’s QuantReg function.

This Python implementation allows you to replicate and explore the findings discussed in the article using Python, providing an alternative to the R and Ruby versions.

You may also enjoy