Connection Between OLS and Theil-Sen Estimators

In regression analysis, the Ordinary Least Squares (OLS) estimator and the Theil-Sen estimator are pivotal techniques. Traditionally, these methods have been viewed as distinct, each with unique properties and applications. However, a fascinating link between them emerges when considering the OLS estimator as a weighted average of slopes between all possible pairs of points. This article examines the mathematical underpinnings of this connection and explores its implications.

Understanding the OLS Estimator

The OLS estimator aims to find the best-fit line for a given set of data points by minimizing the sum of the squared differences between observed and predicted values. For a simple linear regression model, $y = \alpha + \beta x + \epsilon$, the slope $\beta$ is given by:

This expression highlights that $\beta$ is a ratio of the covariance of $x$ and $y$ to the variance of $x$. However, it can also be represented in a different form:

where the weights $W_{i,j}$ are defined as:

This formulation reveals that $\beta$ is a weighted average of the slopes between all pairs of points $(i, j)$. The weights emphasize pairs separated farther in predictor space. Pairs with $x_i=x_j$ have undefined pairwise slopes and contribute zero to the OLS pairwise identity because their squared-distance weight is zero.

Derivation of the Weighted Average

To understand this derivation, consider the classic formula for $\beta$ in OLS:

Expanding the terms in the numerator:

Given that $\sum_{i} (x_i - \bar{x}) = 0$, the term simplifies to:

The pairwise identity includes a factor of $n$:

and similarly

The factor cancels in the ratio, which yields the pairwise weighted-average representation.

Theil-Sen Estimator: A Robust Alternative

The Theil-Sen estimator provides a robust estimate of the slope $\beta$ by taking the median of the slopes between all pairs of points. It is defined as:

This approach is less sensitive to outliers compared to OLS, making it a preferred choice in certain scenarios where robustness is crucial.

Robustness of Theil-Sen

The robustness of the Theil-Sen estimator stems from its reliance on medians rather than means. Medians are inherently more resistant to outliers, which can disproportionately influence the mean. In practical applications, this robustness translates to more reliable estimates in the presence of anomalous data points.

Connecting OLS and Theil-Sen Estimators

The revelation that OLS can be viewed as a weighted average of the slopes between all pairs of points brings to light a profound connection with the Theil-Sen estimator. Specifically:

and

show that both estimators fundamentally rely on the same set of pairwise slopes. The key difference lies in the weighting scheme: OLS uses a specific set of weights $W_{i,j}$, while Theil-Sen employs the median.

Mathematical Insights

1. Weighted Average vs. Median: The OLS estimator uses a weighted average, assigning different importance to each pair of points based on the squared difference in $x$ values. In contrast, Theil-Sen uses the median, treating all pairs equally and focusing on the central tendency.
2. Sensitivity to Outliers: Due to its weighting scheme, OLS can be more sensitive to outliers, especially if pairs with large $x$ differences are outliers. Theil-Sen, using the median, remains robust in such cases.
3. Efficiency vs. Robustness: Under the Gauss-Markov assumptions, including homoskedastic uncorrelated errors and conditional mean zero, OLS is BLUE among linear unbiased estimators. Normality is not required for that result; Gaussian errors add exact finite-sample distributional results and likelihood interpretations. Theil-Sen trades some efficiency under the classical Gaussian model for substantially greater resistance to contamination. Non-normality by itself is not sufficient reason to prefer it; the relevant question is robustness to the kinds of departures that matter.

Implications of the Connection

1. Unified Perspective: Viewing OLS as a weighted average aligns it more closely with robust methods like Theil-Sen, suggesting a spectrum of estimators that balance efficiency and robustness. This perspective encourages a more holistic understanding of regression techniques.
2. Enhanced Understanding: This connection provides deeper insights into the nature of OLS, highlighting its reliance on pairwise comparisons, a perspective traditionally associated with non-parametric methods. This enriched understanding can inform the choice of estimators in practical applications.
3. Potential for Hybrid Methods: Recognizing the link between these estimators could inspire new hybrid techniques that leverage the strengths of both OLS and Theil-Sen. For instance, one could develop estimators that dynamically adjust weights based on data distribution, blending efficiency and robustness.

Practical Applications

Regression Analysis in Econometrics

In econometrics, where data can often be messy and contain outliers, understanding the trade-offs between OLS and Theil-Sen estimators is crucial. Econometricians can use the insights from the connection between these methods to choose the most appropriate estimator for their specific context.

Machine Learning and Data Science

In machine learning, robust regression methods are essential for building models that generalize well to new data. The connection between OLS and Theil-Sen can guide the development of algorithms that balance precision and robustness, improving model performance in real-world applications.

Statistical Software Implementation

Statistical software packages can incorporate hybrid estimators that draw on the strengths of both OLS and Theil-Sen. By providing users with more options, software developers can enhance the analytical capabilities available to statisticians and data scientists.

Conclusion

The discovery that the OLS estimator can be interpreted as a weighted average of pairwise slopes bridges the gap between it and the Theil-Sen estimator. This connection enriches our understanding of regression analysis and opens avenues for developing robust yet efficient estimation methods. By appreciating the nuances of these weighting schemes, statisticians and data scientists can make more informed choices in their analytical endeavors.

The link between OLS and Theil-Sen estimators underscores the importance of understanding the underlying mechanics of statistical methods. As we continue to explore these connections, we can develop more sophisticated and versatile tools for data analysis, ultimately enhancing our ability to extract meaningful insights from complex datasets.

Breakdown and leverage are different robustness problems

Theil-Sen is resistant to vertical contamination because the median of pairwise slopes cannot be moved arbitrarily by a small number of bad observations. But high-leverage contamination in $x$ can still be influential. Robust regression is not one property; leverage, vertical outliers, heteroskedasticity, and heavy tails require different diagnostics.

For multivariate regression, the simple pairwise-slope representation no longer extends directly in the same way, and robust estimators such as least-trimmed squares, MM-estimation, or robust covariance approaches address different goals.

References

• Theil, H. (1950). A rank-invariant method of linear and polynomial regression analysis. Proceedings of the Koninklijke Nederlandse Akademie van Wetenschappen, 53.
• Sen, P. K. (1968). Estimates of the regression coefficient based on Kendall's tau. Journal of the American Statistical Association, 63(324), 1379–1389.
• Wilcox, R. R. (2017). Introduction to Robust Estimation and Hypothesis Testing (4th ed.). Academic Press.