Error Terms in Linear and Logistic Regression

Linear and logistic regression both model a conditional distribution. The difference is not that one model has “error” and the other does not. The difference is how random variation is represented. For a Gaussian linear model,

For binary logistic regression,

with

The first writes a disturbance term explicitly. The second places randomness directly in the Bernoulli conditional distribution.

Model error is not the same thing as residual

In linear regression, the latent error is

It is part of the data-generating model and is not observed directly. After fitting, the residual is

Residuals depend on the estimated model and therefore are not identical to the latent errors. In matrix form,

where

is the hat matrix. This means residuals have a covariance structure induced by the fitted model even when the original errors are independent.

The linear conditional mean

The central regression model is

Ordinary least squares estimates

If

then under the classical fixed-design argument,

Normality is not required for this unbiasedness result.

What Gaussian errors add

If we assume

then OLS is also the maximum-likelihood estimator for $\beta$ and exact finite-sample t and F distributions become available. That is a stronger assumption than OLS estimation itself requires. The usual hierarchy is:

• conditional mean zero for unbiasedness;
• standard regularity conditions for consistency;
• homoskedastic uncorrelated errors for Gauss-Markov efficiency among linear unbiased estimators;
• Gaussian errors for exact normal-theory likelihood and finite-sample inference.

These conclusions should not be merged into one statement such as “normal errors make OLS unbiased and efficient.”

Heteroskedasticity

If

OLS coefficients can remain unbiased or consistent under the appropriate exogeneity assumptions. What fails is the ordinary homoskedastic covariance formula. A heteroskedasticity-consistent covariance estimator can then be used for inference. So variance misspecification and conditional-mean misspecification are different problems.

Logistic regression has a stochastic model

For binary outcomes,

where

The conditional mean is

and the conditional variance is

The random variation is therefore explicit in the Bernoulli distribution. There is no need to add an independent Gaussian error to the logit equation.

The likelihood

For independent observations, the Bernoulli likelihood is

The log-likelihood is

Maximum likelihood chooses

The likelihood is not “the error term.” It is the probability model used to estimate the coefficients.

Logistic residuals exist

Logistic regression has several useful residual definitions. The response residual is

The Pearson residual is

Deviance residuals measure the signed contribution of each observation to model deviance. These residuals can reveal lack of fit, unusual observations, or systematic structure not captured by the model. So “logistic regression has no residuals” is false.

Latent-variable representation

Logistic regression can also be represented through a latent variable:

with

where $\varepsilon_i$ follows a logistic distribution. This representation helps explain why the logit link appears. But the latent scale is not directly observed, so the coefficient scale depends on the fixed logistic error distribution.

Coefficients live on different scales

In linear regression,

is a conditional change in the mean response per unit change in predictor $j$, holding the other modeled predictors fixed. In logistic regression,

is a conditional change in log-odds. Exponentiating gives an odds ratio:

Neither coefficient is automatically causal. That interpretation requires a causal design or identification assumptions.

Classification metrics do not replace model diagnostics

A logistic model can have high classification accuracy and still be poorly calibrated. AUC can be high while predicted probabilities are systematically too extreme. Useful diagnostics include:

• calibration plots;
• Brier score;
• log loss;
• residuals;
• leverage and influence;
• separation checks;
• out-of-sample validation.

The statistical model and the classification decision are related but distinct layers.

Dependence

Both linear and logistic regression can be misspecified when observations are dependent. Examples include:

• repeated measurements;
• patients within hospitals;
• time series;
• spatial data;
• family clusters.

For binary repeated measures, options include GEE, mixed-effects logistic regression, or cluster-robust inference depending on the estimand. The Bernoulli mean model alone does not define dependence among observations.

Overdispersion and binary data

For one Bernoulli observation,

is fixed by the mean. But grouped binomial data can exhibit extra-binomial variation due to unmodeled heterogeneity or dependence. That can be handled through beta-binomial models, random effects, quasi-likelihood, or robust covariance methods depending on the source of variation.

Conclusion

Linear and logistic regression do not differ because one “has error” and the other does not. They differ because they specify different conditional distributions:

Residuals are fitted diagnostics in both settings. The inferential assumptions belong to the full probability model, not to a vague idea of “error handling.”

References

• McCullagh, P., & Nelder, J. A. (1989). Generalized Linear Models (2nd ed.). Chapman & Hall.
• White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48(4), 817–838.
• Pregibon, D. (1981). Logistic regression diagnostics. Annals of Statistics, 9(4), 705–724.