Introduction to Ordinal Regression

When working with ordinal data—data that is categorical and ordered but without a precise numeric distance between categories—it’s important to use the right statistical models for accurate analysis. Ordinal regression is one such model designed to handle these cases. While simple linear regression assumes continuous outcomes, and logistic regression deals with binary outcomes, ordinal regression fills the gap by modeling outcomes that have a natural order but are not continuous.

In this article, we’ll dive into the principles of ordinal regression, using a real-world dataset to guide the discussion. We’ll explore how the cumulative approach to ordinal regression works and show you how to interpret the results using marginal effects—a method that makes regression outputs more intuitive and accessible.

What is Ordinal Data?

Ordinal data refers to categories that have a meaningful order or ranking but no clear numerical intervals between them. For example, in a survey asking respondents to rate their interest in politics as:

  1. None
  2. Not much
  3. Some
  4. A good deal

While these categories are clearly ranked, we cannot say how much more interested someone who chooses “Some” is compared to someone who chooses “Not much”. This distinction makes ordinal data unique, and ordinary linear regression is not appropriate for analyzing it.

Why Use Ordinal Regression?

The goal of ordinal regression is to model the probability of membership in each level of the ordinal outcome variable. It allows us to understand how predictor variables (such as age, gender, or education) affect the likelihood of an individual belonging to each category of the outcome variable (like levels of political interest). This type of model is frequently used in social sciences, medical research, and marketing, where survey data or scales are common.

The Cumulative Logit Model: Architecture and Concepts

Ordinal regression models come in various forms, but we will focus on the cumulative logit model, the most common approach for ordinal outcomes. This model is a type of proportional odds model, and it extends the logic of logistic regression to ordered categories. Essentially, instead of modeling a binary outcome, we model cumulative probabilities up to a certain threshold.

Mathematical Representation

Mathematically, we can express the cumulative logit model as follows:

\[\log\left(\frac{P(Y \leq j)}{P(Y > j)}\right) = \alpha_j - \beta X\]

Where:

  • \(P(Y \leq j)\) represents the probability that the outcome is in category \(j\) or lower.
  • \(\alpha_j\) is the intercept (threshold) for category \(j\).
  • \(\beta\) represents the coefficients for the predictors \(X\), which explain how the covariates influence the probability of belonging to different categories.

In a cumulative logit model, we estimate a separate intercept for each threshold between the ordered categories. The logit link function converts the output into probabilities, making the model flexible enough to handle ordinal data. This approach assumes that the effect of each predictor is the same across all thresholds—a property known as the proportional odds assumption.

Interpreting Ordinal Regression Models

One of the challenges of ordinal regression is interpreting the results. Coefficients in an ordinal regression model reflect the change in the log-odds of being in or below a certain category of the outcome variable. However, log-odds are not always intuitive, so interpreting the raw coefficients can be tricky.

This is where marginal effects come in. Marginal effects transform the coefficients into probabilities, making it easier to understand how changes in predictor variables affect the likelihood of belonging to different outcome categories.

Marginal Effects: Simplifying Interpretation

In statistical models like ordinal regression, marginal effects allow us to see how changes in a predictor variable affect the predicted probabilities of different outcomes. Instead of interpreting raw coefficients (which often lack practical meaning), marginal effects give us a clearer view of the model’s impact.

What Are Marginal Effects?

Marginal effects are partial derivatives that show how the predicted probability of an outcome changes with respect to a predictor variable, holding all other variables constant. In simple terms, marginal effects reveal how a one-unit change in a predictor variable (such as age) affects the probability of an outcome (such as the level of interest in politics).

For categorical variables, marginal effects are referred to as conditional effects or group contrasts, since categorical variables don’t have slopes in the same way continuous variables do. For these, we compare differences in predicted probabilities between categories.

Why Use Marginal Effects?

In an ordinal regression context, marginal effects translate the abstract log-odds into tangible probabilities, making it easier to answer real-world questions. Instead of asking, “What is the log-odds of someone being highly interested in politics?”, you can ask, “How does age affect the probability of being highly interested in politics?”

These effects are typically calculated on the outcome scale of interest, which in this case is probability, making them much more interpretable for decision-makers and stakeholders who are not statisticians.

A Practical Example: Exploring Political Interest

To illustrate how ordinal regression and marginal effects work together, we’ll use a real dataset: the 2022 Australian Election Study (AES). This dataset includes responses to survey questions about political interest and various demographic factors. We’ll explore how these factors, such as age, gender, and education, influence respondents’ interest in politics.

The Dataset

The survey asks respondents how much interest they usually have in what’s going on in politics, with answers on a four-point ordinal scale:

  1. None
  2. Not much
  3. Some
  4. A good deal

We’ll use this ordinal outcome variable, along with predictors like age, gender, employment status, and years of tertiary education.

Data Preprocessing

Before we can model the data, we need to clean it. This involves filtering out irrelevant or erroneous values and recoding categorical variables into more meaningful labels. The data preprocessing step ensures that our model is working with high-quality inputs, which is essential for producing reliable results.

We also convert categorical variables (such as gender and employment status) into appropriate formats for use in Python’s statsmodels package.

Modeling Political Interest Using Ordinal Regression

To model political interest, we will use the proportional odds logistic regression model, implemented in Python using the OrderedModel from statsmodels. This model fits our ordinal outcome by estimating cumulative probabilities across the different levels of political interest.

from statsmodels.miscmodels.ordinal_model import OrderedModel

# Defining the ordinal regression model
model = OrderedModel(aes_data['A1'], 
                     aes_data[['years_tertiary_study', 'age', 'work', 'managerial', 'gender', 'sexual_orientation']], 
                     distr='logit')

# Fitting the model
mod_fit = model.fit(method='bfgs')

# Summary of the model
print(mod_fit.summary())

After fitting the model, we can examine the coefficients, intercepts, and other summary statistics. However, interpreting these coefficients can be difficult because they are in the form of log-odds. Therefore, we turn to marginal effects to make sense of the results.

Using Marginal Effects to Interpret the Model

In Python, we can calculate marginal effects using methods similar to R’s marginaleffects package. We will compute predictions and slopes (marginal effects) for our key variables, such as age, gender, and employment status, and see how these factors influence political interest.

Predictions

Let’s start by calculating the predicted probabilities for different values of age. This will help us see how age influences the likelihood of a respondent’s political interest.

import numpy as np

# Define new data for prediction
new_data = np.linspace(aes_data['age'].min(), aes_data['age'].max(), 100)

# Predicted probabilities
pred_probs = mod_fit.predict(new_data)

# Visualize the predicted probabilities
import matplotlib.pyplot as plt
plt.plot(new_data, pred_probs)
plt.xlabel('Age')
plt.ylabel('Predicted Probability')
plt.title('Predicted Political Interest by Age')
plt.show()

This plot shows how the predicted probability of each level of political interest changes with age. For example, as age increases, the probability of reporting “A good deal” of political interest tends to rise, while the probability of reporting “Not much” decreases.

Slopes

Next, we calculate the marginal effects (slopes) for the variable age, which gives us a more precise understanding of how the probability of political interest changes as age increases.

# Calculate marginal effects for age
marginal_effects = mod_fit.get_margeff(at='mean', method='dydx')

# Summary of marginal effects
print(marginal_effects.summary())

The slopes reveal how much the probability of each level of political interest changes as age increases by one unit. We can also compute slopes for other variables, such as years of tertiary education or employment status, to explore their effects on political interest.

Ordinal Regression in Practice

Ordinal regression is a powerful tool for modeling outcomes that are ordered but not continuous. While the model’s coefficients can be challenging to interpret, the use of marginal effects transforms these outputs into probabilities that are easy to understand. This makes ordinal regression a valuable tool for fields like political science, healthcare, and marketing, where ordinal data is common.

By applying this model to the Australian Election Study dataset, we can see how factors such as age, gender, and education influence political interest. With the help of marginal effects, we can move beyond raw coefficients and interpret the results in a meaningful way, allowing for more informed decision-making.

Appendix: Python Code for Ordinal Regression

For a complete Python implementation of ordinal regression using the statsmodels package, refer to the code below:

# Import necessary libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from statsmodels.miscmodels.ordinal_model import OrderedModel

# Data Preprocessing
aes_data = pd.read_csv('https://raw.githubusercontent.com/hendersontrent/hendersontrent.github.io/master/files/aes22_unrestricted_v2.csv')

# Clean the data, filter erroneous values, and recode categorical variables
aes_data = aes_data.rename(columns={'G2': 'years_tertiary_study', 'G4': 'work', 'G5_D': 'managerial', 'H1': 'gender', 'H1b': 'sexual_orientation', 'H2': 'dob'})
aes_data['age'] = 2022 - aes_data['dob']
aes_data = aes_data[aes_data['A1'].isin([1, 2, 3, 4])]
aes_data = aes_data[(aes_data['years_tertiary_study'] != 999) & (aes_data['work'].between(1, 7)) & (aes_data['managerial'].between(1, 5))]
aes_data['A1'] = aes_data['A1'].map({4: 'None', 3: 'Not much', 2: 'Some', 1: 'A good deal'})

# Define the ordinal regression model
model = OrderedModel(aes_data['A1'], aes_data[['years_tertiary_study', 'age', 'work', 'managerial', 'gender', 'sexual_orientation']], distr='logit')

# Fit the model
mod_fit = model.fit(method='bfgs')

# Print the summary of the model
print(mod_fit.summary())

# Predict probabilities
new_data = np.linspace(aes_data['age'].min(), aes_data['age'].max(), 100)
pred_probs = mod_fit.predict(new_data)

# Plot predicted probabilities
plt.plot(new_data, pred_probs)
plt.xlabel('Age')
plt.ylabel('Predicted Probability')
plt.title('Predicted Political Interest by Age')
plt.show()

# Calculate marginal effects
marginal_effects = mod_fit.get_margeff(at='mean', method='dydx')
print(marginal_effects.summary())

You may also enjoy