A Comprehensive Guide to Survival Function Estimation Methods

22 minute read

Introduction

Survival analysis is a crucial statistical tool used to estimate the time until an event of interest occurs. This can be applied in various fields such as medical research, engineering, and customer analytics. Understanding how long it takes for a specific event to happen, like the failure of a machine part or the dropout of a customer, can inform decision-making and strategy.

Survival functions, denoted as S(t), represent the probability that the event of interest has not occurred by a specific time t. Estimating these functions accurately is essential for predicting future events and understanding the underlying patterns in data. There are multiple methods for estimating survival functions, which can be broadly categorized into parametric and non-parametric approaches.

Parametric methods assume a specific distribution for the survival times, leading to smoother and often more interpretable functions. These methods are powerful when the assumed model closely matches the actual data. On the other hand, non-parametric methods do not assume any specific distribution, providing more flexibility and robustness, especially when the true distribution of survival times is unknown or complex.

This article explores both parametric and non-parametric methods for estimating survival functions, discussing their applications, advantages, and limitations. Additionally, we touch on how machine learning techniques can offer alternative approaches to survival analysis, expanding the toolkit available to researchers and practitioners.

Parametric Survival Functions

Assumptions and Basics

Parametric approaches to survival analysis rely on the assumption that the survival function follows a specific mathematical form or distribution. This assumption simplifies the estimation process by reducing the problem to estimating the parameters of the chosen distribution.

The fundamental idea is that at the start of the study, all subjects are alive or have not yet experienced the event of interest. This initial condition is represented by \(S(0) = 1\), meaning the survival probability at time zero is 100%. As time progresses, the probability that a subject survives (or does not experience the event) decreases. This results in a survival function \(S(t)\) that is a monotonically decreasing function over time, eventually approaching zero as the event becomes inevitable.

Common Assumptions in Parametric Models

  1. Exponential Distribution:
    • Assumes a constant hazard rate over time, implying that the event is equally likely to occur at any time point. This model is simple but may not be realistic for all types of data.
  2. Weibull Distribution:
    • Generalizes the exponential distribution by allowing the hazard rate to vary over time. It can model increasing, constant, or decreasing hazard rates, making it more flexible.
  3. Log-Normal Distribution:
    • Assumes that the logarithm of the survival time follows a normal distribution. This model is useful for data that are skewed and have a long tail.
  4. Gamma Distribution:
    • Provides a flexible model for survival times with a wide range of shapes depending on its parameters.

Why Use Parametric Models?

Parametric models are powerful because they offer several significant advantages in survival analysis:

  1. Smooth and Continuous Estimates:
    • Parametric models provide smooth and continuous estimates of the survival function. This smoothness can make the survival function easier to interpret and analyze compared to the stepwise functions produced by non-parametric methods.
  2. Predictive Power:
    • By fitting a specific distribution to the data, parametric models can be used to make predictions about future survival times. This is particularly useful in contexts where forecasting is crucial, such as medical prognosis or reliability engineering.
  3. Extrapolation Beyond Observed Data:
    • Parametric models allow for extrapolation beyond the range of the observed data. This means they can predict survival probabilities at times not represented in the sample, which can be invaluable for long-term planning and decision-making.
  4. Parameter Estimation:
    • The parameters estimated in parametric models often have meaningful interpretations. For instance, in an exponential model, the parameter ( \lambda ) represents the constant hazard rate, providing insights into the underlying process.
  5. Efficiency:
    • Parametric methods can be more statistically efficient than non-parametric methods when the model assumptions hold true. This means they can produce more precise estimates with the same amount of data.
  6. Ease of Use:
    • Once the appropriate parametric form is chosen, the process of estimating the parameters and deriving the survival function is straightforward, often involving well-established statistical techniques and software implementations.

Practical Applications

  • Medical Research:
    • In clinical trials, parametric models can predict patient survival times under different treatment regimes, aiding in the evaluation of treatment efficacy.
  • Engineering and Reliability:
    • In reliability engineering, parametric models help estimate the lifespan of components or systems, guiding maintenance schedules and improving safety measures.
  • Business and Marketing:
    • Businesses can use parametric models to predict customer churn rates, helping to tailor retention strategies and improve customer lifetime value.

Example of Extrapolation

To illustrate the power of parametric models for extrapolation, consider a dataset of patient survival times following a new treatment. Suppose we fit a Weibull distribution to this data. The Weibull model is particularly flexible, allowing for varying hazard rates over time, making it suitable for many types of survival data.

Estimating Within the Observed Range

Within the observed time range, the Weibull model provides smooth estimates of the survival probabilities at any given time \(t\). For example, if we observed survival times up to 5 years, the model can estimate the probability of a patient surviving up to each point within those 5 years.

\[S(t) = e^{-(\lambda t)^\gamma}\]

Here, \(\lambda\) and \(\gamma\) are parameters estimated from the data. The parameter \(\lambda\) is a scale parameter, and \(\gamma\) is a shape parameter that adjusts the hazard function’s shape over time.

Predicting Beyond the Observed Range

One of the key strengths of the parametric approach is its ability to predict survival probabilities beyond the observed data range. For instance, if our dataset includes survival data up to 5 years, the Weibull model can be used to predict survival rates at 6, 7, or even 10 years.

This extrapolation capability is particularly useful in long-term studies where early data collection may not cover the entire period of interest. By fitting a Weibull model, researchers and practitioners can make informed predictions about future survival trends, even in the absence of long-term data.

Practical Application

Imagine a clinical trial for a new cancer treatment where the longest follow-up data available is 5 years. Using the Weibull model, oncologists can estimate the probability of survival at 10 years, providing valuable insights for long-term treatment planning and patient counseling.

For example, if the estimated survival probability at 5 years is 60%, the model might predict a 40% survival probability at 10 years, offering a quantitative basis for discussions about prognosis and future research directions.

The ability to extrapolate survival probabilities beyond the observed data range is a significant advantage of parametric models. This feature enables researchers to make long-term predictions and strategic decisions based on early data, illustrating the practical utility of parametric survival analysis.

While parametric models offer these advantages, it is essential to ensure that the chosen model accurately reflects the data’s underlying patterns. Mis-specification of the model can lead to incorrect conclusions, highlighting the importance of model validation and selection.

Limitations

However, the primary limitation of parametric survival analysis is the risk of mis-specification. If the chosen parametric form does not accurately represent the true survival function, the resulting estimates can be biased and misleading. For instance, assuming an exponential distribution when the data actually follow a Weibull distribution can lead to incorrect conclusions about survival probabilities and hazard rates.

Model Fit and Validation

To mitigate this risk, it is crucial to validate the model assumptions with the data. This involves:

  1. Goodness-of-Fit Tests:
    • Statistical tests such as the Kolmogorov-Smirnov test, Chi-square test, or likelihood ratio tests can help assess how well the chosen model fits the data.
  2. Residual Analysis:
    • Analyzing the residuals (the differences between observed and predicted values) can indicate whether the model appropriately captures the underlying patterns in the data.
  3. Graphical Methods:
    • Visual tools such as Q-Q plots, P-P plots, and survival plots can provide intuitive assessments of model fit. If the plotted points deviate significantly from the expected line, this may suggest a poor fit.
  4. Comparing Multiple Models:
    • Comparing the chosen model with alternative parametric and non-parametric models can highlight potential issues and guide the selection of a more appropriate model.

Practical Implications

In practical applications, failing to validate the model can lead to significant consequences. For example, in medical research, incorrect survival estimates can impact treatment decisions and patient outcomes. In business, misestimating customer churn rates can result in suboptimal marketing strategies and revenue forecasts.

Balancing Flexibility and Interpretability

While parametric models offer a structured and interpretable framework, this comes with the trade-off of potentially restrictive assumptions. Researchers must balance the desire for smooth, continuous estimates with the need for model flexibility to accurately capture the true survival function.

Parametric survival analysis involves selecting a specific distribution for the survival function and estimating its parameters. This approach provides a structured and interpretable framework but requires careful consideration of the underlying assumptions. Validation of the chosen model is essential to ensure accurate and reliable estimates. By combining statistical tests, graphical methods, and comparative analyses, researchers can improve the robustness and validity of their parametric survival models.

Exponential Survival Function

One commonly used parametric model in survival analysis is the exponential survival function. This model is characterized by its simplicity and the assumption of a constant hazard rate over time.

\[S(t) = e^{-\lambda t}\]

Understanding the Model

  • Constant Hazard Rate: The exponential model assumes that the event of interest (e.g., failure, death) occurs at a constant rate, regardless of how long an individual has already survived. This is a key assumption that simplifies the analysis but may not always be realistic.

  • Parameter \(\lambda\): The parameter \(\lambda\) represents the constant hazard rate. It is estimated from the data and determines how quickly the survival probability decreases over time. A higher \(\lambda\) indicates a higher risk of the event occurring in a given time period.

Example Calculation

Consider a dataset where the estimated hazard rate \(\lambda\) is 0.1 per year. The exponential survival function can be used to calculate the probability that a subject survives beyond a specific time \(t\):

For \(t = 5\) years,

\[S(5) = e^{-0.1 \times 5} = e^{-0.5} \approx 0.607\]

This means that there is approximately a \(60.7\%\) chance that a subject will survive beyond 5 years.

Applications

The exponential model is widely used in various fields due to its simplicity:

  • Medical Research: Estimating the survival time of patients after treatment.
  • Engineering: Assessing the reliability and failure rates of components or systems.
  • Business: Modeling customer retention rates over time.

Advantages

  • Simplicity: The exponential model is mathematically straightforward and easy to implement.
  • Interpretability: The constant hazard rate provides a clear and intuitive measure of risk over time.
  • Analytical Convenience: The model’s simplicity allows for closed-form solutions and straightforward maximum likelihood estimation.

Limitations

However, the simplicity of the exponential model comes with significant limitations:

  • Constant Hazard Assumption: The assumption of a constant hazard rate is often unrealistic. Many real-world processes exhibit increasing or decreasing hazard rates over time.
  • Lack of Flexibility: The model cannot accommodate varying hazard rates, making it less suitable for complex survival data.
  • Model Fit: If the data do not conform to the exponential distribution, the model’s estimates may be inaccurate.
  • Extrapolation: The exponential model cannot predict survival probabilities beyond the observed data range, limiting its long-term forecasting capabilities.
  • Validation: It is essential to validate the exponential model’s assumptions and fit to the data to ensure its reliability.
  • Alternative Models: Researchers should consider alternative parametric models that offer more flexibility and better fit to the data when the exponential model is not appropriate.

While the exponential survival function offers a useful starting point for survival analysis, its assumptions may not hold in all situations. Researchers should carefully consider the appropriateness of the exponential model for their specific data and explore alternative models if necessary. Despite its limitations, the exponential model’s simplicity and interpretability make it a valuable tool in many applications.

Other Parametric Models

While the exponential survival function is straightforward, other parametric models offer greater flexibility and can better fit various types of survival data. These models include:

  • Weibull Distribution
  • Gamma Distribution
  • Log-Normal Distribution
  • Log-Logistic Distribution

Survival functions for these distributions are derived from their cumulative distribution functions (CDFs), where:

[ S(t) = 1 - \text{CDF}(t) ]

Weibull Distribution

The Weibull distribution is highly versatile and can model increasing, decreasing, or constant hazard rates. Its flexibility makes it suitable for many applications.

[ S(t) = e^{-(\lambda t)^\gamma} ]

  • Parameters:
    • ( \lambda ): scale parameter
    • ( \gamma ): shape parameter
  • Applications: Reliability engineering, medical survival studies

Gamma Distribution

The Gamma distribution can accommodate various shapes of hazard functions, making it useful for modeling skewed survival times.

[ S(t) = 1 - \text{Gamma CDF}(t; \alpha, \beta) ]

  • Parameters:
    • ( \alpha ): shape parameter
    • ( \beta ): rate parameter
  • Applications: Lifespan of technical devices, biological lifetimes

Log-Normal Distribution

The Log-Normal distribution is appropriate when the logarithm of the survival time is normally distributed, often used for right-skewed data.

[ S(t) = 1 - \Phi \left( \frac{\ln t - \mu}{\sigma} \right) ]

  • Parameters:
    • ( \mu ): mean of the logarithm of the survival time
    • ( \sigma ): standard deviation of the logarithm of the survival time
  • Applications: Time-to-event data in finance, medical research

Log-Logistic Distribution

The Log-Logistic distribution can model survival times with a hazard function that increases initially and then decreases, useful for certain biological and industrial data.

[ S(t) = \left(1 + \left(\frac{t}{\alpha}\right)^\beta\right)^{-1} ]

  • Parameters:
    • ( \alpha ): scale parameter
    • ( \beta ): shape parameter
  • Applications: Medical research, industrial reliability studies

Limitations

Parametric methods rely on the assumption that the chosen model fits the data well. If the data does not conform to the assumed model, the estimates can be inaccurate. Key limitations include:

  • Assumption Dependency: The accuracy of parametric models heavily depends on the correctness of the assumed distribution. Mis-specification can lead to biased estimates and incorrect conclusions.
  • Model Selection: Choosing the correct parametric model requires domain knowledge and often involves trial and error, which can be time-consuming.
  • Flexibility: Although more flexible than the exponential model, parametric models may still be less adaptable to complex data patterns compared to non-parametric methods.

In summary, while parametric models offer structured and interpretable ways to estimate survival functions, they require careful selection and validation to ensure they accurately reflect the underlying data patterns. Researchers should consider these limitations and validate their models rigorously.

Non-Parametric Survival Functions

Kaplan-Meier Estimator

Non-parametric methods do not assume a specific shape for the survival function. One of the most widely used non-parametric methods is the Kaplan-Meier estimator, which constructs a step function from the observed data:

\[S(t) = \prod_{t_i \leq t} \left(1 - \frac{d_i}{n_i}\right)\]

where \(d_i\) is the number of events (e.g., deaths or failures) at time \(t_i\), and \(n_i\) is the number of individuals at risk just prior to time \(t_i\).

How It Works

  1. Data Collection:
    • Collect survival data, recording the time of the event for each individual and whether the data is censored (i.e., the event did not occur during the observation period).
  2. Calculate Survival Probabilities:
    • At each event time, calculate the probability of surviving past that time based on the number of events and the number of individuals at risk.
  3. Construct the Step Function:
    • Multiply the survival probabilities sequentially to construct the step function representing the survival curve.

Example Calculation

Consider a dataset with the following survival times and events:

Time (t_i) Events (d_i) At Risk (n_i)
1 1 5
2 1 4
3 1 3
4 0 2
5 1 2

The Kaplan-Meier survival function can be calculated as:

\(S(1) = \left(1 - \frac{1}{5}\right) = 0.80\) \(S(2) = 0.80 \times \left(1 - \frac{1}{4}\right) = 0.60\) \(S(3) = 0.60 \times \left(1 - \frac{1}{3}\right) = 0.40\) \(S(4) = 0.40 \times \left(1 - \frac{0}{2}\right) = 0.40\) \(S(5) = 0.40 \times \left(1 - \frac{1}{2}\right) = 0.20\)

The resulting step function would show the probability of survival at each time point.

Advantages and Limitations

Advantages

  • No Assumptions about the Shape of the Survival Function:
    • The Kaplan-Meier estimator does not impose a predefined shape on the survival function, making it highly flexible and suitable for various types of data.
  • Handles Censored Data:
    • It effectively incorporates censored observations, providing a more accurate estimate of the survival function.
  • Intuitive and Visual Representation:
    • The step function is easy to interpret and visualize, allowing for straightforward comparisons between different groups or treatments.

Limitations

  • No Extrapolation:
    • The Kaplan-Meier estimator cannot extrapolate beyond the observed data points. Its predictions are limited to the range of the collected data.
  • Stepwise Nature:
    • The resulting step function can be less smooth and harder to interpret for predictive purposes compared to parametric models.
  • Data Intensity:
    • Requires a large sample size to provide precise estimates, especially for long-term survival probabilities.

The Kaplan-Meier estimator is a robust and flexible tool for estimating survival functions without assuming a specific parametric form. Its ability to handle censored data and provide an intuitive step function makes it valuable in many applications. However, its limitations in extrapolation and the stepwise nature of the survival curve should be considered when choosing the appropriate method for survival analysis.

Machine Learning Approaches

Machine learning models can also be used to estimate survival functions, providing a flexible and powerful alternative to traditional statistical methods. These models can predict curves and handle complex patterns in the data without strict assumptions.

Common Machine Learning Models for Survival Analysis

  1. Survival Trees and Random Forests:
    • These models extend decision trees to handle censored survival data. Random Survival Forests aggregate multiple survival trees to improve prediction accuracy and robustness.
    • Advantages: Can model complex interactions and non-linear relationships; handles high-dimensional data well.
    • Limitations: Interpretation can be challenging; requires tuning of hyperparameters.
  2. Cox Proportional Hazards Model:
    • Although traditionally a semi-parametric model, machine learning techniques like regularization (LASSO, Ridge) and gradient boosting can enhance its performance.
    • Advantages: Widely used and interpretable; can handle large datasets and incorporate many covariates.
    • Limitations: Assumes proportional hazards, which may not hold in all datasets.
  3. Neural Networks for Survival Analysis:
    • Deep learning models, such as DeepSurv and neural networks with Cox regression, can capture complex, non-linear relationships in survival data.
    • Advantages: High flexibility and ability to model complex interactions; can be combined with other neural network architectures.
    • Limitations: Requires large datasets and significant computational resources; potential for overfitting.
  4. Support Vector Machines (SVM) for Survival Analysis:
    • SVM can be adapted for survival analysis by modifying the loss function to handle censored data.
    • Advantages: Effective in high-dimensional spaces; robust to overfitting with appropriate kernel functions.
    • Limitations: Interpretation is less straightforward; requires careful tuning of kernel and regularization parameters.

Advantages of Machine Learning Models

  • Flexibility:
    • Machine learning models do not require strict assumptions about the distribution of survival times. This allows them to model complex and non-linear relationships within the data.
  • Handling High-Dimensional Data:
    • These models can effectively handle large numbers of covariates, making them suitable for modern datasets with many features.
  • Predictive Performance:
    • Machine learning models often provide superior predictive accuracy compared to traditional parametric methods, especially in datasets with intricate patterns and interactions.

Limitations of Machine Learning Models

  • Interpretability:
    • Many machine learning models, especially deep learning approaches, are considered “black boxes.” This can make it difficult to interpret the results and understand the underlying relationships in the data.
  • Computational Complexity:
    • These models often require substantial computational resources and time, particularly when training on large datasets.
  • Overfitting:
    • Without careful regularization and validation, machine learning models can overfit the training data, reducing their generalizability to new data.

Practical Applications

  • Healthcare:
    • Predicting patient outcomes and survival times based on complex medical records and imaging data.
  • Finance:
    • Estimating the time to default for loans and credit risk assessment.
  • Marketing:
    • Modeling customer lifetime value and predicting churn based on customer behavior and transaction history.

Machine learning approaches offer a powerful and flexible alternative for survival analysis, capable of modeling complex patterns in the data without stringent assumptions. While they present challenges in terms of interpretability and computational demands, their advantages in handling high-dimensional data and improving predictive performance make them valuable tools in various fields. Careful model selection, validation, and tuning are essential to harness the full potential of machine learning in survival analysis.

Applications of Survival Analysis

Beyond Medical Research

Survival analysis is not limited to medical studies. It is useful in scenarios involving incomplete data, known as right censoring. Here are some common applications:

  • Customer Churn Rate:
    • Purpose: Estimate how long customers will continue using a service.
    • Application: Businesses can use survival analysis to identify patterns and predict when customers are likely to churn. This helps in developing retention strategies and improving customer satisfaction.
    • Example: A subscription-based service may analyze customer data to predict the likelihood of customers canceling their subscriptions within the next year.
  • Product Return Rates:
    • Purpose: Predict the rate at which sold items are returned.
    • Application: Retailers can use survival analysis to understand return behavior and manage inventory and return policies effectively.
    • Example: An e-commerce platform might analyze return rates of electronic gadgets to anticipate the volume of returns and plan logistics accordingly.
  • Lifetime Value of Shoppers:
    • Purpose: Estimate the total value a customer will bring over time.
    • Application: Businesses can forecast the future revenue from customers, enabling better financial planning and targeted marketing campaigns.
    • Example: An online retailer could use past purchase data to estimate the future spending of a new customer over a defined period.

Additional Applications

  • Engineering and Reliability:
    • Purpose: Assess the reliability and failure rates of components or systems.
    • Application: Engineers use survival analysis to predict the lifespan and maintenance schedules of machinery and equipment.
    • Example: In the automotive industry, manufacturers might analyze the failure rates of car parts to improve design and warranty policies.
  • Finance:
    • Purpose: Estimate the time to default for loans and assess credit risk.
    • Application: Financial institutions use survival analysis to predict the likelihood of borrowers defaulting on loans, which helps in risk management and decision-making.
    • Example: A bank might analyze loan repayment data to predict the probability of default for different borrower profiles.
  • Sociology and Demography:
    • Purpose: Study time-to-event data in social and demographic research.
    • Application: Researchers can analyze events such as marriage, divorce, or employment duration to understand societal trends and factors influencing these events.
    • Example: A study might use survival analysis to examine the duration of marriages and identify factors associated with higher divorce rates.

Survival analysis is a versatile tool with applications far beyond medical research. Its ability to handle censored data and provide insights into time-to-event relationships makes it invaluable in various fields such as business, engineering, finance, and sociology. By leveraging survival analysis, organizations and researchers can make informed decisions, optimize strategies, and gain a deeper understanding of the dynamics affecting their areas of interest.

Conclusion

Survival analysis offers powerful tools for dealing with incomplete data and predicting the time until events occur. By understanding both parametric and non-parametric methods, researchers and analysts can choose the best approach for their specific needs.

Key Takeaways

  • Flexibility and Applicability:
    • Survival analysis can be applied across various domains such as medical research, engineering, finance, and marketing. Its versatility in handling different types of time-to-event data makes it an essential technique.
  • Parametric Methods:
    • These methods assume a specific distribution for survival times, providing smooth and continuous estimates. They are suitable for scenarios where the data closely follows the assumed model, allowing for extrapolation beyond observed data.
  • Non-Parametric Methods:
    • Methods like the Kaplan-Meier estimator do not assume a specific shape for the survival function, making them highly flexible and robust. They are particularly useful when the true distribution of survival times is unknown or complex.
  • Machine Learning Approaches:
    • Machine learning models offer a powerful alternative, capable of modeling complex patterns without strict assumptions. These models can handle high-dimensional data and provide superior predictive performance in many cases.

Practical Implications

  • Model Selection and Validation:
    • Choosing the appropriate model and validating its assumptions are crucial steps in survival analysis. Researchers should use statistical tests, graphical methods, and comparative analyses to ensure the accuracy and reliability of their models.
  • Handling Censored Data:
    • Survival analysis techniques are specifically designed to handle censored data, providing more accurate and meaningful estimates than traditional methods.

Future Directions

As data availability and computational power continue to grow, the integration of machine learning with traditional survival analysis methods is likely to expand. This hybrid approach can leverage the strengths of both fields, offering even more robust and flexible tools for analyzing time-to-event data.

Final Thoughts

By leveraging the appropriate survival analysis methods, researchers and analysts can gain valuable insights into the timing and occurrence of critical events. This enables better decision-making, strategy development, and understanding of underlying processes across various fields. Whether using parametric, non-parametric, or machine learning approaches, the key is to carefully consider the specific needs and characteristics of the data at hand.

Appendix: Python Code Examples

This appendix provides Python code examples for performing survival analysis using both parametric and non-parametric methods.

Setup

First, ensure you have the necessary libraries installed:

!pip install lifelines matplotlib

Import Libraries

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from lifelines import KaplanMeierFitter, WeibullFitter, CoxPHFitter

Sample Data

Let’s create a sample dataset for demonstration:

data = {
    'duration': [5, 6, 6, 2, 4, 4, 1, 2, 3, 3],
    'event_observed': [1, 0, 1, 1, 1, 1, 1, 0, 1, 1]
}
df = pd.DataFrame(data)

Kaplan-Meier Estimator

The Kaplan-Meier estimator is a non-parametric method to estimate the survival function:

kmf = KaplanMeierFitter()
kmf.fit(durations=df['duration'], event_observed=df['event_observed'])

# Plotting the survival function
kmf.plot_survival_function()
plt.title('Kaplan-Meier Survival Curve')
plt.xlabel('Time')
plt.ylabel('Survival Probability')
plt.show()

Weibull Model

The Weibull model is a flexible parametric method:

wf = WeibullFitter()
wf.fit(durations=df['duration'], event_observed=df['event_observed'])

# Plotting the survival function
wf.plot_survival_function()
plt.title('Weibull Survival Curve')
plt.xlabel('Time')
plt.ylabel('Survival Probability')
plt.show()

Cox Proportional Hazards Model

The Cox Proportional Hazards model is a semi-parametric method often enhanced with machine learning techniques:

# Adding a covariate for demonstration
df['age'] = [50, 60, 65, 45, 55, 50, 40, 70, 60, 50]

cph = CoxPHFitter()
cph.fit(df, duration_col='duration', event_col='event_observed')

# Print the summary
cph.print_summary()

# Plotting the survival function
cph.plot()
plt.title('Cox Proportional Hazards Model')
plt.show()

Random Survival Forest (continued)

For more complex data, a Random Survival Forest can be used. This requires the scikit-survival library:

!pip install scikit-survival

from sksurv.ensemble import RandomSurvivalForest
from sksurv.util import Surv

# Prepare the data
data_y = Surv.from_dataframe('event_observed', 'duration', df)
data_x = df[['age']]

# Fit the model
rsf = RandomSurvivalForest(n_estimators=100, min_samples_split=10)
rsf.fit(data_x, data_y)

# Plotting the survival function for a specific individual
individual = np.array([[50]])  # Age 50
survival_prob = rsf.predict_survival_function(individual)
times = np.linspace(0, 10, 100)

plt.step(times, survival_prob[0](times), where="post")
plt.title('Random Survival Forest - Survival Function')
plt.xlabel('Time')
plt.ylabel('Survival Probability')
plt.show()

This appendix has provided code examples for performing survival analysis using both parametric and non-parametric methods. The examples demonstrate how to fit and visualize survival functions using the Kaplan-Meier estimator, Weibull model, Cox Proportional Hazards model, and Random Survival Forest. These tools offer a range of approaches to handle different types of survival data, helping researchers and analysts to choose the best method for their specific needs.

You may also enjoy