This hub is for the work that sits between a question and a model: exploring a dataset, choosing what to measure, deciding whether an intervention worked, and keeping an analysis honest once it runs in production. It is the second-largest subject on the site, and many of its articles are applied case studies.
Start here
- Exploratory Data Analysis: A Beginner's Guide
- Understanding Statistical Significance in Data Analysis
- Exploring Kernel Density Estimation
- Synthetic Control: Evaluating an Intervention on One Unit
Analysis in Python
Exploratory analysis with pandas, density estimation, entropy, outlier and anomaly detection, and worked examples in the scientific Python stack.
A Database for Analysis: Rows, Columns, Indexes and the Planner
The same five million orders answer an analytical query in 2 milliseconds or in 11 seconds, and a key lookup in 17 microseconds or 570, depending on how the ...
Read article
Exploratory Data Analysis: A Beginner's Guide
Discover the essential steps of Exploratory Data Analysis (EDA) and how to gain insights from your data before building models.
Read article
Exploring Kernel Density Estimation: A Powerful Tool for Data Analysis
Kernel Density Estimation (KDE) is a non-parametric technique offering flexibility in modeling complex data distributions, aiding in visualization, density e...
Read article
Understanding Normality Tests: A Deep Dive into Their Power and Limitations
An in-depth look at normality tests, their limitations, and the necessity of data visualization.
Read article
Exploratory Data Analysis (EDA) Techniques with Pandas
Explore how to perform effective Exploratory Data Analysis (EDA) using Pandas, a powerful Python library. Learn data loading, cleaning, visualization, and ad...
Read article
Entropy in Data Science and Machine Learning: A Deep Dive
Explore the deep connection between entropy, data science, and machine learning. Understand how entropy drives decision trees, uncertainty measures, feature ...
Read article
Validating Anomaly Detection Models: Lessons from COPOD
COPOD is a popular anomaly detection model, but how well does it perform in practice? This article discusses critical validation issues in third-party models...
Read article
Building Energy Efficiency Analysis with Python and Machine Learning
Explore how Python and machine learning can be applied to analyze and improve building energy efficiency. Learn key techniques for assessing sustainability, ...
Read articleStatistical modelling in practice
Generalised linear models, latent class analysis, count models, missing data in clinical research, and choosing between tests.
Synthetic Control: Evaluating an Intervention on One Unit
One plant got the new maintenance regime. Before-after says it did nothing, because demand was rising. Comparing against the other plants says it did too lit...
Read articleExploring Kernel Density Estimation: A Powerful Tool for Data Analysis
Kernel Density Estimation (KDE) is a non-parametric technique offering flexibility in modeling complex data distributions, aiding in visualization, density e...
Read articleUnderstanding Normality Tests: A Deep Dive into Their Power and Limitations
An in-depth look at normality tests, their limitations, and the necessity of data visualization.
Read article
T-Test vs. Z-Test: When and Why to Use Each
This article provides an in-depth comparison between the t-test and z-test, highlighting their differences, appropriate usage, and real-world applications, w...
Read article
Latent Class Analysis: Unveiling Hidden Patterns in Data
Introduction
Read articleStatistical Analysis with Generalized Linear Models
Introduction
Read article
Handling Missing Data in Clinical Research
Abstract
Read article
Data Analysis Skills with Z-Scores: A Quick Guide
Understanding the z-score can significantly enhance your data analysis skills. Here's a quick guide to what z-scores are and why they matter:
Read articleDecisions and interventions
Did the change work, and for whom? Synthetic control, uplift modelling and counterfactual evaluation of decision policies.
Counterfactual Evaluation for Decision Policies
Counterfactual evaluation helps teams estimate how a new decision policy might perform before deploying it to users, patients, customers, or operations.
Read articleUplift Modeling for Targeted Interventions
Uplift modeling estimates treatment effect heterogeneity so interventions can target the people, assets, or cases most likely to benefit.
Read articleSynthetic Control: Evaluating an Intervention on One Unit
One plant got the new maintenance regime. Before-after says it did nothing, because demand was rising. Comparing against the other plants says it did too lit...
Read articleModels and data in production
Silent data-quality failures, drift detection, validating anomaly detectors, monitoring with wearables and IoT sensors, and what to check before deployment.
Silent Failures: When the Pipeline Changes and the Metric Moves
The average order value fell two percent overnight and three teams spent a day looking for the cause. Nothing about customers changed. The enrichment service...
Read article
Prevalence Shift and Base-Rate Drift in Machine Learning
Prevalence shift occurs when the base rate of the outcome changes, breaking thresholds, workloads, and probability interpretation even when the model ranking...
Read article
The Impact of Predictive Maintenance on Operational Efficiency: A Data Science Perspective
A data-driven investigation into predictive maintenance's operational value across industries, exploring statistical models, machine learning architectures, ...
Read articleRemote Monitoring and Elderly Care: How IoT and Big Data are Keeping Seniors Safe
The integration of IoT and big data is revolutionizing elderly care by enabling remote monitoring systems that track vital signs, detect emergencies, and ens...
Read articleUsing Wearable Technology and Big Data for Health Monitoring
Wearable devices generate real-time health data that, combined with big data analytics, offer transformative insights for chronic disease monitoring, early d...
Read articleData Science Projects: Ensuring Success Before Deployment
This checklist helps Data Science professionals ensure thorough validation of their projects before declaring success and deploying models.
Read articleValidating Anomaly Detection Models: Lessons from COPOD
COPOD is a popular anomaly detection model, but how well does it perform in practice? This article discusses critical validation issues in third-party models...
Read article
The Limitations of Hypothesis Testing for Detecting Data Drift: A Bayesian Alternative
Explore the challenges of using traditional hypothesis testing for detecting data drift in machine learning models and learn how Bayesian probability offers ...
Read articleHealthcare and ageing
Readmission risk, fall prediction, remote monitoring, chronic disease, clinical text and the ethics of AI in elderly care.
Predicting Hospital Readmissions for Elderly Patients Using Machine Learning
Machine learning models are revolutionizing post-hospitalization care by predicting hospital readmissions in elderly patients, helping healthcare providers o...
Read articleRemote Monitoring and Elderly Care: How IoT and Big Data are Keeping Seniors Safe
The integration of IoT and big data is revolutionizing elderly care by enabling remote monitoring systems that track vital signs, detect emergencies, and ens...
Read articleData-Driven Approaches to Managing Chronic Diseases in the Elderly
Data science is revolutionizing chronic disease management among the elderly by leveraging predictive analytics to monitor disease progression, manage medica...
Read articleUsing Machine Learning to Predict and Prevent Falls in the Elderly
Machine learning is revolutionizing fall prevention in elderly care by predicting the likelihood of falls through wearable sensor data, mobility analysis, an...
Read articleUsing Wearable Technology and Big Data for Health Monitoring
Wearable devices generate real-time health data that, combined with big data analytics, offer transformative insights for chronic disease monitoring, early d...
Read article
Natural Language Processing (NLP) in Healthcare: Extracting Insights from Unstructured Data
Natural Language Processing (NLP) is revolutionizing healthcare by enabling the extraction of valuable insights from unstructured data. This article explores...
Read articlePredictive maintenance
Measuring whether a maintenance programme pays for itself, dashboards, cloud and edge analytics, maintenance text, and the effect on operations.
Evaluating the ROI of Predictive Maintenance: A Practical Measurement Framework
Predictive maintenance only creates value when better predictions change maintenance decisions. This article explains how to measure that value without confu...
Read article
Data Visualization and Dashboards for Predictive Maintenance
Predictive maintenance dashboards should not merely display sensor data. They should help teams decide what to inspect, when to act, and which risks matter m...
Read article
Cloud Computing and Edge Analytics in Predictive Maintenance
Predictive maintenance systems rarely live entirely in the cloud or entirely at the edge. Effective architectures split work across sensors, gateways, plant ...
Read article
Label Noise in Supervised Learning: When the Target Cannot Be Trusted
Label noise is one of the most damaging data quality problems in supervised learning because it corrupts the target the model is trained to imitate.
Read article
Competing Risks in Healthcare and Predictive Maintenance
Competing risks occur when more than one event can happen, and one event changes or prevents the chance of observing another.
Read articleApplications of Mathematics and Machine Learning in Industrial Management: A Comprehensive Review
This review explores the transformative applications of mathematical optimization and machine learning in industrial management, with a focus on production p...
Read articleLatest in Data Science
A Database for Analysis: Rows, Columns, Indexes and the Planner
The same five million orders answer an analytical query in 2 milliseconds or in 11 seconds, and a key lookup in 17 microseconds or 570, depending on how the ...
Read article
Why Exact Post-Selection Confidence Intervals Can Be Enormous
An exact 95% interval of [-71.7, 2.5] for a unit-variance Gaussian mean is not a bug. It is what conditioning on a selection event costs when the selected va...
Read articleThe Trouble With Smooth Curves in Small Simulation Studies
Simulation studies often evaluate a method on a coarse parameter grid and then draw a smooth curve through the results. The plot looks persuasive. The proble...
Read articleWhy I Don't Automatically Reach for Machine Learning
Machine learning is often treated as the default destination of a data project. I prefer the opposite order: understand the problem, write down the structure...
Read articleA Negative Monte Carlo Result Is Still a Result
A simulation that refuses to confirm the theory you hoped to see is not a failed simulation. It may be the most useful part of the project. The difficult par...
Read articleSilent Failures: When the Pipeline Changes and the Metric Moves
The average order value fell two percent overnight and three teams spent a day looking for the cause. Nothing about customers changed. The enrichment service...
Read articleSampling Uncertainty Can Dominate Representation Uncertainty
A better representation can recover structure that a linear summary cannot see, but that does not mean representation choice is immediately the main source o...
Read articleWhen One Simulation Metric Tells the Wrong Story
A parameter setting can have the smaller typical estimation error and still be much more sensitive to starting values. Another can look poor under a toleranc...
Read articleEvaluating the ROI of Predictive Maintenance: A Practical Measurement Framework
Predictive maintenance only creates value when better predictions change maintenance decisions. This article explains how to measure that value without confu...
Read articleEvery article in this subject is listed under Data Science in the category index. For neighbouring subjects see Statistics, Machine Learning and Research Methods.