Real-Time Epidemiological Surveillance: Streaming Data Is Not Enough

Real-time epidemiological surveillance is often framed as a data-engineering problem.

Ingest events quickly.

Process them in a streaming engine.

Update a dashboard.

That is necessary infrastructure.

It is not the surveillance method.

A useful system has two coupled layers:

Low-latency processing cannot compensate for delayed diagnoses, incomplete reports, duplicate records, or a badly calibrated outbreak detector.

Event time and processing time

Streaming systems distinguish the time an event occurred from the time the platform received it.

Let

be event time and

processing time.

In public-health data,

can vary substantially because of:

• laboratory turnaround;
• reporting practices;
• weekends;
• batching;
• data-entry delay;
• network failure.

A system that groups only by processing time can create artificial spikes when delayed records arrive together.

Late data

Modern streaming engines use concepts such as watermarks to decide when an event-time window is sufficiently complete for computation.

But epidemiological reporting has no magical point at which late data cease to exist.

The engineering policy must be connected to the epidemiological reporting-delay model.

A one-hour lateness threshold may be reasonable for telemetry and absurd for laboratory surveillance.

Revisions are part of the data model

Case counts can be revised because records are:

• deduplicated;
• reclassified;
• assigned to a different onset date;
• corrected after laboratory confirmation;
• removed after quality review.

A real-time system must therefore support updates, not only append-only events.

The current estimate for historical date $t$ is a versioned object:

where $v$ is the data vintage.

This is essential for honest retrospective evaluation.

Nowcasting

If the recent count is incomplete, the system can estimate the final count.

Let

be cases occurring at time $t$ and reported with delay $d$.

A nowcast estimates

before all delays have arrived.

The uncertainty in that estimate should propagate into any alerting layer.

Treating the partial count as if it were final biases recent trends downward.

Streaming architecture

A generic architecture can include:

1. event ingestion;
2. schema validation;
3. deduplication;
4. event-time assignment;
5. late-data handling;
6. aggregation;
7. statistical model update;
8. alert generation;
9. audit logging.

The streaming platform may be Apache Flink, Kafka Streams, Spark Structured Streaming, or another system.

The statistical requirements should not depend on one vendor.

Why Flink can be useful

Flink supports stateful event-time processing and watermarks.

Those capabilities fit surveillance streams where records arrive out of order.

For example, a keyed stream can maintain counts by:

• region;
• syndrome;
• age group;
• laboratory;
• time window.

Stateful processing makes it possible to update derived quantities as new reports arrive.

The epidemiological validity still depends on how those quantities are defined.

Idempotence and duplicate reports

Health records may be resent.

If a repeated message increments the count twice, the streaming system creates a false outbreak.

Events should have stable identifiers when possible.

The update operation should be idempotent:

Applying the same event twice should not change the state after the first application.

This is a data-quality requirement with direct statistical consequences.

Alerting is a sequential-testing problem

Suppose each region is evaluated every hour.

A naive rule

at each time point produces repeated opportunities for false alarms.

Long-run false-alert behavior depends on the entire sequential procedure.

Useful performance measures include:

• alerts per month under baseline;
• average run length;
• detection delay;
• probability of detection;
• calibration across regions.

One threshold from a static hypothesis test is not enough.

Baselines drift

Healthcare usage changes over time.

So do:

• population size;
• diagnostic access;
• coding practice;
• vaccination coverage;
• circulating pathogens;
• reporting systems.

An outbreak detector needs a baseline model that can adapt without immediately absorbing the outbreak it is supposed to detect.

That creates a bias-variance trade-off in baseline updating.

Privacy and access control

Real-time health streams can contain highly sensitive information.

System design should minimize unnecessary identifiers and enforce access control.

Useful principles include:

• data minimization;
• role-based access;
• encryption;
• audit trails;
• retention policy;
• aggregation when individual-level detail is unnecessary.

Privacy is not a downstream dashboard setting.

It belongs in the data architecture.

Reproducibility

A dashboard value should be traceable to:

• source records;
• schema version;
• transformation code;
• model version;
• configuration;
• data vintage.

Without lineage, a real-time number cannot be reproduced after the stream has changed.

This is especially important when public-health decisions depend on an alert.

Failure modes

A surveillance platform can fail while every server remains healthy.

Examples include:

• silent underreporting;
• a laboratory feed disappearing;
• duplicated messages;
• changed coding standards;
• delayed batch arrival;
• demographic fields becoming missing;
• geographic reassignment.

Monitoring should therefore include data-quality metrics, not only CPU and latency.

Machine learning is optional

An outbreak detector does not need a neural network.

Simple count models, state-space models, CUSUM procedures, or Bayesian surveillance models can be easier to calibrate and audit.

Machine learning may help with:

• text classification;
• syndrome extraction;
• anomaly scoring;
• missing-data prediction.

The model should enter because it solves a defined problem, not because the architecture is “real time.”

Conclusion

Real-time surveillance is not just fast data.

The pipeline is

A streaming framework can implement the pipeline.

It cannot define the epidemiological model for us.

References

• Höhle, M. (2007). Surveillance: An R package for the monitoring of infectious diseases. Computational Statistics, 22, 571–582.
• Salmon, M., Schumacher, D., & Höhle, M. (2016). Monitoring count time series in R: Aberration detection in public health surveillance. Journal of Statistical Software, 70(10).
• Apache Flink documentation. Event time and watermarks.