Drift Detectors¶
This document describes detector classes under src/apeiron/drift_detection/, the drift_detection config section, and how detector outputs drive continual learning.
Core Types¶
Defined in src/apeiron/drift_detection/detectors/base.py:
LearningRegime:stablecontinual_learningfine_tuningretrain
DriftSignalfields:regimedrift_detecteddrift_scoreconfidence(optional)metadata(optional dictionary)
BaseDriftDetectorinterface:update(value: float, **kwargs) -> DriftSignalreset() -> None
Global Drift Config¶
src/apeiron/config/configuration.py defines DriftDetectionCfg:
Key |
Default |
Meaning |
|---|---|---|
|
|
Detector class selected by |
|
|
Check drift every N monitored batches. If |
|
|
How buffered metric values are aggregated before detector update. Supported by monitor: |
|
|
Index into |
|
|
If true, detector state resets after each CL event. |
|
|
Monitoring stops after this many stream extensions. |
|
|
ADWIN confidence/sensitivity parameter. |
|
|
ADWIN regime threshold (CL boundary). |
|
|
ADWIN regime threshold (fine-tuning boundary). |
|
|
KSWIN significance level. |
|
|
KSWIN reference window size. |
|
|
KSWIN recent sample window size. |
|
|
Page-Hinkley warm-up count before detection is meaningful. |
|
|
Page-Hinkley change magnitude parameter. |
|
|
Page-Hinkley trigger threshold. |
|
|
Page-Hinkley forgetting factor. |
|
|
Sub-detector names combined by |
|
|
Voting rule: |
Detector Selection (detector_name)¶
src/apeiron/drift_detection/load_drift_detector.py currently supports:
ADWINDetectorKSWINDetectorPageHinkleyDetectorModelPerformanceDetectorEvalDetector(maps toModelEvalDetector)EnsembleDetector(combines the detectors listed inensemble_detectorsunderensemble_voting)
Detector Classes And Options¶
ADWINDetector (src/apeiron/drift_detection/detectors/statistical_detectors.py)¶
Brief Explanation:
ADaptive WINdowing (ADWIN) keeps a variable-length sliding window of the monitored metric and cuts it in two whenever the means of the two sub-windows differ significantly; a cut signals drift and the window shrinks to the recent regime. It handles both gradual and abrupt shifts, needs no fixed window size, and works well as the general-purpose default for a scalar such as loss or error rate.
Constructor options:
adwin_delta: confidence bound for a window cut. Lower = stricter test, so fewer/later detections and fewer false alarms; Higher = more sensitive. Typical range0.001 - 0.01(default0.002).adwin_minor_threshold: recent-drift-rate boundary below which drift maps to thecontinual_learningregime. Higher pushes more events into CL rather than fine-tuning. Range0 - 1(default0.3).adwin_moderate_threshold: drift-rate boundary above which drift escalates toretrain; between the two thresholds isfine_tuning. Must be>= minor_threshold; range0 - 1(default0.6).name(constructor-only; not exposed in config)
Behavior:
Updates on one scalar value at each check.
Computes
drift_scoreas recent drift frequency over up to 100 updates.Emits regime based on
drift_detectedand score thresholds.
KSWINDetector¶
Brief Explanation:
Kolmogorov-Smirnov WINdowing (KSWIN) runs a two-sample KS test comparing the most recent kswin_stat_size samples against a reference window of kswin_window_size samples, firing when the two distributions differ at significance level kswin_alpha. As it compares full distributions rather than just means, it catches shape/variance changes an averaging detector would miss, making it a fit for cases where the distribution shifts without the mean moving much.
Constructor options:
kswin_alpha: KS-test significance level. Lower = stricter, fewer false alarms but slower to react; Higher = more sensitive. Typical range0.001 -0.01(default0.005).kswin_window_size: total samples retained; the reference window iswindow_size - stat_size. Larger = more stable baseline but more memory and slower adaptation. Typically50 - 300(default100).kswin_stat_size: number of most-recent samples KS-tested against the reference. Larger = smoother but laggier detection. Must be< window_size; typically20 - 50(default30).minor_threshold(constructor default only; not in config loader call): drift-rate boundary for thecontinual_learningregime,0 - 1(default0.3).moderate_threshold(constructor default only; not in config loader call): drift-rate boundary for theretrainregime,0 - 1(default0.6).name(constructor-only)
Behavior:
Uses Kolmogorov-Smirnov windowing from
river.Score is recent drift frequency over up to 50 updates.
PageHinkleyDetector¶
Brief Explanation:
The Page-Hinkley test accumulates the running deviation of the metric from its historical mean and fires once that cumulative sum exceeds ph_threshold. It is cheap, low-memory, and quick to react, making it a good fit for real-time monitoring of abrupt mean shifts.
Constructor options:
ph_min_instances: warm-up samples observed before detection can fire. Higher = more stable baseline but slower first detection. Typically20 - 100(default30).ph_delta: slack subtracted from each deviation, i.e. the smallest change treated as meaningful. Higher = ignores small fluctuations (fewer false alarms); Lower = more sensitive. Typically0.001 - 0.05(default0.005).ph_threshold: cumulative-sum value that triggers drift. Higher = needs a larger/longer shift, so fewer/later detections; Lower = more sensitive. Scale depends on the metric; typically10- 100(default50).ph_alpha: forgetting factor for the running mean; closer to1weights history more (slower to adapt), lower forgets faster. Range just below1, e.g.0.99 - 0.9999(default0.9999).minor_threshold(constructor default only; not in config loader call): drift-rate boundary for thecontinual_learningregime,0 - 1(default0.3).moderate_threshold(constructor default only; not in config loader call): drift-rate boundary for theretrainregime,0 - 1(default0.6).name(constructor-only)
Behavior:
Online mean-shift detector from
river.Score is recent drift frequency over up to 50 updates.
ModelPerformanceDetector (src/apeiron/drift_detection/detectors/model_performance_detector.py)¶
Brief Explanation:
A batch-level detector built on Evidently’s DataDriftPreset that, rather than watching a single scalar, compares a reference dataset (data, predictions, targets captured when the model was healthy) against an incoming batch and flags drift when the share of drifted columns exceeds drift_share_threshold. This gives a richer, feature-aware signal but requires reference data up front, so use it for distribution-level analysis across many features (see the integration note below before wiring it into the monitor).
Constructor options:
reference_data: baseline feature table (pandas.DataFrame) captured when the model was healthy; incoming batches are compared against it.reference_predictions: baseline model predictions aligned withreference_data; optional, enables prediction-drift analysis.reference_targets: baseline ground-truth labels; optional, enables target/performance-drift analysis.drift_share_threshold: fraction of columns that must be flagged as drifted before overall drift fires. Higher = more tolerant (fewer alarms), lower = more sensitive. Range0 - 1(Evidently default0.5).minor_threshold: drift-score boundary for thecontinual_learningregime,0- 1(default0.3).moderate_threshold: drift-score boundary for theretrainregime,0- 1(default0.6).name
Behavior modes:
Batch mode: accepts
data(pandas.DataFrame) and optionalpredictions/targets, runs EvidentlyDataDriftPreset.Scalar fallback mode: if given only
value, uses internal simple score logic.
Integration note:
The class requires reference initialization (
set_reference(...)or constructor reference data) beforeupdate(...).Current loader path instantiates it without reference data, so it is not plug-and-play in
ContinuousMonitorwithout additional initialization code.
ModelEvalDetector (detector_name = "EvalDetector")¶
Brief Explanation:
The most direct detector: on each update it re-evaluates the model via modelHarness.eval() and compares each metric against a reference value, respecting a per-metric higher_is_better flag, declaring drift when current performance falls short of the reference. It is a straightforward “has the model gotten worse?” check. It is best to use when you have trustworthy reference validation metrics and want an interpretable signal.
Constructor options:
nameonly.
Expected update(...) kwargs:
modelHarness: the harness whose.eval()is called to compute current validation metrics.reference_validation_metrics: baseline metric values to compare against; must be the same length/order asmodelHarness.eval()output.higher_is_better: per-metric dict/flags stating direction of “good”;Truemeans a drop below reference counts as drift,Falsemeans a rise above reference does.
Integration note:
ContinuousMonitor._check_drift()currently passes only one scalar metric toupdate(...).Using
EvalDetectorin the current monitor flow requires extra wiring to pass the expected kwargs.
EnsembleDetector¶
Brief Explanation:
Wraps several sub-detectors and combines their signals under a voting rule so you can trade sensitivity against false-alarm rate: any reacts to the first detector that fires, unanimous requires full agreement, majority sits in between.
Config options:
ensemble_detectors: list of sub-detector names to combine, e.g.["ADWINDetector", "KSWINDetector"]. Each is built from the same[drift_detection]block, so a given detector type can appear at most once. Required; an empty list raisesValueError.ensemble_voting: how the sub-detector verdicts combine (defaultmajority):majority– fires when strictly more than half the detectors fire. Balanced.any(aliasor) – fires when at least one detector fires. Most sensitive.unanimous(aliasesall,and) – fires only when every detector fires. Most conservative. Names are case-insensitive; an unrecognized value raisesValueErrorat load time.
Behavior:
All sub-detectors are updated on every call (no short-circuiting), so
detector.reset()and internal windows stay in sync.The returned
drift_scoreis the mean sub-detector score andconfidencethe mean of the non-Nonesub-detector confidences (Noneif no detector reports one), regardless of the voting rule.The regime is a plurality vote over the sub-detector regimes, independent of
drift_detected.metadatacarriesvoting,n_votes,n_detectors, and the per-detector verdicts.
Example:
[drift_detection]
detector_name = "EnsembleDetector"
ensemble_detectors = ["ADWINDetector", "KSWINDetector", "PageHinkleyDetector"]
ensemble_voting = "majority"
detection_interval = 10
# Sub-detectors read their usual hyperparameters from this same block
adwin_delta = 0.002
kswin_alpha = 0.005
ph_threshold = 50
How Monitor Uses Detectors¶
src/apeiron/driver/continuous_monitor.py does:
Evaluate validation batches and buffer metric vectors.
At every
detection_interval, extract columnmetric_index.Aggregate buffer by
aggregation(mean/median/last).Call
detector.update(agg_metric).If
drift_detected, run continual learning and optionallydetector.reset().
Practical Config Snippet¶
[drift_detection]
detector_name = "ADWINDetector"
detection_interval = 10
aggregation = "mean"
metric_index = 0
reset_after_learning = false
max_stream_updates = 20
adwin_delta = 0.002
adwin_minor_threshold = 0.3
adwin_moderate_threshold = 0.6
Code for the workflow in the Monitor implementation:
detector = load_drift_detector(cfg)
data = modelHarness.get_stream_dataloader()
for batch_idx, batch in tqdm(
enumerate(val_loader),
desc="Inference on batches",
leave=False,
):
# Inference on batch and compute all metrics
metrics = self._evaluate_batch(batch)
metric_buffer.append(metrics)
metric_idx = cfg.drift_detection.metric_index
metric_values = [m[metric_idx] for m in metric_buffer]
# aggregate metrics
aggregation = cfg.drift_detection.aggregation
if aggregation == "mean":
agg_metric = float(np.mean(metric_values))
elif aggregation == "median":
agg_metric = float(np.median(metric_values))
elif aggregation == "last":
agg_metric = float(metric_values[-1])
drift_signal = detector.update(agg_metric)
if drift_signal.drift_detected:
handle_drift(drift_signal)
self.detector.reset()
modelHarness.update_data_stream()