Model Harness¶
This document describes the model harness contract and the concrete harness classes currently shipped in the repository.
Base Class Contract¶
All harnesses inherit from BaseModelHarness in src/apeiron/model/torch_model_harness.py.
Required methods:
Method |
Return |
Purpose |
|---|---|---|
|
|
Returns optimizer used by continual learning loops. |
|
|
Advances stream state and rebuilds current loaders. |
|
|
Returns the loader for the current stream of data. |
|
|
Returns loaders for the current drift state. |
|
|
Returns historical replay loaders used by CL methods. |
|
callable loss fn |
Returns criterion compatible with model output/labels. |
Deprecated since version 0.1.0: Earlier versions spelled the optimizer hook get_optmizer(). That name still
works: a harness that implements get_optmizer() is bridged onto
get_optimizer() automatically and raises a DeprecationWarning at class
definition. Rename the method to get_optimizer() — the misspelled alias will
be removed in a future release.
Important inherited behavior:
eval()loops overget_train_dataloaders()[1]and aggregates all metrics inself.eval_metrics.history_eval()loops over historical validation data and returnsNoneif there is no history._unpack(batch)assumes(x, y)tuples. Override it if your loader format differs.
Required/expected attributes:
Attribute |
Type |
Meaning |
|---|---|---|
|
|
Model moved to |
|
|
Frozen configuration object. |
|
|
Ordered metric map. Metric order controls |
|
|
Used by |
Checkpointing¶
The base class provides automatic checkpointing when configured via [model] section:
[model]
name = "my_model"
max_ckpts = 5 # Keep last 5 checkpoints (0 disables)
ckpts_path = "./output/checkpoints"
Checkpoints are saved automatically after each continual learning adaptation.
Default Checkpoint Format¶
By default, BaseModelHarness saves only model weights (model.state_dict()):
# Default behavior - saves weights only
torch.save(model.state_dict(), checkpoint_path)
This works for models that don’t require additional metadata (preprocessing state, feature names, etc.) to be loaded.
Custom Checkpoint Format¶
If your model requires metadata beyond weights (e.g., preprocessing scalers, feature names, architecture parameters), override build_checkpoint_payload():
class MyHarness(BaseModelHarness):
def __init__(self, cfg: Config):
# ... load model with metadata ...
self._feature_names = checkpoint["feature_names"]
self._scaler = checkpoint["scaler"]
self._input_dim = checkpoint["input_dim"]
super().__init__(cfg=cfg, model=model)
def build_checkpoint_payload(self) -> dict:
"""Save full checkpoint with metadata."""
return {
"model_state_dict": self.model.state_dict(),
"feature_names": self._feature_names,
"scaler": self._scaler,
"input_dim": self._input_dim,
}
This ensures that:
Checkpoints saved during CL runs can be loaded directly as pretrained models
No separate “reference” checkpoints are needed to recover metadata
Save and load formats match exactly
Checkpoint Management¶
Checkpoints are named
drift_adaptation_<event>.ptwhere<event>is the drift event numberA
latestfile tracks the most recent checkpointFIFO policy: oldest checkpoints are deleted when
max_ckptslimit is exceededSet
max_ckpts = 0to disable checkpointing entirely
Runtime Lifecycle¶
examples/utils.py:get_examplepicks a harness fromcfg.data.name.ContinuousMonitor.run()callsmodelHarness.update_data_stream()once before monitoring starts.Monitoring evaluates per-batch metrics from
eval_metrics.On drift,
ContinuousTrainerpulls both current and historical loaders from the harness.
Config Keys Used By Harnesses¶
Config key |
Used by |
Meaning |
|---|---|---|
|
CIFAR/ImageNet harnesses |
Backbone architecture name used by |
|
All provided harnesses |
Optional checkpoint to load before streaming starts. |
|
example selector + CIFAR internals |
Chooses harness ( |
|
ImageNet harness |
Root directory containing |
|
all harnesses |
Loader batch size. |
|
all harnesses |
DataLoader worker count. |
|
all harnesses |
Initial optimizer learning rate. |
|
all harnesses |
Drives deterministic drift augmentation sampling ( |
|
base class + checkpoint load |
Device placement for model/tensors. |
Concrete Harnesses¶
MNIST_CNN (examples/mnist/model.py)¶
Default model: internal
Cnn.Criterion:
torch.nn.NLLLoss.Optimizer: Adam over all model params with
cfg.train.init_lr.Data source: torchvision MNIST from
./data(downloads automatically if missing).Drift mechanism: affine transform sampled each stream update and applied to full train/val views.
Historical loaders: available from the second stream update onward (
task_counter > 1).
CIFAR_VISION (examples/cifar/model.py)¶
Model wrapper:
VisionModelCifarwith either CNN-family or ViT-family backend.Criterion:
torch.nn.CrossEntropyLoss.Optimizer: Adam with
cfg.train.init_lr.Data source: torchvision CIFAR from
./data.Supports
cfg.data.nameofcifar10orcifar100internally.get_example(...)currently routes onlycifar10to this harness.
Supported model.name values for CIFAR/ImageNet loaders:
alexnetvgg11vgg16vgg19inceptionresnet18resnet34resnet50resnet101resnext50_32x4dresnext101_32x8ddensenet121densenet169densenet201regnet_x_400mfregnet_x_8gfregnet_x_16gfvit16bvit16lvit32lvit14hvit14g
IMAGENET_VISION (examples/imagenet/model.py)¶
Model wrapper:
VisionModelImageNet.Criterion:
torch.nn.CrossEntropyLoss.Optimizer: Adam with
cfg.train.init_lr.Data source:
torchvision.datasets.ImageFolder.Requires
cfg.data.pathwith this layout:<path>/train/<class_id>/*.JPEG<path>/val/<class_id>/*.JPEG
Common Pitfalls¶
metric_indexin drift detection is position-based, not name-based. Keep metric order stable ineval_metrics.get_stream_dataloader()must return a non-Noneloader afterupdate_data_stream().get_train_dataloaders()must return non-Noneloaders afterupdate_data_stream().If your batch is not exactly
(x, y), override_unpackin your harness.
Creating a Custom Model Harness¶
A custom model harness is the integration point between a user model and Apeiron. The harness allows Apeiron to monitor model performance, detect drift, trigger continual learning, and manage retraining without requiring modifications to the underlying scientific model.
Required Files¶
A custom example typically consists of:
examples/<example_name>/
├── __init__.py
├── model.py
├── utils.py
└── <example_name>.toml
In addition, the example must be registered in:
examples/utils.py
through the get_example() factory.
Step 1: Create the Harness Class¶
All custom harnesses must inherit from BaseModelHarness.
class MyHarness(BaseModelHarness):
def __init__(self, cfg):
model = MyModel()
super().__init__(cfg=cfg, model=model)
The base class handles:
Device placement
Checkpoint support
Evaluation loops
Continual-learning integration
Drift-monitor integration
Step 2: Define Evaluation Metrics¶
Apeiron’s drift detectors operate on metric streams.
At least one metric should be exposed through:
self.eval_metrics = {
"accuracy": accuracy,
}
or for regression:
self.eval_metrics = {
"mse": regression_mse,
}
The order of metrics is important because drift detection uses:
metric_index = 0
which refers to the first metric in eval_metrics.
Step 3: Implement Required Methods¶
Optimizer¶
def get_optimizer(self):
return torch.optim.Adam(
self.model.parameters(),
lr=self.cfg.train.init_lr,
)
Criterion¶
def get_criterion(self):
return torch.nn.CrossEntropyLoss()
or for regression:
def get_criterion(self):
return torch.nn.MSELoss()
Current Training Loaders¶
def get_train_dataloaders(self):
return self.train_loader, self.val_loader
These loaders are used during continual-learning updates.
Stream Loader¶
def get_stream_dataloader(self):
return self.stream_loader
This loader is used during monitoring and drift detection.
Historical Loaders¶
def get_hist_dataloaders(self):
if self.task_counter == 0:
return None, None
return self.hist_train_loader, self.hist_val_loader
Historical loaders provide replay data for continual-learning algorithms such as EWC and KFAC.
Stream Updates¶
def update_data_stream(self):
self.task_counter += 1
self.train_loader = build_train_loader()
self.val_loader = build_val_loader()
self.stream_loader = build_stream_loader()
This function is called by the monitor whenever a new stream segment becomes active.
(Optional) Custom Checkpoint Format¶
If your model requires metadata beyond weights (e.g., preprocessing scalers, feature names, architecture parameters), override build_checkpoint_payload():
def build_checkpoint_payload(self):
"""Save checkpoint with all metadata needed for loading."""
return {
"model_state_dict": self.model.state_dict(),
"feature_names": self._feature_names,
"scaler": self._scaler,
"input_dim": self._input_dim,
}
This ensures saved checkpoints can be loaded directly without separate reference files.
When to use:
Your model loads with preprocessing state (scalers, normalizers, encoders)
Your architecture requires parameters beyond what’s in the config (e.g., inferred input dimensions)
You want checkpoints to be self-contained and portable
When to skip:
Your model only needs weights (
state_dict()) - the default behavior is fineAll loading parameters come from the config file
See the “Checkpointing” section above for details.
Step 4: Build Dataset Utilities¶
Dataset loading should be implemented in utils.py.
Typical responsibilities include:
Reading datasets
Feature preprocessing
Label extraction
Train/validation splitting
Drift simulation
DataLoader construction
Example:
def make_loader(dataset, batch_size):
return DataLoader(
dataset,
batch_size=batch_size,
shuffle=True,
)
Step 5: Register the Example¶
Add a branch to examples/utils.py:
elif cfg.data.name == "mydataset":
from examples.mydataset.model import MyHarness
return MyHarness(cfg=cfg)
The value must match:
[data]
name = "mydataset"
Step 6: Create the Configuration File¶
Create:
examples/<example_name>/<example_name>.toml
Required and optional parameters are described in configurations.md.
Only keys defined in configuration.py are allowed. Custom dataset-specific parameters should be implemented inside the harness or utility code unless the configuration dataclasses are extended.
Step 7: Validate the Integration¶
Validate TOML syntax:
python -c "import tomllib; tomllib.load(open('examples/mydataset/mydataset.toml', 'rb')); print('TOML OK')"
Validate factory registration:
uv run python -c "from examples.utils import get_example; print('factory OK')"
Step 8: Run a Smoke Test¶
Before running a full experiment, execute a small CPU-only test:
uv run python -m src.main \
--config examples/mydataset/mydataset.toml \
--set train.max_iter=2 \
--set drift_detection.max_stream_updates=2 \
--set drift_detection.detection_interval=1 \
--set device=cpu \
--set logging.backend=none
A successful smoke test confirms:
Configuration loading
Harness construction
Dataset loading
Drift-monitor integration
Continual-learning integration
Metric reporting
Common Errors¶
KeyError: 'model'¶
The TOML file is missing the [model] section.
ImportError when loading the harness¶
The factory registration in examples/utils.py is missing or incorrect.
TypeError: DataCfg.__init__() got an unexpected keyword argument ...¶
A TOML key is not defined in the configuration dataclasses.
IndexError: list index out of range during drift detection¶
No evaluation metrics are being emitted. Ensure self.eval_metrics contains at least one metric.
get_stream_dataloader() returns None¶
The monitor cannot evaluate data until update_data_stream() constructs and returns a valid stream loader.
Summary¶
A custom harness only needs to provide six core behaviors:
Construct the model.
Construct the optimizer.
Construct the criterion.
Provide current train/validation loaders.
Provide historical replay loaders.
Update the active data stream.
Once those interfaces are implemented, Apeiron can automatically provide monitoring, drift detection, continual learning, checkpointing, logging, and experiment management for the underlying scientific model.