Profilers¶
FLOP and wall-time measurement built on PyTorch’s FlopCounterMode. See
FLOPS Profiler for usage.
Compute performance using PyTorch FlopCounterMode with timing.
FlopCounterMode Limitations & Coverage:¶
COVERED OPERATIONS (with FLOP counting): - Linear layers (nn.Linear, F.linear): matmul + bias - Convolution (nn.Conv1d/2d/3d): conv operations - Matrix multiplication (torch.matmul, @, torch.mm, torch.bmm) - Element-wise ops: add, sub, mul, div (count as FLOPs) - Activation functions: ReLU, GELU, SiLU, etc. (counted as element-wise) - Normalization: LayerNorm, BatchNorm (counted) - Loss functions: CrossEntropyLoss, MSELoss (counted as element-wise + reductions)
NOT COVERED or LIMITATIONS: - Indexing, slicing, reshape, transpose (NO FLOPs, just memory ops) - Custom autograd Functions - need explicit FLOP annotations - Some advanced operations - check flop_counter.py for full list Reference: https://github.com/pytorch/pytorch/blob/main/torch/utils/flop_counter.py
Relying on the accuracy of torch’s flop_counter for general use.
Optimizers require FLOP estimation as flop_counter does not automatically count all non-module operations.
For profiling the optimizer step, user must measure_flops_optimizer(tag=”opt”, model=model, device=cfg.device).
- class apeiron.profilers.count_flops.FLOPSProfiler[source]¶
Bases:
objectProfiler for measuring FLOPs and execution time of PyTorch operations.
This class uses PyTorch’s FlopCounterMode to automatically count FLOPs for most operations. For optimizer steps, use measure_flops_optimizer() which employs torch.profiler for accurate FLOP estimation.
- warmup_iters¶
Number of warmup iterations before profiling
- profiles¶
Dictionary storing FLOP and time measurements per tag
- tag¶
Current measurement tag (set during active profiling)
- start_time¶
Start time of current measurement
- flop_counter¶
FlopCounterMode instance for current measurement
- __init__(warmup_iters=1)[source]¶
Initialize the FLOPSProfiler.
- Parameters:
warmup_iters (int) – Number of warmup iterations (default: 1)
- Return type:
None
- measure_flops(tag='default')[source]¶
Context manager for measuring FLOPs and time for a code block.
Uses FlopCounterMode to automatically count FLOPs for supported operations. Best for forward/backward passes of neural network modules.
- Parameters:
tag (str) – Identifier for this measurement session (default: “default”)
- Yields:
self – The profiler instance
- Return type:
Generator[FLOPSProfiler, None, None]
Example
>>> profiler = FLOPSProfiler() >>> with profiler.measure_flops("forward"): ... output = model(input)
- measure_flops_optimizer(model, device, tag='optimizer')[source]¶
Context manager for measuring FLOPs and time for optimizer step.
Uses torch.profiler to estimate FLOPs for optimizer operations, which are not captured by FlopCounterMode. Estimates total FLOPs by multiplying per-element operations by the number of trainable parameters.
- Parameters:
- Yields:
self – The profiler instance
- Return type:
Generator[FLOPSProfiler, None, None]
Example
>>> profiler = FLOPSProfiler() >>> with profiler.measure_flops_optimizer(model, "cuda", "optimizer"): ... optimizer.step()
- get_performance()[source]¶
Calculate average performance metrics across all measurements.
Computes the mean FLOPs, time, and throughput (FLOP/s) for each tag.
- Returns:
{tag}_flop: Average FLOPs per operation
{tag}_time: Average time per operation (seconds)
{tag}_flops: Average throughput (FLOP/s)
- Return type:
Dictionary containing averaged metrics with keys