Skip to content

Changelog

Changelog

All notable changes to echelon3 are documented here. Format follows Keep a Changelog loosely; versions follow SemVer once 1.0.0 ships.

0.7.0 — 2026-07-08

Added

  • torch.compile support (experimental, opt-in). trainer.config.compile: true (with optional compile_mode) compiles the network — kernel fusion to cut launch overhead, the lever for small nets that under-use a big GPU where bf16 does nothing (launch-bound, not compute-bound). Compiled before the DDP wrapper; ddp.unwrap() and checkpoint save/load now also strip torch.compile's _orig_mod. prefix, so checkpoints stay interchangeable with uncompiled runs. Off by default; verified single-GPU and on 4×H200 DDP (trains + checkpoints round-trip). The actual speedup and any shape-driven recompiles are model-dependent (see guide/ddp.md).

Changed

  • PairTrainer calls the network positionally — net(base, query, True) instead of net(base, query, return_features=True). Pair nets name the third argument differently (return_features, return_intermediates, …); passing it by position keeps the trainer agnostic to the name.

0.6.0 — 2026-07-08

Added

  • Trainer.compute_losses(source, labels, net=None) — an extension seam for the forward pass and loss routing. It runs inside the trainer's autocast and returns (predictions, {name: (loss, weight)}); both training and validation call it, so a subclass customizes what the network is fed and how losses map to its outputs without ever re-implementing the precision / scaler / closure / DDP machinery, which stays in the base. Behaviour is unchanged — the default compute_losses is the previous single-input forward.
  • echelon3.trainers.pair.PairTrainer — a trainer for two-image ("pair" / image-in-image) inputs. Consumes ((base, query), gt) batches (via pair_collate_fn), calls net(base, query, return_features) and delegates loss routing to pair_losses(heatmap, features, labels) (default: every loss on the heatmap). A domain-specific image-in-image trainer becomes a natural subclass overriding only pair_losses — see guide/extending.md.

0.5.2 — 2026-07-08

Fixed

  • GrayscaleCLAHE is now picklable, so it survives DataLoader workers started with the spawn start method (and spawn-only platforms). It stored a bare cv2.CLAHE C++ handle, which cannot be pickled — any run whose data workers pickle the dataset died with TypeError: cannot pickle 'cv2.CLAHE' object. The handle is now dropped on pickle and rebuilt from the stored clip / grid params on unpickle. Under the default fork start method the transform is inherited and never pickled, so single-GPU runs and the example smokes never triggered it — reproduced and fixed under a spawn DataLoader on 4×H200.

0.5.1 — 2026-07-08

Fixed

  • README (the PyPI project description) rewritten for 0.5.0. It still documented the removed DataParallel fallback and presented torchrun as the only multi-GPU path. Now it shows the built-in gpus=[...] launcher (no torchrun needed), the bf16-by-default mixed precision, the full CLI set, and all three example smokes. Docs-only release — no code changes.

0.5.0 — 2026-07-08

Multi-GPU and performance release. Breaking: DataParallel is gone and mixed precision defaults to bf16.

Changed

  • Multi-GPU is built-in DDP, launched from the CLI — no torchrun needed. Pass gpus=[0,1,2,3] (a root config key; default = all visible GPUs on the node) and echelon3 spawns one DDP worker per GPU via PyTorch's elastic_launch, wiring up RANK/LOCAL_RANK/WORLD_SIZE/MASTER_* itself. torchrun and multi-node runs still work unchanged through the environment-variable path. Applies to echelon3-train and echelon3-finetune.
  • Mixed precision (AMP) on by default. Training, evaluate and run autocast in bf16 on capable GPUs (fp32 on CPU / unsupported GPUs). Set trainer.config.precision: fp32 to restore full fp32. precision: fp16 uses a GradScaler; with closure optimizers (SAM/LBFGS, which double-backward) it falls back to bf16. TF32 matmul and cudnn.benchmark are on by default (trainer.config.tf32, trainer.config.cudnn_benchmark).
  • Checkpoints save the unwrapped state_dict (no module. prefix); older module.-prefixed checkpoints still load (prefix stripped automatically).

Removed

  • DataParallel. Multiple GPUs always run as DDP; a single process drives one GPU. device_ids no longer selects multiple GPUs — use gpus. The run / evaluate / export CLIs load checkpoints directly instead of wrapping the network in DataParallel.

0.4.1 — 2026-07-07

Fixed

  • IoU.compute (segmentation metric) returned nan whenever ignore_index was unset (the default): use_idx[None] = False added an axis and wiped the whole keep-mask, so nanmean ran over an empty array. Now the mask is only touched when there are classes to ignore. Verified end to end — a Segmenter smoke on synthetic masks moves mIoU from ~0.85 to ~0.96.

Added

  • examples/segmentation/: a self-contained semantic-segmentation smoke (synthetic image/mask generator, a tiny dependency-free backbone, a Segmenter
  • cross-entropy + IoU config) mirroring examples/detector/.

0.4.0 — 2026-07-07

Fixed

  • Detection pipeline now trains end to end (CenterNet-style HeatmapDetector, YOLO-format DetectionDataset). Several defects along the path were fixed:
  • Trainer.set_to_device now stacks the per-sample image tensors that VariableDataLoader produces into one (N, C, H, W) batch, and leaves the variable-length box lists as Python lists instead of calling .to() on them (image-in-image (base, query) pairs are unaffected).
  • The train/validate steps no longer assume labels is a tensor (they read labels.shape only when it is), so list-valued detection targets work.
  • metrics.base.Metric gained a no-op .to() so custom metrics (mAP, EER, AUC, IoU, …) interoperate with the trainer's uniform metric.to(device).
  • HeatmapBasedDetectionLoss uses the penalty-reduced CornerNet/CenterNet focal loss on the sigmoid heatmaps the head emits, instead of sigmoid_focal_loss (which double-applied a sigmoid and crushed the gradient, so heatmap peaks never formed).
  • DecodeHeatmaps.decode keeps YOLO/Albumentations boxes as normalized floats (they were cast to LongTensor, truncating every coordinate to 0).
  • Class labels are coerced to integer tensors in both the encoder and the mAP metric (Albumentations round-trips them through float arrays).

Added

  • detection extra (faster-coco-eval) — the mAP metric needs a COCO backend; mAP takes an optional backend argument (default faster_coco_eval).
  • examples/detector/: a self-contained CenterNet-like detector smoke — synthetic YOLO dataset generator, a tiny dependency-free heatmap backbone, and a config that trains, validates (mAP rises above zero) and checkpoints on CPU or GPU.

0.3.1 — 2026-07-07

Added

  • Full MkDocs (Material) documentation site published to GitHub Pages at https://veryviolet.github.io/echelon3/: getting-started, concepts (how-it-works, run anatomy), guides (extending, DDP, ONNX export) and reference (config schema, built-in components, CLI). docs extra + a docs workflow that mkdocs gh-deploys on every v* tag.
  • [project.urls] with a Documentation link, so PyPI shows it.

0.3.0 — 2026-07-05

Added

  • echelon3-finetune CLI: warm-start from a checkpoint (init_from), freezing by regex patterns (finetune.freeze_patterns), head-only training and per-layer parameter groups (finetune.param_groups). With none of those blocks present it behaves exactly like echelon3-train.

0.2.0 — 2026-07-05

Added

  • ONNX export: ModelExporter base + OnnxExporter (preprocess → net → postprocess wrapped into a single graph), echelon3-export CLI, export section in the smoke example, CI step exporting the smoke model and verifying it with onnxruntime.
  • All CLIs insert the current working directory into sys.path, so zoo repositories can reference their local packages from configs (module: my_zoo.nets.foo) when running from the repo root.

Fixed

  • create_exporters instantiated torch.nn.Identity incorrectly when no preprocess/postprocess is configured.

0.1.0 — 2026-07-05

First public release. Core of the framework extracted from the internal echelon2 codebase and cleaned up:

Added

  • Config-driven component factory (echelon3.creator): every component is a module / type / config YAML triple resolved by dynamic import, with a file-path fallback for project-local extensions.
  • Trainers: baseline Trainer (DataParallel + DDP via torchrun, global batch size semantics, multi-metric keep-best checkpointing, multiple named test loaders), MultiHeadTrainer.
  • Generic datasets (folder-hive / CSV classification, segmentation pairs, detection, multi-head binary masks), balanced / classwise / multipart dataloaders, albumentations augment + torch preprocess pipeline.
  • Generic losses and metrics for classification, segmentation (OHEM, boundary, clDice, Lovasz, multi-head BCE/IoU) and heatmap detection.
  • Nets: ClassifierNet / Segmenter composition wrappers, timm backbone adapter, DDRNet, SegFormer-style heads, DASPP/PSP/FPN necks, CDC layers.
  • Checkpoint manager, tensorboard mlops logger, weight loaders (full/partial), CLI entry points echelon3-train / echelon3-evaluate / echelon3-run (Hydra, --config-dir).
  • Smoke example: synthetic dataset generator + minimal classifier config.

Changed vs internal predecessor

  • All hardcoded 'cuda' calls removed — CPU training works.
  • Vendored copies of timm and mmsegmentation dropped; external timm is used, the only needed mmseg op (resize) lives in echelon3.nets.ops.
  • Heavy/optional dependencies (mosaicml SAM, segmentation-models-pytorch) import lazily and install via extras (echelon3[sam], echelon3[smp]).