Versioning Rebate Agreement Schemas

A rebate agreement schema is never finished — vendors add channel scopes, finance introduces new tier semantics, and a compliance change forces a field to become mandatory that used to be optional. Every one of those edits is a chance to silently break replay: an accrual posted last quarter under the old shape must still recompute to the exact same cent this quarter, or the audit trail no longer defends the number on the ledger. This page documents the exact procedure for evolving the schema without that regression — classifying each change as additive or breaking, bumping schema_version and recomputing a deterministic version hash, shipping a dual-read window so both shapes coexist, and validating historical records under the pinned version they were written against. It is the implementation-level companion to agreement schema design, which owns the entity topology and the version registry this procedure writes into.

Prerequisites

Before evolving a live schema, confirm the following are in place:

• Git-versioned schema manifests. The canonical schema lives as a serialized artifact under source control — one file per schema_version — so any historical shape is retrievable by tag, not reconstructed from memory. The agreement schema design stage owns the entity topology these manifests describe; versioning governs how that topology changes over time.
• Pydantic v2 models. All validation snippets use Pydantic v2 (pydantic>=2.6). Each schema_version maps to a concrete model class so a record can be re-parsed under the exact shape it was written against.
• A migration harness. A registered, ordered set of migrate(vN) -> vN+1 transforms that upgrade a record from any prior version to the current one — reversible where the change is additive, forward-only where it is breaking.
• decimal.Decimal discipline. Every rate, threshold, and accrual amount is parsed via decimal.Decimal, never a JSON float. A retype that touches a monetary field is always breaking, and float drift would make the version hash non-reproducible across platforms.
• Roles. Write access to the schema registry and migration harness is scoped to a schema_owner role; the reconciliation_etl service role reads pinned versions but cannot mutate a published manifest. Posted accruals are immutable to both.

Step-by-Step Implementation

Step 1 — Classify the change as additive or breaking

Every schema edit is one of two kinds, and the classification decides the version bump, the migration cost, and whether old readers survive. An additive change never invalidates a record that parsed under the previous shape: adding an optional field, adding a member to an enum, or introducing a nullable field with a default. A breaking change invalidates at least one previously valid record: retyping a field, dropping or renaming one, making an optional field required, or tightening a constraint (a narrower enum, a stricter bound). When in doubt, classify as breaking — the cost of an unnecessary major bump is a migration you did not need; the cost of a missed one is a silently corrupted replay.

from enum import Enum

class ChangeKind(str, Enum):
    additive = "additive"
    breaking = "breaking"

def classify_change(old_fields: dict, new_fields: dict) -> ChangeKind:
    """Diff two field->spec maps; any removal, retype, or new-required is breaking."""
    for name, spec in old_fields.items():
        if name not in new_fields:
            return ChangeKind.breaking            # dropped / renamed
        if new_fields[name]["type"] != spec["type"]:
            return ChangeKind.breaking            # retype
    for name, spec in new_fields.items():
        if name not in old_fields and spec.get("required"):
            return ChangeKind.breaking            # new required field
    return ChangeKind.additive

Validation check: assert that adding {"channel_group": {"type": "str", "required": False}} returns additive, while retyping rate from str to float, or adding a required settlement_terms, returns breaking. A classifier that ever returns additive for a removal or retype is the root cause of broken replay.

Step 2 — Bump schema_version and recompute a deterministic version hash

Semantic versioning encodes the classification: an additive change bumps the minor component (2.5.0 → 2.6.0), a breaking change bumps the major and resets minor (2.6.0 → 3.0.0). The schema_version string is human-facing; the load-bearing identifier is a version hash — a SHA-256 over the canonically serialized schema. Canonical means sorted keys, no insignificant whitespace, and a fixed field order, so the same schema always hashes to the same digest on every machine. Pin this hash onto every record the version writes; it is what a claim references three stages downstream to prove which terms were in force.

import hashlib, json

def canonical_schema(fields: dict) -> str:
    """Stable serialization: sorted keys, no whitespace drift, fixed separators."""
    return json.dumps(fields, sort_keys=True, separators=(",", ":"))

def version_hash(fields: dict) -> str:
    return hashlib.sha256(canonical_schema(fields).encode("utf-8")).hexdigest()

def bump(version: str, kind: ChangeKind) -> str:
    major, minor, patch = (int(p) for p in version.split("."))
    if kind is ChangeKind.breaking:
        return f"{major + 1}.0.0"
    return f"{major}.{minor + 1}.0"

Validation check: compute version_hash(fields) twice from dicts built in different key orders and assert the digests are identical; then assert that any real field change produces a different digest. A version bump without a hash change means the manifest and the identifier have drifted apart.

Step 3 — Ship a dual-read window

A breaking change cannot flip atomically — in-flight records, replays, and lagging producers still emit the old shape. During a dual-read window the engine accepts both the current version N and its predecessor N-1, dispatching each record to the model that matches its pinned schema_version. The window stays open until every producer has cut over and the backfill in Step 4 completes; only then is N-1 retired. This is how a schema evolves without a coordinated stop-the-world deploy.

from pydantic import BaseModel, ValidationError

class AgreementV2(BaseModel):      # schema_version 2.x
    agreement_id: str
    rate: str                      # decimal string
    channel: str

class AgreementV3(BaseModel):      # schema_version 3.x (rate retyped, tier added)
    agreement_id: str
    rate: str
    channel: str
    tier_id: str

READERS = {2: AgreementV2, 3: AgreementV3}

def dual_read(record: dict) -> BaseModel:
    major = int(record["schema_version"].split(".")[0])
    model = READERS.get(major)
    if model is None:
        raise ValueError(f"no reader registered for schema major {major}")
    return model.model_validate(record)

Validation check: feed the reader one record tagged 2.6.0 and one tagged 3.0.0 and assert both parse to their respective models; feed a record tagged with an unregistered major and assert it raises rather than defaulting to the newest reader. Silent coercion to the current shape is the failure this window exists to prevent.

Step 4 — Migrate and validate old records under the pinned version

With both readers live, upgrade historical records through the migration harness — but validate each first against the version it was written under, never the current one. The upgrade path is an ordered chain of migrate(vN) -> vN+1 steps; running it end to end is what lets a v2 record participate in a v3 accrual run while preserving the original for audit. Crucially, the migrated record must recompute to the same accrual it produced before, or the migration itself introduced drift.

from decimal import Decimal

def migrate_v2_to_v3(rec: dict) -> dict:
    out = dict(rec)
    out["tier_id"] = "DEFAULT"          # additive default, no re-rating
    out["schema_version"] = "3.0.0"
    return out

def replay_safe(rec: dict) -> dict:
    written_major = int(rec["schema_version"].split(".")[0])
    READERS[written_major].model_validate(rec)      # validate under the PINNED version
    upgraded = migrate_v2_to_v3(rec) if written_major == 2 else rec
    READERS[3].model_validate(upgraded)             # validate under current version
    return upgraded

def accrual(rec: dict, units: Decimal) -> Decimal:
    return (Decimal(rec["rate"]) * units).quantize(Decimal("0.01"))

Validation check: for a sampled v2 record, assert accrual(rec, units) == accrual(replay_safe(rec), units) to the cent. If the migrated record rates differently, the migration re-mapped a value it should have preserved — stop and treat it as a breaking defect, not a backfill.

Step 5 — Gate the release with a synthetic-corpus test

Before the new version is published, run a frozen synthetic corpus — a fixed set of records spanning every historical version, each paired with its known-correct accrual — through the migration and rating path. The gate passes only when every record still recomputes to its recorded amount and every version hash is stable. Wiring this into CI is what turns “we think replay is safe” into a build that fails loudly when it is not.

CORPUS = [
    {"rec": {"schema_version": "2.6.0", "agreement_id": "A1",
             "rate": "0.035", "channel": "GROCERY"},
     "units": Decimal("10000"), "expected": Decimal("350.00")},
    {"rec": {"schema_version": "3.0.0", "agreement_id": "A2",
             "rate": "0.05", "channel": "DTC", "tier_id": "T2"},
     "units": Decimal("4000"), "expected": Decimal("200.00")},
]

def gate() -> None:
    for case in CORPUS:
        upgraded = replay_safe(case["rec"])
        got = accrual(upgraded, case["units"])
        assert got == case["expected"], (
            f"replay drift for {case['rec']['agreement_id']}: "
            f"{got} != {case['expected']}"
        )
    print("synthetic corpus green: replay is reproducible")

Validation check: run gate() in CI as a required status check. Deliberately introduce a breaking retype without a major bump and confirm the gate fails; this proves the corpus actually guards the invariant rather than passing vacuously.

Common Failure Modes and Fixes

1. A silent retype breaks replay. Changing rate from a decimal string to a JSON number, or an int to a Decimal, is classified as additive by a lazy diff and shipped as a minor bump. Old records then re-rate under new parsing and the accrual moves. Fix: treat any type change as breaking in Step 1, and let the Step 5 corpus fail the build when a re-rated amount diverges from its recorded value.

2. Missing version-hash bump. Editing the manifest but reusing the previous digest — or bumping schema_version without recomputing the hash — leaves records pointing at terms that no longer match. Fix: derive the hash from the canonical schema on every publish (Step 2) and assert it changed whenever any field spec changed; never hand-edit the digest.

3. Re-mapping historical rows under new terms. Running the current model over old records “to clean them up” rewrites the version they were written with and re-rates closed periods. Fix: validate every record under its pinned version first (Step 4), keep the original immutable, and only ever append a migrated copy — never overwrite in place.

4. Non-deterministic hash. Serializing the schema with unsorted keys, locale-dependent formatting, or a float field makes the digest differ across machines, so the same schema fails its own hash check in CI. Fix: canonicalize with sort_keys=True and fixed separators (Step 2), and keep all monetary fields as Decimal strings so serialization is byte-stable.

5. Dropped nullable field. Removing a field that “was always null anyway” is treated as harmless, but any record that carried a value loses it and any reader that required it breaks. Fix: classify a drop as breaking regardless of observed population, keep the field readable through the dual-read window (Step 3), and only retire it after the corpus confirms no accrual depends on it.

Operational Checklist

• Every schema edit is classified additive or breaking by an automated diff, defaulting to breaking when ambiguous.
• Additive changes bump minor; breaking changes bump major and reset minor.
• schema_version and a SHA-256 version hash over the canonical schema are recomputed and pinned onto every record.
• The version hash is derived from a sorted-key, whitespace-stable serialization and is identical across machines.
• Breaking changes ship behind a dual-read window that accepts version N and N-1 until every producer cuts over.
• Historical records validate under the version they were written with before any migration is applied.
• Each migrated record recomputes to a byte-identical accrual; divergence is treated as a breaking defect.
• Original records are immutable; migrations append rather than overwrite, preserving the audit trail.
• A frozen synthetic corpus spanning every historical version runs as a required CI gate before publish.

Frequently Asked Questions

When is a change additive versus breaking? Additive means no previously valid record becomes invalid — a new optional field, a widened enum, a nullable default. Breaking means at least one prior record or reader stops working — a retype, a drop, a rename, a newly required field, or a tightened constraint. A retype of a monetary field is always breaking even when the string value looks the same, because it changes how the value parses and therefore how it rates. When the diff is ambiguous, classify as breaking; an extra major bump is cheap, a missed one corrupts replay.

Why hash the schema instead of trusting the version string? The schema_version string is human-assigned and can drift from the actual manifest — someone bumps the number but forgets the field, or edits the field but reuses the number. A SHA-256 over the canonically serialized schema is derived, not declared, so it changes if and only if the shape changes. Pinning that digest onto each record gives every downstream claim a tamper-evident reference to the exact terms in force, which is what makes an accrual defensible in an audit.

How long should a dual-read window stay open? Until every producer emits the new version and the historical backfill in Step 4 has migrated and re-validated all affected records. There is no fixed duration — the window closes on a condition, not a clock. Retiring version N-1 while a lagging producer still emits it, or before the corpus confirms every prior accrual still reproduces, reintroduces exactly the silent breakage the window exists to prevent.

How do I keep historical accruals reproducible after a breaking change? Never re-rate an old record under new terms. Validate it under the pinned version it was written with, migrate a copy forward through the harness while preserving the original, and assert the migrated record recomputes to the same accrual to the cent. The synthetic-corpus gate makes that assertion a required build check, so any change that would move a historical number fails before it ships.

Related

• Parent cluster: Agreement Schema Design — the entity topology and version registry this procedure writes into.
• How to Map Vendor Rebate Tiers in JSON — encoding the tier array whose retypes and reorderings this versioning discipline governs.
• Core Architecture & Promotion Mapping — the broader reconciliation architecture that consumes each pinned schema version.
• Fallback Routing Logic — where records that fail validation against every registered version are routed for adjudication.

Up one level: Agreement Schema Design