Claim Validation & Rule Engine Configuration

In vendor rebate and trade promotion reconciliation, the claim validation layer is the financial control plane: every claim a vendor submits is a contractual assertion that money is owed, and the rule engine is the only thing standing between that assertion and a posted accrual. When validation is ad hoc — spreadsheet lookups, tribal knowledge, a senior analyst eyeballing exceptions — the failure modes are predictable and expensive. Duplicate payouts slip through on remapped SKUs, claims reference sales dated outside the promotion window, tier breakpoints get miscalculated against float arithmetic, and none of it survives an audit because there is no deterministic record of why a claim passed.

This page describes how to build the engine that prevents all of that: a configuration-driven, version-controlled, fully auditable validation system that turns claims into accruals without manual triage. It is the parent topic for four implementation areas — SKU resolution, temporal alignment, volume thresholds, and confidence scoring — and it sits directly downstream of the data ingestion normalization pipelines that feed it clean, typed records, and downstream of the agreement registry built in core architecture and promotion mapping. The audience is concrete: Python ETL developers who own the pipeline, trade finance analysts who defend the accrual, and vendor managers who answer for the dispute.

Canonical Data Model

A validation engine is only as deterministic as the entities it reasons over. Before any rule fires, four record types must be resolvable to stable, hashable identifiers: the agreement (the contracted terms), the claim (the vendor’s assertion of amount due), the sales fact (the source-of-truth transactions the claim is measured against), and the validation result (the immutable outcome the engine emits). The relationships between them are what the rule graph traverses, so they have to be explicit rather than inferred at query time.

The agreement is the anchor. It carries a version hash, an effective interval, a scope (channel, territory, product hierarchy), and an ordered set of rebate tiers. A claim references exactly one agreement version — never the agreement in the abstract — so that a contract amended mid-cycle does not silently re-validate historical claims under new terms. Each claim decomposes into claim lines at the SKU-and-period grain, and each line is the unit the engine actually evaluates. The sales fact table is the measurement substrate: POS scans, distributor sell-through, and EDI 844/810 movement, normalized to a single unit-of-measure and currency upstream so the engine never performs conversion inside a rule.

The schema fields below are the minimum contract the engine depends on. Treat them as a versioned interface: adding a field is backward-compatible, renaming or retyping one is a breaking change that bumps the schema version and forces a re-hash.

Modeling the result as a first-class entity — not a column mutated in place — is what makes the system auditable. A ValidationResult is written once, carries the full ordered rule_trace that produced it, and is never updated; a re-run produces a new result with a new evaluation timestamp, so the history of how a claim was adjudicated is reconstructable forever.

Pipeline Design Principles

The engine runs as a stateless evaluation stage between ingestion and settlement, and four properties make its output trustworthy: idempotency, deterministic hashing, schema versioning, and temporal anchoring.

Idempotency. Re-running validation over the same inputs must produce byte-identical results. That means no rule may read wall-clock time, generate a random tie-breaker, or depend on row ordering. In practice the evaluation function is pure: (claim_line, agreement_version, sales_slice, config_version) -> ValidationResult. Upserts into the result store key on the tuple (claim_line_id, config_version) so a replayed batch overwrites cleanly instead of duplicating.

Deterministic hashing. Every input that can change an outcome is folded into a content hash. The agreement carries a scope_hash; the rule configuration carries a config_version that is itself a SHA-256 over the canonicalized rule manifest. Persisting these alongside the result lets an auditor prove that a given accrual was produced by a specific contract version and a specific rule set, with no ambiguity.

import hashlib
import json
from decimal import Decimal

def config_hash(manifest: dict) -> str:
    """Stable hash over a rule manifest. Sorted keys + Decimal-safe encoding
    guarantee the same bytes on every environment."""
    canonical = json.dumps(
        manifest,
        sort_keys=True,
        separators=(",", ":"),
        default=lambda o: f"D:{o}" if isinstance(o, Decimal) else o,
    )
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

Schema versioning. Rule definitions and entity schemas evolve. Both are stored as version-controlled manifests in Git, promoted through environments (dev → staging → prod) via pull request, and validated against a Pydantic v2 model on load so a malformed config fails fast at deploy time rather than mid-batch. Validating the manifest itself is the cheapest gate in the system:

from decimal import Decimal
from pydantic import BaseModel, Field, model_validator

class Tier(BaseModel):
    lower_bound: Decimal = Field(ge=0)
    upper_bound: Decimal | None = None      # None == open-ended top tier
    rate: Decimal = Field(ge=0, le=1)       # fraction, e.g. 0.0250 == 2.5%

class Agreement(BaseModel):
    agreement_id: str
    agreement_version: int = Field(ge=1)
    tiers: list[Tier]

    @model_validator(mode="after")
    def tiers_must_be_ordered_and_contiguous(self) -> "Agreement":
        bounds = sorted(self.tiers, key=lambda t: t.lower_bound)
        for prev, nxt in zip(bounds, bounds[1:]):
            if prev.upper_bound is None or prev.upper_bound != nxt.lower_bound:
                raise ValueError("tiers must be contiguous and non-overlapping")
        return self

Temporal anchoring. Every date the engine compares is anchored to UTC and floored to the day at ingestion, so the rule layer never reasons about timezones. This is the contract that the data ingestion normalization pipelines are responsible for upholding; the validation engine treats any non-anchored timestamp as a hard reject rather than guessing.

Rule Evaluation Architecture

Claims are evaluated as a directed acyclic graph of rules, not a linear checklist. Each node is a pure predicate that returns pass, fail, or flag plus structured metadata; edges encode precedence and short-circuit conditions. A DAG — rather than nested if blocks — is what lets analysts reason about evaluation order, prove there are no circular dependencies, and add a rule without re-deriving the whole flow. This is the same evaluation discipline the eligibility rule framework applies at the agreement-mapping layer; the claim engine inherits its precedence model and extends it with claim-specific predicates.

The graph runs in four phases, each a gate on the next:

1. Structural and identity validation. Schema conformance, vendor-ID resolution, and agreement-version reference. A failure here is terminal — there is nothing to measure against an unresolved claim. Identity resolution leans on robust SKU mapping and deduplication so that a remapped GTIN or a promotional pack variant lands on the correct internal hierarchy node instead of generating a phantom duplicate.
2. Dimensional alignment. Channel, territory, and most critically the promotion window. Temporal misalignment is the single largest driver of overpayment, which is why date window alignment checks run here against the UTC-anchored sales facts, applying any contracted grace period explicitly rather than implicitly.
3. Quantitative verification. Volumes, rates, and tier breakpoints measured against source-of-truth sales. Volume threshold validation cross-references claimed quantities against POS and sell-through, enforces accrual ceilings, and confirms that a claimed tier was actually earned before any liability is recognized.
4. Decision routing. The terminal phase aggregates every upstream outcome into a route. Auto-approve, exception queue, or manual review is decided by scoring and confidence models that combine rule outcomes, vendor history, and data completeness into a single risk score.

Precedence is explicit and total: a fail in an earlier phase short-circuits to routing and is never silently overridden by a later pass; a flag does not stop evaluation but is carried forward and depresses the confidence score. Encoding this as a DAG makes the order a property of the configuration, not of the code, so changing precedence is a reviewed manifest edit rather than a refactor.

from enum import Enum

class Outcome(str, Enum):
    PASS = "pass"
    FAIL = "fail"
    FLAG = "flag"

def evaluate(line, agreement, sales_slice, rules) -> list[tuple[str, Outcome]]:
    trace: list[tuple[str, Outcome]] = []
    for rule in rules:                     # rules are a topologically sorted DAG
        outcome = rule.apply(line, agreement, sales_slice)
        trace.append((rule.id, outcome))
        if outcome is Outcome.FAIL and rule.short_circuit:
            break                          # terminal; route straight to exceptions
    return trace

Financial Modeling & Accrual Generation

Once a claim line clears verification, the engine computes the accrual — and this is where correctness is non-negotiable. Trade agreements almost never pay a flat rate; they pay across incremental tiers, where each band of volume earns its own rate, or retroactive tiers, where crossing a breakpoint re-rates the entire volume. Conflating the two is the most common source of payout disputes in rebate reconciliation, and the difference is purely a matter of which summation the engine applies.

All monetary arithmetic uses decimal.Decimal with an explicit context and rounding mode. Float arithmetic introduces drift that is invisible on a single line and material across a quarter-end accrual run; IFRS 15 and ASC 606 revenue-recognition treatment depends on the number being reproducible to the cent, so float is simply not permitted anywhere in the settlement path. This is the same decimal discipline the payout structure modeling cluster establishes for the agreement side.

For an incremental schedule the rebate is the sum, over each tier, of the volume falling inside that band multiplied by the band rate:

from decimal import Decimal, ROUND_HALF_UP, getcontext

getcontext().prec = 28

def incremental_rebate(volume: Decimal, tiers: list["Tier"]) -> Decimal:
    """Each band earns its own rate; only the marginal volume in a band is re-rated."""
    total = Decimal("0")
    for t in tiers:
        upper = t.upper_bound if t.upper_bound is not None else volume
        band = max(Decimal("0"), min(volume, upper) - t.lower_bound)
        total += band * t.rate
    return total.quantize(Decimal("0.0001"), rounding=ROUND_HALF_UP)

def retroactive_rebate(volume: Decimal, tiers: list["Tier"]) -> Decimal:
    """Crossing a breakpoint re-rates the ENTIRE volume at the highest earned rate."""
    earned = next(
        t.rate for t in reversed(tiers)
        if volume >= t.lower_bound
    )
    return (volume * earned).quantize(Decimal("0.0001"), rounding=ROUND_HALF_UP)

The accrual the engine posts is the validated rebate, not the claimed amount. When the two disagree, the delta is the dispute, and it is routed — never written off and never auto-paid. The result record carries both figures plus the tier trace so a finance analyst can reconstruct the calculation line-by-line during month-end close, where the order of operations (accrual freeze → retroactive re-rate → audit export) is itself a controlled sequence.

Exception Taxonomy & Routing

Production data is never clean, and a validation engine that halts on the first imperfect record is unusable. Exceptions are therefore a first-class output, classified into a stable taxonomy so that routing, SLA, and root-cause analysis can be driven by code rather than by an analyst’s judgment. Each unmatched or flagged record exits with an explicit mismatch code and a priority, and the routing layer maps that code to a queue and a deadline.

When a primary path fails on missing reference data or an upstream timeout, the engine degrades rather than aborts: a fallback chain evaluates against cached master data or a simplified rule subset, tags the record for post-processing reconciliation, and keeps the batch moving so operational SLAs hold during partial degradation. The escalation and adjudication mechanics of those chains are owned by the fallback routing logic cluster; the claim engine’s job is to emit the correct code and never to silently approve a misaligned claim. Failed claims land in a quarantine table with their mismatch code intact, generating an audit-ready dispute packet — claimed vs. validated amount, the rule trace, and the contract version — that compresses vendor cycle time without sacrificing control.

Governance, Audit Trail & Access Control

The validation engine is a system of financial record, so its governance posture has to satisfy SOX and survive external audit. Three controls are load-bearing.

Immutable, signed audit logs. Every rule evaluation writes an append-only log entry: input hashes, config_version, agreement_version, the ordered outcome, and the resulting route. Entries are never updated. To make tampering detectable, each ValidationResult is cryptographically signed and the rule_trace is hash-chained, so any retroactive edit breaks the chain and surfaces in reconciliation.

import hmac, hashlib

def sign_result(result_bytes: bytes, key: bytes) -> str:
    return hmac.new(key, result_bytes, hashlib.sha256).hexdigest()

Role-based access control. Configuration and data are segregated by least privilege. Vendor-facing portals expose only contracted terms and settlement statements; ETL developers can deploy rule manifests through the promotion pipeline but cannot mutate posted results; trade finance analysts can adjudicate exceptions and export audit trails but cannot edit the rule graph. RBAC tags travel with every entity so authorization is enforced at the field level, not just the endpoint.

Configuration as code with deterministic testing. Rule definitions live in Git with pull-request approval, automated linting, and environment promotion. Parameterization beats branching: thresholds and experimental scoring models toggle via config and feature flags, never via deployed code paths. A synthetic claim corpus covering the hard cases — leap-year promotions, partial shipments, retroactive amendments, mid-cycle tier crossings — runs on every change so a manifest edit cannot regress a known boundary. The combination turns claim validation from a reactive bottleneck into a proactive control: lower margin leakage, faster vendor payouts, and a reconciliation system that adapts as trade-promotion strategy evolves.

Frequently Asked Questions

What happens when a retroactive tier is crossed mid-cycle? The engine recomputes the accrual under retroactive_rebate, which re-rates the entire cumulative volume at the newly earned rate, and posts the delta between the prior accrual and the re-rated figure rather than a fresh full accrual. The original ValidationResult is preserved; a new result with a new timestamp records the re-rate so the audit trail shows both the pre- and post-crossing state.

Why is decimal.Decimal mandatory instead of float? Float arithmetic accumulates rounding drift that is immaterial per line but material across a quarter-end run, and it is not reproducible across platforms. Because IFRS 15 / ASC 606 treatment requires accruals reproducible to the cent and the result hash must be identical on every environment, the settlement path uses Decimal with a fixed context and explicit ROUND_HALF_UP.

How do I change a validation rule without redeploying the pipeline? Rules are version-controlled manifests, not code. Edit the manifest, open a pull request, let the Pydantic schema check and synthetic-corpus tests run, and promote dev → staging → prod. The engine loads the new config_version, and because results key on that version, replays are unambiguous and old accruals remain attributable to the rule set that produced them.

What stops a remapped SKU from generating a duplicate payout? Identity validation runs first and resolves every GTIN and legacy SKU to a canonical internal hierarchy node before any amount is measured. Deduplication keys on the resolved node plus period, so a vendor submitting the same movement under two GTINs collapses to one claim line instead of two accruals.

Related

• SKU Mapping & Deduplication — resolve GTINs, legacy SKUs, and pack variants to a single internal hierarchy.
• Date Window Alignment Checks — match claims to eligible sales dates under UTC anchoring and grace periods.
• Volume Threshold Validation — verify claimed quantities and tier breakpoints against source-of-truth sales.
• Scoring & Confidence Models — route claims to auto-approval or manual review by risk score.
• Core Architecture & Promotion Mapping — the agreement registry and rule framework this engine validates against.
• Data Ingestion & Normalization Pipelines — the upstream stage that delivers typed, anchored records to the engine.

Up one level: Home