Detecting Duplicate Deduction Claims

A retailer deduction that lands twice is one of the quietest margin leaks in trade reconciliation: the second hit clears remittance the same way the first did, references the same promotion, and looks legitimate until someone reconciles the paid-allowance ledger by hand. The failure is rarely a fraudulent retailer — it is a re-transmitted remittance file, a chargeback re-keyed under a new document number, or an off-invoice allowance that was already paid and is now being deducted again off-invoice. This page documents the exact procedure for catching that case before it posts: build a deterministic fingerprint over the deduction facts, match exact repeats, generate near-duplicate candidates by blocking, confirm the deduction against already-paid allowances, then quarantine the duplicate group behind a single survivor and route the rest to recovery. It is the implementation-level companion to the duplicate claim detection cluster inside Claim Validation & Rule Engine Configuration. Its scope is narrow and deliberate — a deduction taken more than once, not two vendors overlapping on one promotion, which is handled separately in deduplicating overlapping vendor claims.

Prerequisites

Before wiring duplicate-deduction detection into the validation engine, confirm the following are in place:

  • Normalized deduction facts. Each deduction arrives from the data ingestion normalization pipelines reduced to a canonical shape: a resolved vendor_id, a promotion_ref, the source invoice_ref/document number, a fiscal period, and a single-currency amount. Fingerprinting an un-normalized code produces false uniqueness — the same deduction under two spellings looks like two deductions.
  • A paid-allowance ledger. A queryable record of every off-invoice allowance and prior deduction already settled, keyed on (vendor_id, promotion_ref, period). Without it, Step 4 cannot tell a genuine repeat from a first-time claim.
  • Python 3.11+ with pydantic>=2.6. All validation snippets use Pydantic v2 models. Every monetary field is decimal.Decimal, never float.
  • A quarantine queue and recovery hook. A place to hold suspected duplicates and an interface into deduction and short-pay routing so a confirmed duplicate becomes a recovery item rather than a dead flag.
  • Access role. Write access to the deduction_dup_log and read access to the paid-allowance ledger — typically the reconciliation_etl service role. Releasing a held deduction back to payment requires a separate financial-override permission.
Duplicate-deduction detection: fingerprint, exact match, near-duplicate blocking, ledger confirmation, quarantine Normalized deduction facts — vendor, promotion reference, invoice reference, period, and amount — feed a SHA-256 fingerprint. A deduction whose fingerprint already exists is an exact duplicate and is quarantined immediately. A deduction with a new fingerprint is placed in a blocking bucket keyed on vendor, promotion, and period; candidates in the same bucket whose amounts fall within a decimal tolerance and whose periods overlap are near-duplicate candidates. Each candidate is confirmed against the paid-allowance ledger: if a matching allowance was already paid, the deduction is a duplicate. Confirmed duplicates form a group; the earliest-posted member is kept as the survivor and the rest are quarantined and routed to recovery. Everything else clears as a distinct deduction. Normalized deduction vendor · promotion_ref · invoice_ref period · amount (Decimal) SHA-256 fingerprint canonical key → hex digest rounded amount, sorted fields Fingerprint seen before? exact-duplicate index lookup yes — exact dup no Blocking bucket key = vendor + promotion + period Near-dup candidate? |Δamount| ≤ tol · periods overlap Confirm vs paid ledger allowance already settled? yes — confirmed dup no match → clears as distinct ! Quarantine group by fingerprint keep earliest survivor hold the rest Route to recovery deduction / short-pay routing · one log row per held member survivor stays payable

Step-by-Step Implementation

Step 1 — Build a deterministic deduction fingerprint

The fingerprint is the anchor for everything downstream, so it must be a pure function of the deduction’s identity — the same facts must always yield the same digest on every host, in every run. Canonicalize before hashing: lower-case and trim the vendor and promotion references, quantize the amount to the currency’s minor unit so "100" and "100.00" collapse to one value, and hash a sorted, delimited key with SHA-256.

python
import hashlib
from decimal import Decimal, ROUND_HALF_UP
from pydantic import BaseModel, field_validator

class Deduction(BaseModel):
    deduction_id: str
    vendor_id: str
    promotion_ref: str
    invoice_ref: str
    period: str            # ISO 'YYYY-MM' fiscal period
    amount: Decimal
    posted_at: str         # ISO 8601, for survivor selection

    @field_validator("amount", mode="before")
    @classmethod
    def _reject_float(cls, v):
        if isinstance(v, float):
            raise ValueError("amount must be a string or Decimal, never float")
        return v

def fingerprint(d: Deduction) -> str:
    amt = d.amount.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)
    key = "|".join([
        d.vendor_id.strip().lower(),
        d.promotion_ref.strip().lower(),
        d.invoice_ref.strip().lower(),
        d.period.strip(),
        str(amt),
    ])
    return hashlib.sha256(key.encode("utf-8")).hexdigest()

Validation check: build two Deduction records with the same facts but amount as Decimal("100") and Decimal("100.00"), and one with a trailing-space vendor id; assert all three produce an identical fingerprint. Then change only invoice_ref and assert the digest changes. A fingerprint that varies on whitespace or trailing zeros is too strict and will miss exact repeats.

Step 2 — Detect exact duplicates via the fingerprint

An exact duplicate is a deduction whose fingerprint already exists in the index of accepted deductions. This is a constant-time lookup, and it catches the most common case — a re-transmitted remittance file or a chargeback re-keyed with identical facts. Keep the first fingerprint occurrence as the reference and mark every later collision.

python
class DuplicateIndex:
    def __init__(self):
        self._seen: dict[str, str] = {}   # fingerprint -> first deduction_id

    def check(self, d: Deduction) -> str | None:
        fp = fingerprint(d)
        original = self._seen.get(fp)
        if original is None:
            self._seen[fp] = d.deduction_id
            return None                    # first sighting, accept
        return original                    # exact duplicate of `original`

Validation check: feed the same deduction twice and assert the second call returns the first deduction_id; feed a deduction that differs only in amount by one cent and assert check returns None. The exact path must never merge two deductions that differ in any fingerprinted field — that is what Step 3 is for.

Step 3 — Generate near-duplicate candidates via blocking

A retailer re-billing the same allowance rarely reproduces every field byte-for-byte: the amount drifts by a rounding cent, or the second hit lands in an adjacent period. Comparing every deduction against every other is quadratic and does not scale, so partition the population into blocking buckets keyed on the stable fields — (vendor_id, promotion_ref, period) — and only compare within a bucket. Two deductions in the same bucket are near-duplicate candidates when their amounts fall within a decimal.Decimal tolerance and their periods overlap.

python
from collections import defaultdict

AMOUNT_TOL = Decimal("0.05")   # accept sub-nickel drift as the same deduction

def block_key(d: Deduction) -> tuple[str, str]:
    return (d.vendor_id.strip().lower(), d.promotion_ref.strip().lower())

def near_dup_candidates(rows: list[Deduction]) -> list[tuple[str, str]]:
    buckets: dict[tuple, list[Deduction]] = defaultdict(list)
    for d in rows:
        buckets[block_key(d)].append(d)
    pairs = []
    for members in buckets.values():
        for i, a in enumerate(members):
            for b in members[i + 1:]:
                if abs(a.amount - b.amount) <= AMOUNT_TOL and _periods_overlap(a, b):
                    pairs.append((a.deduction_id, b.deduction_id))
    return pairs

def _periods_overlap(a: Deduction, b: Deduction) -> bool:
    # same period, or adjacent months for a cross-period re-bill
    return abs(_month_index(a.period) - _month_index(b.period)) <= 1

def _month_index(period: str) -> int:
    y, m = period.split("-")
    return int(y) * 12 + int(m)

Validation check: construct two deductions in the same vendor/promotion block, amounts Decimal("500.00") and Decimal("500.03"), in "2026-06" and "2026-07"; assert the pair is returned. Widen one amount past AMOUNT_TOL and assert it drops out. Blocking must not key on amount — if it did, a one-cent drift would land the two deductions in different buckets and the near-duplicate would never be compared.

Step 4 — Confirm candidates against already-paid allowances

A near-duplicate candidate is a suspicion, not a verdict. Confirm it against the paid-allowance ledger: if an off-invoice allowance or prior deduction matching this (vendor_id, promotion_ref, period) was already settled, the current deduction is a genuine duplicate of a paid item and must not pay again. This is the step that catches the subtle case — a deduction that duplicates an allowance already paid off-invoice, where there is no second deduction to fingerprint against, only a prior settlement.

python
def confirm_against_ledger(d: Deduction, ledger) -> bool:
    """True when this deduction duplicates an allowance already settled."""
    paid = ledger.paid_allowances(
        vendor_id=d.vendor_id,
        promotion_ref=d.promotion_ref,
        period_window=(d.period,),
    )
    for entry in paid:
        same_amount = abs(entry.amount - d.amount) <= AMOUNT_TOL
        if same_amount and entry.invoice_ref != d.invoice_ref:
            return True     # already paid under a different document — duplicate
    return False

Validation check: stub the ledger to return one paid allowance with a matching amount under a different invoice_ref and assert confirm_against_ledger returns True; then stub it to return the same invoice_ref and assert False — a deduction referencing the very document that was paid is the settlement of that item, not a second claim on it.

Step 5 — Quarantine the group and route to recovery

Once a set of deductions is confirmed to be the same item, treat them as one group: pick a single survivor and hold the rest. The survivor is the earliest-posted member so the payable that legitimately clears is deterministic and the same on every replay. Every held member writes one immutable deduction_dup_log row and is handed to recovery — a duplicate that is flagged but never routed is a report, not a recovery.

python
import json
from datetime import datetime, timezone

def quarantine_group(group: list[Deduction], reason: str, sink) -> dict:
    survivor = min(group, key=lambda d: d.posted_at)
    held = [d for d in group if d.deduction_id != survivor.deduction_id]
    fp = fingerprint(survivor)
    for d in held:
        record = {
            "fingerprint": fp,
            "duplicate_of": survivor.deduction_id,
            "held_deduction_id": d.deduction_id,
            "vendor_id": d.vendor_id,
            "promotion_ref": d.promotion_ref,
            "period": d.period,
            "amount": str(d.amount),
            "reason": reason,               # 'exact' | 'near_dup' | 'ledger_confirmed'
            "ts": datetime.now(timezone.utc).isoformat(),
        }
        record["decision_hash"] = hashlib.sha256(
            json.dumps(record, sort_keys=True).encode()
        ).hexdigest()
        sink.write_dup_log(record)
        sink.route_to_recovery(d.deduction_id, duplicate_of=survivor.deduction_id)
    return {"survivor": survivor.deduction_id, "held": [d.deduction_id for d in held]}

Validation check: pass a three-member group with distinct posted_at timestamps and assert exactly the earliest survives, two rows land in deduction_dup_log, and route_to_recovery is called twice. Re-run with identical input and assert every decision_hash is byte-stable — a shifting hash means a non-snapshotted value (usually a live timestamp) leaked into the record body.

Common Failure Modes and Fixes

  1. Fingerprint too strict — real repeats slip through. Hashing the raw amount string or an untrimmed vendor id makes "100.00" and "100", or "ACME " and "acme", look distinct, so the second deduction pays. Fix: canonicalize before hashing (Step 1) — lower-case, strip, and quantize the amount to the minor unit so identity is stable across sources.
  2. Fingerprint too loose — distinct deductions collapse. Dropping invoice_ref from the key merges two legitimately different deductions on the same promotion into one, suppressing a valid claim. Fix: keep every identity field in the fingerprint and let Step 3’s tolerance — not a coarse fingerprint — absorb near-matches.
  3. Float amount comparison. Comparing amount as float, or building the tolerance test with 0.05 instead of Decimal("0.05"), reintroduces binary-rounding error that flips a boundary case unpredictably. Fix: keep amounts in decimal.Decimal end to end and reject float at ingestion with the _reject_float validator.
  4. Cross-period duplicate missed. Blocking on an exact period puts a June deduction and its July re-bill in different buckets, so they are never compared. Fix: block on the stable vendor/promotion key and let the _periods_overlap predicate span adjacent months (Step 3), catching the re-bill that crossed a period boundary.
  5. False positive on a legitimate re-bill, and no recovery link. A retailer may legitimately re-issue a deduction after correcting a rejected one — same amount, new document — which is not a duplicate. Auto-holding it without ledger confirmation freezes a valid claim; conversely, flagging a real duplicate but never routing it leaves the margin leak open. Fix: gate the hold on Step 4’s ledger confirmation, and always call route_to_recovery (Step 5) so a confirmed duplicate becomes a recoverable item in deduction and short-pay routing, not a stranded flag.

Operational Checklist

Frequently Asked Questions

Why hash the deduction facts instead of comparing rows directly? A fingerprint turns exact-duplicate detection into a constant-time index lookup, and it gives every downstream record — the quarantine log, the recovery item — one stable key that reconstructs the match point-in-time. Direct row comparison is quadratic and, worse, leaves no reproducible anchor for an auditor to verify why two deductions were treated as the same item.

How is this different from deduplicating overlapping vendor claims? This procedure targets the same deduction taken more than once — a re-transmitted remittance, a re-keyed chargeback, or a deduction duplicating an already-paid allowance — where the fix is to keep one survivor and recover the rest. Overlapping vendor claims are two genuinely different claims that happen to touch the same promotion or SKU; there the resolution is precedence and merge logic, not survivor-and-recover.

How do I keep a legitimate re-bill from being flagged as a duplicate? Do not hold on the near-duplicate match alone. Confirm against the paid-allowance ledger first: only a deduction whose matching allowance was already settled under a different document is a true duplicate. A re-issued deduction that corrects a previously rejected one has no paid counterpart in the ledger, so it clears as a distinct, payable item.

Which member of a duplicate group is kept payable? The earliest-posted member is the survivor, chosen by posted_at so the outcome is deterministic and identical on every replay. The survivor stays payable; every later member is quarantined, logged with a stable decision_hash, and routed to recovery so the second hit is clawed back rather than paid.

Up one level: Duplicate Claim Detection

Back to