Building a Claim Confidence Scoring Model
A vendor rebate claim reaches the validation layer already carrying a verdict on each individual check — the amount reconciles or it does not, the SKU resolves or it does not, the sale falls inside the promotion window or it does not — yet an analyst still has to decide, for the claim as a whole, whether to pay it, look at it, or throw it back. Encoding that decision as one number that a machine can compute the same way every time, and that a human can pick apart afterward, is the job of a claim confidence score. This page builds that score end to end: a transparent, weighted, rule-based number in [0, 1] that maps to an auto-approve / manual-review / reject routing decision. It is the hands-on companion to the scoring and confidence models topic, which owns the broader confidence-modeling design inside claim validation rule engine configuration; here we focus only on constructing, calibrating, and logging the score itself. The score stays a weighted sum of explainable signals — never an opaque model — because trade finance has to defend every auto-approved dollar to an auditor.
Prerequisites
Before building the score, confirm the following are in place:
- Validated claim facts. The score consumes the per-check results, not raw submissions. Each claim must already carry a resolved SKU, a matched agreement version, the claimed amount, and the modeled accrual it is measured against. Amount modeling itself belongs to payout structure modeling; the score only reads the resulting figures.
- Feature signals defined and bounded. Every input to the score must be normalizable to a
Decimalin[0, 1], where1means “maximally confident.” Signals that are natively risks (duplicate risk) are inverted before entering the sum. - A written threshold policy. The reject and approve cut-points are a business decision owned by trade finance, versioned like any other agreement term. The model reads them; it does not choose them silently.
- A labeled corpus. Calibration needs a set of historical claims with a known correct disposition (
approve,review,reject) so thresholds can be measured rather than guessed. - Python 3.11+ with
pydantic>=2.6and the standard-librarydecimalmodule. Every monetary quantity — claimed amount, modeled accrual, variance — usesdecimal.Decimal, neverfloat. - Access roles. The
reconciliation_etlservice role computes and logs scores; only trade finance roles edit the weight and threshold policy; nobody can auto-approve a claim whose score log row is absent.
Step-by-Step Implementation
Step 1 — Declare weighted feature signals as versioned policy
The score is sum(weight_i * signal_i). Its transparency depends entirely on the weights living in a declared, versioned policy rather than being scattered as magic numbers through the scoring code. Store the weights as decimal strings and require that they sum to exactly 1.000 so the raw weighted sum is already normalized into [0, 1].
{
"policy_id": "conf-score-policy",
"policy_version": "3.1.0",
"weights": {
"match_completeness": "0.30",
"amount_variance": "0.25",
"duplicate_risk": "0.20",
"date_window_alignment": "0.15",
"sku_resolution_confidence": "0.10"
},
"thresholds": {"reject_below": "0.40", "approve_at": "0.75"}
}from decimal import Decimal
from pydantic import BaseModel, field_validator, model_validator
class ScorePolicy(BaseModel):
policy_id: str
policy_version: str
weights: dict[str, Decimal]
thresholds: dict[str, Decimal]
@field_validator("weights", "thresholds", mode="before")
@classmethod
def _reject_float(cls, v):
if any(isinstance(x, float) for x in v.values()):
raise ValueError("weights and thresholds must be strings, not floats")
return v
@model_validator(mode="after")
def _weights_sum_to_one(self) -> "ScorePolicy":
total = sum(self.weights.values())
if total != Decimal("1.000"):
raise ValueError(f"weights must sum to 1.000, got {total}")
return selfValidation check: load a policy whose weights sum to 0.95 and assert it raises ValidationError. A weight vector that does not sum to one silently rescales every score and makes the thresholds meaningless.
Step 2 — Normalize each signal to a bounded Decimal
Each raw check result becomes a Decimal in [0, 1]. Two rules keep this deterministic and auditable: the money-based amount_variance signal is computed with Decimal arithmetic against an explicit tolerance, and duplicate_risk — natively a risk where high is bad — is inverted to 1 - risk so that every signal points the same way (higher is more confident). Clamp each signal to the range so a stray out-of-band input can never distort the sum.
from decimal import Decimal, ROUND_HALF_UP
ZERO, ONE = Decimal("0"), Decimal("1")
def _clamp(x: Decimal) -> Decimal:
return max(ZERO, min(ONE, x))
def variance_signal(claimed: Decimal, modeled: Decimal,
tolerance: Decimal) -> Decimal:
"""1.0 when claimed equals modeled accrual; decays to 0 at tolerance."""
if tolerance <= ZERO:
raise ValueError("tolerance must be positive")
deviation = abs(claimed - modeled)
return _clamp(ONE - (deviation / tolerance)).quantize(Decimal("0.0001"))
def build_signals(facts: dict) -> dict[str, Decimal]:
return {
"match_completeness": _clamp(Decimal(facts["fields_matched"])
/ Decimal(facts["fields_required"])),
"amount_variance": variance_signal(
Decimal(facts["claimed_amount"]),
Decimal(facts["modeled_accrual"]),
Decimal(facts["variance_tolerance"])),
"duplicate_risk": _clamp(ONE - Decimal(facts["duplicate_risk"])),
"date_window_alignment": _clamp(Decimal(facts["units_in_window"])
/ Decimal(facts["units_total"])),
"sku_resolution_confidence": _clamp(Decimal(facts["sku_confidence"])),
}Validation check: call variance_signal(Decimal("100"), Decimal("100"), Decimal("20")) and assert it returns Decimal("1.0000"); call it with a deviation equal to the tolerance and assert it returns Decimal("0.0000"). A zero variance must earn full confidence, and a claim off by the full tolerance must earn none. The amount_variance signal draws its reference figure from the same modeled accrual the payout structure modeling layer produced, and its duplicate_risk input comes from duplicate claim detection.
Step 3 — Compute the score deterministically
Reduce the signals to a single number. Determinism is the whole point: two runs over the same facts and the same policy version must produce byte-identical scores, so iterate the weights in a stable sorted order, do the arithmetic in Decimal, and quantize the result to a fixed precision with an explicit rounding mode. Return the per-feature contributions alongside the score — they are what make the number explainable.
def score_claim(signals: dict[str, Decimal],
policy: ScorePolicy) -> dict:
contributions = {}
total = Decimal("0")
for feature in sorted(policy.weights): # stable iteration order
weight = policy.weights[feature]
signal = signals[feature]
part = (weight * signal)
contributions[feature] = part.quantize(Decimal("0.0001"),
rounding=ROUND_HALF_UP)
total += part
score = total.quantize(Decimal("0.001"), rounding=ROUND_HALF_UP)
return {"score": score, "contributions": contributions}Validation check: run score_claim twice on identical inputs and assert the two score values are equal and equal to the sum of the reported contributions (within the quantization step). A score that does not reconcile to its own contributions is not explainable, and any run-to-run difference means a non-deterministic input leaked in.
Step 4 — Map the score to a routing decision
Convert the number into an action. The two thresholds carve [0, 1] into three bands: below reject_below the claim is rejected, at or above approve_at it auto-approves, and everything in between routes to a human. Encode the decision as an enum so downstream routing cannot receive a free-text disposition, and validate that the thresholds are ordered.
from enum import Enum
class Decision(str, Enum):
REJECT = "reject"
MANUAL_REVIEW = "manual_review"
AUTO_APPROVE = "auto_approve"
def route(score: Decimal, policy: ScorePolicy) -> Decision:
lo = policy.thresholds["reject_below"]
hi = policy.thresholds["approve_at"]
if not (Decimal("0") <= lo < hi <= Decimal("1")):
raise ValueError("require 0 <= reject_below < approve_at <= 1")
if score < lo:
return Decision.REJECT
if score >= hi:
return Decision.AUTO_APPROVE
return Decision.MANUAL_REVIEWValidation check: assert that a score exactly equal to approve_at returns AUTO_APPROVE (the boundary is inclusive on the approve side) and that a score exactly equal to reject_below returns MANUAL_REVIEW, not REJECT. Off-by-one boundary handling is the most common source of disputes over who should have looked at a claim, so pin the inequalities explicitly. Claims that route to MANUAL_REVIEW on a volume signal often trace back to the same breakpoints validated in volume threshold validation.
Step 5 — Calibrate thresholds against a labeled corpus
Thresholds are not guessed — they are measured against a corpus of claims whose correct disposition is already known. Sweep candidate cut-points, and for each pair report the two numbers finance actually cares about: the auto-approve precision (of the claims the model would auto-approve, what fraction were genuinely correct) and the manual-review load (what fraction of all claims land in the human queue). Pick the tightest thresholds that keep auto-approve precision above the policy floor.
def evaluate(corpus: list[dict], policy: ScorePolicy,
lo: Decimal, hi: Decimal) -> dict:
policy.thresholds = {"reject_below": lo, "approve_at": hi}
approved_ok = approved_all = reviewed = 0
for row in corpus:
decision = route(score_claim(row["signals"], policy)["score"], policy)
if decision is Decision.AUTO_APPROVE:
approved_all += 1
approved_ok += int(row["true_label"] == "approve")
elif decision is Decision.MANUAL_REVIEW:
reviewed += 1
precision = (Decimal(approved_ok) / Decimal(approved_all)
if approved_all else Decimal("0"))
review_load = Decimal(reviewed) / Decimal(len(corpus))
return {"lo": lo, "hi": hi,
"approve_precision": precision.quantize(Decimal("0.001")),
"review_load": review_load.quantize(Decimal("0.001"))}Validation check: raising approve_at must never lower auto-approve precision — assert the swept series is monotonic in that direction. If a higher approve threshold ever admits more wrong approvals, a signal is leaking the label (see failure modes below) and the corpus is not trustworthy.
Step 6 — Log the score and feature contributions for audit
An auto-approved dollar that cannot be explained is a finding waiting to happen. Persist one immutable record per scored claim carrying the policy version, every signal value, every feature contribution, the final score, the decision, and a stable hash over the whole thing. This row is what an auditor replays to confirm the routing was correct and reproducible.
import hashlib, json
from datetime import datetime, timezone
def log_score(claim_id: str, signals: dict[str, Decimal],
scored: dict, decision: Decision,
policy: ScorePolicy, registry) -> dict:
record = {
"claim_id": claim_id,
"policy_version": policy.policy_version,
"signals": {k: str(v) for k, v in sorted(signals.items())},
"contributions": {k: str(v) for k, v in sorted(scored["contributions"].items())},
"score": str(scored["score"]),
"decision": decision.value,
"ts": datetime.now(timezone.utc).isoformat(),
}
blob = json.dumps(record, sort_keys=True)
record["score_hash"] = hashlib.sha256(blob.encode()).hexdigest()
registry.write_score_log(record)
return recordValidation check: re-run log_score with identical inputs (holding ts fixed) and assert the score_hash is stable, and assert every contributions value round-trips back to a Decimal that sums to the logged score. A record whose contributions do not reconstruct its score is not a defensible audit trail.
Common Failure Modes and Fixes
1. Non-deterministic score. Iterating signals or weights in dictionary-insertion order, or folding a wall-clock timestamp or a live registry read into the arithmetic, lets two runs score the same claim differently and destroys reproducibility. Fix: iterate weights in sorted() order (Step 3), snapshot every input, and hash the result (Step 6) so any divergence is caught immediately.
2. Float in the variance feature. Computing abs(claimed - modeled) / tolerance with float reintroduces double-precision drift that pushes borderline claims across a threshold by a fraction of a cent and differs across platforms. Fix: keep claimed amount, modeled accrual, and tolerance as Decimal strings and quantize explicitly, exactly as variance_signal does.
3. Threshold miscalibration. Setting approve_at by intuition rather than measurement either auto-approves bad claims or floods the manual queue. Fix: run the Step 5 sweep against the labeled corpus, pick the thresholds by measured auto-approve precision and review load, and version the chosen cut-points in the policy.
4. Feature leakage. A signal that encodes the final disposition — for example, a “reviewer already blessed this” flag folded into match_completeness — makes calibration look perfect and production behavior collapse. Fix: every feature must derive only from facts known before a human touched the claim; audit the corpus so no signal is a proxy for the label, and treat a suspiciously monotone calibration curve as evidence of leakage.
5. Unexplained decision. Emitting a bare score and a decision, with no record of which signals drove it, leaves an auditor unable to defend an auto-approval and an analyst unable to challenge a rejection. Fix: always persist the per-feature contributions and the policy version (Step 6); the decision must be reconstructable from the logged row alone, without re-running the pipeline.
Operational Checklist
Frequently Asked Questions
Why keep the score a weighted rule-based number instead of a trained model? Trade finance has to defend every auto-approved claim to an auditor, and “the model said so” is not a defensible answer. A weighted sum of bounded signals lets you point at exactly which features earned the confidence and by how much, reproduce the number on demand, and change behavior by editing a versioned policy rather than retraining. The transparency is worth more than any accuracy gain an opaque model might offer on this task.
Why must the feature weights sum to exactly one?
When the weights sum to one and every signal is bounded in [0, 1], the raw weighted sum is already a value in [0, 1] with no separate normalization step to get wrong. It also makes the thresholds interpretable as fractions of total confidence, and it makes each contribution directly comparable. A weight vector that sums to anything else silently rescales the score and detaches it from the thresholds it is compared against.
How are the approve and reject thresholds chosen? They are measured, not guessed. Score a corpus of claims whose correct disposition is already known, sweep candidate cut-points, and for each pair record the auto-approve precision and the resulting manual-review load. Pick the tightest thresholds that hold auto-approve precision above the policy floor, then version those cut-points so a later change is auditable against the calibration that justified it.
What has to be logged for the decision to be auditable? One immutable row per claim carrying the policy version, every normalized signal, every per-feature contribution, the final score, the routing decision, and a stable hash over the record. The test is that an auditor can reconstruct the score from the logged contributions alone and confirm it maps to the logged decision under that policy version — without re-running the pipeline.
Related
- Parent topic: Scoring & Confidence Models — the broader confidence-modeling design this scoring method implements.
- Detecting Duplicate Deduction Claims — the source of the
duplicate_risksignal that feeds this score inverted. - Setting Dynamic Volume Thresholds for Rebates — the volume breakpoints that inform several confidence signals at claim time.
- Payout Structure Modeling — produces the modeled accrual the
amount_variancefeature measures each claim against.
Up one level: Scoring & Confidence Models