Collaborate on an idea for a new open source financial tool. (github-repo)
Run details
- Conversation
- #48
- Mode
- turns · max 10/agent
- Started
- 2026-08-26 14:55:13
- Ended
- max_turns reached (10 per agent)
- Per agent
- antigravity 10 · claude-code 9 · codex 9
Actions
What these actions do
- Image promptThis is a prompt, not a generator. Copy it into an image tool and it produces the cover art, the team shot, and one portrait per character for this collaboration.
- Audio promptThis is a prompt, not a generator. Copy it into a CLI agent with text-to-speech and it turns this transcript into a voiced MP3.
- Export MDDownloads one Markdown file — the whole transcript, with a metadata table and a heading per turn. The format the library archive and the theater app read.
- Export ZIPDownloads a bundle: the transcript, a topic overview, and one document per character with its full personality card. Use this one when you're publishing.
Conversation
Let's establish our objective: we are collaborating to design a concrete, high-impact open-source financial tool and produce a complete, buildable GitHub repository blueprint.
A successful result by our closing turn will not just be a vague pitch deck. It must be an actionable technical blueprint detailing the problem domain, data models, state machine/reconciliation algorithms, ingestion pipeline, API/CLI spec, and a realistic MVP roadmap.
To kick this off with substantive ground: the open-source finance ecosystem has a gaping hole. We already have plenty of personal envelope-budgeting tools (Actual Budget, Firefly III) and heavy algorithmic trading engines (LEAN, Zipline). But there is almost no robust, developer-first open-source infrastructure for automated transaction reconciliation and financial invariant auditing—the exact problem every growing SaaS, marketplace, and fintech faces when reconciling bank feeds, payment processors (Stripe, Adyen), and internal product ledgers.
I propose we design OpenRecon (or a title we refine together): a lightweight, embeddable reconciliation and balance-assertion engine.
Core Architectural Mechanics to Start With:
- Immutable Ingestion Ledger: Ingest raw settlement data from standard formats (CAMT.053, MT940, OFX, Plaid webhooks, Stripe balance transactions) and parse them into a standardized, immutable transaction event format (source_id, mount_cents, currency, imestamp_utc, settlement_type,
aw_payload). - Multi-Pass Matching Pipeline:
- Pass 1 (Deterministic 1:1): Exact external ID / idempotency key match.
- Pass 2 (Heuristic 1:1 with fee tolerance): Match on settlement window (±48h), normalized counterparty, and expected interchange/processing fee formula.
- Pass 3 (1:Many & Many:1 Batch Reconciliation): Payout settlement batch split across multiple customer charges.
- Double-Entry Balance Invariant Checks: Continuously verify that Sum(Assets) - Sum(Liabilities) == Sum(Equity) and flag unreconciled drift / phantom funds immediately via CLI exits or webhook alerts.
Claude, Codex: Where should we push first? Do you think we should focus on an embeddable library (e.g. Rust core with Python/Node bindings) or a self-hosted daemon with a CLI and SQLite/PostgreSQL backend? And what edge cases in financial reconciliation (e.g., FX slippage, chargeback clawbacks) should we bake into the core spec from day one?
Good frame, but I want to correct two things before we build on them, because both will bite us in the spec.
1. The invariant you wrote is not the invariant. Sum(Assets) - Sum(Liabilities) == Sum(Equity) is a report — a balance sheet — not something an engine can continuously assert. It's also undefined across currencies unless you've picked a translation policy (spot at event time? month-end close rate? ASC 830 CTA bucket?). The assertion that actually holds every microsecond is narrower and stronger:
- Per transaction, per currency:
sum(debits) == sum(credits). No multi-currency journal entry balances without an explicit FX-gain/loss line, so the engine must refuse the entry rather than silently net it. - Per account, over any window:
closing_balance == opening_balance + sum(postings). - Per source:
sum(ledger postings tagged source=stripe) == stripe_reported_balanceat a stated cut time.
That third one is the reconciliation product. The first two are just a correct ledger.
2. The hole isn't where you said it is. TigerBeetle already owns the double-entry primitive — open source, Zig, and it does debit/credit with two-phase transfers at absurd throughput. Beancount and hledger own the plaintext ledger and already have plugin hooks and balance assertions. If we ship another ledger core we are competing with those and losing. The genuinely empty square is one layer up: the ingestion + matching + evidence layer between a processor's settlement report and whatever ledger you already run. Modern Treasury and Sequence sell exactly this and there is no credible open-source equivalent. Let's build that and stay ledger-agnostic — emit Beancount directives, or postings over TigerBeetle's API, or plain SQL. Pluggable sink, not our own ledger.
Library vs daemon: library, and Python, not Rust-with-bindings. The people who will actually adopt this are one controller and one backend engineer at a Series A, and they live in pandas and Airflow. Rust buys us throughput we don't need — a marketplace doing $50M GMV at a $60 average order is ~70k transactions a month; that's 840k rows a year. DuckDB scans that in milliseconds on a laptop. Write it in Python against an embedded DuckDB/SQLite file, ship a recon CLI that wraps the same functions, and if someone shows up with 10^8 rows we rewrite the matcher — one module — later. Rust on day one costs us every casual contributor.
The failure mode nobody has named yet: time is two-dimensional. Every reconciliation engine I've seen dies on restatements, not on FX. Stripe amends a balance_transaction; a bank re-issues a CAMT.053 for a value date you already closed; a chargeback lands 45 days after you matched and cleared the original charge. If your matches are mutable rows, you have just silently changed what the books said last quarter, and an auditor asking "what did the March 31 close look like as of April 5" gets an answer that no longer exists.
So I'd make bitemporality non-negotiable in the core schema: every event and every match carries value_time (when it happened economically) and record_time (when we learned it). Matches are never updated or deleted — a new match row carries supersedes_match_id. Every match row also carries rule_id, rule_pack_version, reason_code, and the raw payload hash of both sides. Then "as of" queries are just a WHERE record_time <= ?, and the whole engine becomes replayable: same input events + same rule pack version → byte-identical match set. That property is what makes it testable, and it's the thing heuristic matchers usually can't claim.
Which means your Pass 2 and Pass 3 should not be Python functions. They should be declarative rules — data, versioned in the repo, with a tolerance and a window and a fee formula — so a user can add a rule for their PSP without forking us, and so a rule change is a diff someone can review.
One more thing I'd put in the MVP that isn't engine work at all: an open corpus of anonymized fixtures. Real CAMT.053 files, a Stripe payout with an embedded refund and a chargeback clawback, a Wise multi-currency payout with a mid-market/actual-rate gap, an Adyen batch where one payout covers 1,412 charges. Nobody has published these. Even if our matcher turns out mediocre, that corpus is the asset — it's what lets anyone else test their matcher, and it's how we get contributors.
What I don't have a good answer for: how a partial match is represented when a $12,400.13 payout covers charges summing to $12,400.00 and the 13¢ is an unexplained rounding artifact. Do we post a plenty-small suspense entry automatically under a threshold, or hard-fail? I lean hard-fail with an explicit recon accept-variance command that writes a signed, attributable adjustment — because auto-suspense under a threshold is exactly how a $0.13 bug becomes a $130,000 bug when someone bumps the threshold. Curious whether Codex thinks that's too strict for a first release.
On the variance question: hard-fail is right for the core engine, but I would not make the user hand-edit data to move forward. The command should be explicit and auditable:
recon variance accept --case rc_2026_000184 --amount 13 --currency USD --account processor_rounding --reason "Stripe payout net off by 13 cents vs charges/refunds in report bt_..."
That writes a new adjustment event with accepted_by, accepted_at, reason, case_id, and a hash of the unmatched evidence set. It should never mutate the match. Default threshold is zero. A project can configure a policy like max_auto_variance_cents: 0; if they change it to 5, the rule-pack version changes and old runs remain reproducible. I agree that silent suspense is poison.
I also agree we should not build a ledger core. The better shape is: Python library, DuckDB-backed local workspace, CLI first, pluggable exports. I would make the first supported sink Beancount because it is text-diffable and lets users inspect output without running another service. TigerBeetle can be a later adapter once the evidence layer has proved itself.
Where I disagree slightly: declarative rules are necessary, but “Pass 2 and Pass 3 should not be Python functions” is too absolute. A pure YAML DSL will either be too weak for real PSP oddities or will slowly become a worse programming language. I’d split it:
- Matching rules are declarative specs compiled to SQL for the common 80 percent: IDs, dates, amount tolerances, fee formulas, grouping keys, settlement windows.
- Normalizers and enrichers are versioned Python plugins with a narrow interface: input raw event, output canonical fields plus evidence tags. Those are where ugly PSP-specific parsing lives.
- Every rule/plugin execution records
rule_id,rule_pack_version,code_hash, input payload hashes, and output rows.
Concrete MVP repository shape:
openrecon/
core/events.py: canonical event model, bitemporal columns, payload hashing.core/cases.py: unmatched/mismatched evidence sets, not just “transactions.”rules/engine.py: compiles rule specs to DuckDB queries.connectors/stripe.py: Stripe balance transactions CSV/JSON first.connectors/bank_csv.py: boring CSV bank feed before CAMT.053, because it makes the MVP usable immediately.sinks/beancount.py: exports generated postings plus comments linking back to evidence hashes.cli.py:ingest,match,cases,explain,export,variance accept.fixtures/: synthetic but realistic payout/refund/chargeback datasets with expected matches.
The missing product concept is a “reconciliation case.” A case is the thing a human works: one payout batch, one unmatched bank debit, one chargeback clawback, one FX discrepancy. It owns the evidence graph: raw source rows, derived events, candidate matches, rejected candidates, accepted variance, and exported postings. That gives the CLI a useful surface:
recon cases list --status open --as-of 2026-04-05
recon explain rc_2026_000184
recon export beancount --case rc_2026_000184
For the first fixture, I’d use a Stripe payout of $12,400.13 consisting of 217 charges, 4 refunds, 1 dispute fee, and one 13 cent unexplained remainder. Expected result: 216 exact/fee matches, 5 grouped matches, 1 open variance case. That forces us to design batch matching, evidence display, and variance handling before we drift into a ledger project.
We have clean alignment on the foundation:
- Scope: OpenRecon is an evidence, matching, and case-resolution engine sitting between payment rails/bank feeds and ledger sinks. We do not write another ledger core.
- Tech stack: Python + embedded DuckDB, CLI-first, Beancount as the initial reference sink.
- Immutability & Bitemporality: Value time vs record time on every event, append-only matches with
supersedes_match_id, zero in-place mutations. - Core domain entity: The Reconciliation Case (
core/cases.py) and an explicitrecon variance acceptworkflow that creates audited adjustment events rather than sweeping pennies into silent suspense accounts.
Codex's rule split is the right compromise: YAML-defined matching predicates compiled directly to DuckDB SQL queries, with versioned Python plugins handling raw-to-canonical normalisation.
Let's push into two areas that break most reconciliation tools in practice:
1. Multi-Stage Lifecycle & Temporal Linkage (Disputes / Reserves)
Consider a common lifecycle:
- Day 0: $100.00 charge (Case A: matches customer order).
- Day 20: $100.00 dispute + $15.00 dispute fee debited in a payout batch (Case B).
- Day 45: Dispute won; $100.00 refunded back to merchant + $15.00 dispute fee reversed in a later payout batch (Case C).
If Case A was already closed and exported to Beancount at Month 1 close, how do Case B and Case C link back to Case A without reopening Month 1's book?
I propose each case carries an optional parent_case_id and an explicit lifecycle_state (opened, disputed, settled, reversed). Case B does not alter Case A; it generates new delta postings referencing Case A's canonical transaction ID and attaches the processor's dispute evidence.
2. The Chart of Accounts Bridge & Posting Rules
Reconciliation cannot export to Beancount or TigerBeetle without knowing where things land. If we hardcode account names in connectors, nobody can use it. If we make users write full accounting scripts, they won't use it.
I suggest a declarative accounts.yaml mapping file:
accounts:
operating_bank: "Assets:Bank:Mercury:Checking"
clearing_stripe: "Assets:PaymentProcessors:Stripe:Clearing"
fees_processing: "Expenses:Financial:ProcessorFees"
dispute_fees: "Expenses:Financial:DisputeFees"
rounding_variance: "Expenses:Financial:RoundingVariance"
posting_templates:
stripe_payout_settlement:
- debit: "operating_bank"
amount: "event.net_amount"
- debit: "fees_processing"
amount: "event.total_fees"
- credit: "clearing_stripe"
amount: "event.gross_amount"
Claude, Codex:
- Does this posting template model cover the 1:Many payout split cleanly, or does the template need to be evaluated per child item in the batch?
- What should the exact schema for
reconciliation_casesandmatch_evidencein DuckDB look like to ensure full replayability?
Taking the two questions in order, and the answer to the first is: the template model as written is broken, and the 13¢ fixture is exactly what breaks it.
1. Templates must evaluate per child, and the payout entry is a closure check, not a posting
stripe_payout_settlement reads event.total_fees and event.gross_amount off the payout. But a Stripe payout isn't an event with those fields — it's a batch whose children are balance transactions of six different types (charge, refund, adjustment, dispute, dispute_fee, application_fee), each with its own fee, its own currency, and its own sign. Aggregating them into total_fees at the payout level means the engine derives net from the reported payout amount, which is the one number you must never trust, because it's the number you're reconciling against.
Worked, with the fixture pinned down:
| child | count | amount |
|---|---|---|
| charges (gross) | 217 | +13,000.00 |
| processing fees | 217 | −416.00 |
| refunds | 4 | −69.00 |
| dispute | 1 | −100.00 |
| dispute fee | 1 | −15.00 |
| children sum | 440 | 12,400.00 |
| payout, as reported by Stripe | 12,400.13 |
Under the facilitator's template the engine computes net from event.net_amount = 12,400.13, debits the bank 12,400.13, and the entry balances perfectly. The 13¢ vanishes. Under per-child evaluation you get 440 entries that move 12,400.00 into clearing_stripe, one settlement entry that moves 12,400.13 out of it, and a residual clearing balance of −0.13 that the case cannot close on. That residual is the finding.
So I'd restate it as one rule: a batch case may close only when the post-export balance of its clearing account is exactly zero. Templates are keyed on canonical child event type, never on the batch. The batch contributes one settlement entry and one invariant.
2. Two schema corrections before the DDL
Money. Kill amount_cents. JPY has exponent 0, BHD and KWD have 3, and if anyone ever points a connector at USDC you need 6. Store amount_minor BIGINT plus currency CHAR(3) plus exponent SMALLINT, and never a float anywhere in the pipeline — DuckDB will happily give you 0.1 + 0.2 in a SUM if you let it.
Mutable state contradicts the append-only stance. lifecycle_state as a column on the case row is an in-place update, which is the thing we just said we don't do. Make it derived from an append-only case_state_transitions table; current state is the latest row by record_time. Same argument as matches.
And parent_case_id is too narrow. The day-45 reversal relates to the dispute case and to the original charge case, and the payout batch contains 217 charge cases. That's a DAG. One nullable parent pointer forces you to pick one edge and lose the other, then re-parent later. Use an edge table — it's four columns and it never needs a migration:
CREATE TABLE case_links (
from_case_id VARCHAR NOT NULL,
to_case_id VARCHAR NOT NULL,
link_type VARCHAR NOT NULL, -- contains | reverses | corrects | settles
record_time TIMESTAMPTZ NOT NULL,
PRIMARY KEY (from_case_id, to_case_id, link_type)
);
3. The replayability schema
-- what we were handed, byte for byte
CREATE TABLE source_files (
file_id VARCHAR PRIMARY KEY, -- blake2b of bytes
connector VARCHAR NOT NULL,
connector_ver VARCHAR NOT NULL,
ingested_at TIMESTAMPTZ NOT NULL,
byte_len BIGINT NOT NULL
);
CREATE TABLE raw_rows (
raw_id VARCHAR PRIMARY KEY, -- blake2b(file_id || row_ordinal || bytes)
file_id VARCHAR NOT NULL,
ordinal BIGINT NOT NULL,
payload JSON NOT NULL
);
-- canonical, bitemporal, append-only
CREATE TABLE events (
event_id VARCHAR PRIMARY KEY, -- derived, see below
raw_id VARCHAR NOT NULL,
source VARCHAR NOT NULL, -- stripe | mercury | adyen
external_id VARCHAR, -- bt_1P..., end-to-end id
kind VARCHAR NOT NULL, -- charge | fee | refund | dispute | payout | ...
amount_minor BIGINT NOT NULL,
currency CHAR(3) NOT NULL,
exponent SMALLINT NOT NULL,
value_time TIMESTAMPTZ NOT NULL, -- when it happened economically
record_time TIMESTAMPTZ NOT NULL, -- when we learned it
supersedes_id VARCHAR, -- restatement chain
normalizer VARCHAR NOT NULL,
code_hash VARCHAR NOT NULL
);
CREATE TABLE matches (
match_id VARCHAR PRIMARY KEY, -- derived, see below
rule_id VARCHAR NOT NULL,
rule_pack_version VARCHAR NOT NULL,
reason_code VARCHAR NOT NULL, -- EXACT_ID | FEE_TOLERANCE | BATCH_SUM | ...
evidence_digest VARCHAR NOT NULL, -- blake2b over sorted member event_ids
record_time TIMESTAMPTZ NOT NULL,
supersedes_match_id VARCHAR
);
CREATE TABLE match_members (
match_id VARCHAR NOT NULL,
event_id VARCHAR NOT NULL,
side VARCHAR NOT NULL, -- internal | external
PRIMARY KEY (match_id, event_id)
);
The piece that actually buys replayability isn't the columns — it's that no id is a sequence. match_id = blake2b(rule_id || rule_pack_version || evidence_digest), event_id = blake2b(raw_id || normalizer || code_hash). Autoincrement ids make a rerun produce a diff even when nothing changed, which destroys the golden-file test suite that's supposed to be our proof of correctness.
And a determinism trap specific to our stack: DuckDB's hash joins are parallel, so candidate ordering is not stable across runs or across machines with different core counts. Any rule query that can produce competing candidates must end with an explicit ORDER BY (value_time, external_id, event_id) and pick by an explicit rank, or the same input will produce different matches on a 4-core CI runner than on the dev's 16-core laptop. That bug takes about three weeks to find.
One operational thing nobody has flagged: DuckDB is single-writer. One process holds the write lock on the file. So recon ingest running as four parallel Airflow tasks will fail, not queue. Either the CLI takes an advisory lock and serializes, or ingest writes Parquet per source and a single merge step loads them. I'd pick the second — Parquet-per-ingest is also a free append-only archive of exactly what each run saw, which is half of the audit story for nothing.
On the dispute lifecycle: I agree Case B never touches Case A. But there's a subtlety in your day-45 example — when the dispute is won, Stripe returns the $100 and reverses the $15 as two separate balance transactions in different payouts sometimes weeks apart. So "reversed" is not one state transition, it's two partial ones, and a case sitting in disputed with $100 back and $15 outstanding is a legitimate long-lived state. Whatever state machine we draw needs to tolerate partial reversal rather than treating it as an error.
The template critique is right: payout-level posting templates are dangerous because they let the reported settlement number balance itself. The export layer should make the discrepancy visible in the clearing account, then the case layer decides whether it is closed, open, or closed-with-accepted-variance.
I would model posting rules at two levels, but only one level is allowed to create accounting lines from money movement:
event_posting_templates: evaluated per canonical child event. These create charge, fee, refund, dispute, reserve, and reversal postings.case_closure_rules: evaluated over a case graph. These assert things likesum(Assets:PaymentProcessors:Stripe:Clearing, case_id) == 0andsum(child.amount_minor) == settlement.amount_minorper currency/exponent.
The payout event can create one bank-vs-clearing settlement posting because that is real money landing in the bank. It cannot summarize child gross/fee/refund activity. That keeps the 13 cent fixture exposed: children move 12,400.00 through clearing, payout moves 12,400.13 out, closure fails with -0.13.
I’d tighten the schema around cases. matches alone does not tell the UI or CLI what a human is resolving.
CREATE TABLE reconciliation_cases (
case_id VARCHAR PRIMARY KEY, -- blake2b(case_type || natural_key || first_record_time)
case_type VARCHAR NOT NULL, -- payout_batch | bank_transfer | dispute | fx_variance
source VARCHAR NOT NULL,
natural_key VARCHAR NOT NULL, -- Stripe payout id, dispute id, bank statement line id
currency CHAR(3), -- nullable only for genuinely multi-currency cases
exponent SMALLINT,
opened_record_time TIMESTAMPTZ NOT NULL,
rule_pack_version VARCHAR NOT NULL,
evidence_digest VARCHAR NOT NULL
);
CREATE TABLE case_events (
case_id VARCHAR NOT NULL,
event_id VARCHAR NOT NULL,
role VARCHAR NOT NULL, -- settlement | child | fee | dispute | reversal | bank_line
record_time TIMESTAMPTZ NOT NULL,
PRIMARY KEY (case_id, event_id, role)
);
CREATE TABLE case_state_transitions (
transition_id VARCHAR PRIMARY KEY, -- content hash, not sequence
case_id VARCHAR NOT NULL,
from_state VARCHAR,
to_state VARCHAR NOT NULL, -- open | matched | variance_pending | exported | closed
reason_code VARCHAR NOT NULL,
actor VARCHAR NOT NULL, -- system | user:<id>
record_time TIMESTAMPTZ NOT NULL,
evidence_digest VARCHAR NOT NULL
);
CREATE TABLE accepted_variances (
variance_id VARCHAR PRIMARY KEY,
case_id VARCHAR NOT NULL,
amount_minor BIGINT NOT NULL,
currency CHAR(3) NOT NULL,
exponent SMALLINT NOT NULL,
account_key VARCHAR NOT NULL,
reason VARCHAR NOT NULL,
actor VARCHAR NOT NULL,
record_time TIMESTAMPTZ NOT NULL,
evidence_digest VARCHAR NOT NULL
);
One correction to the replay schema: raw_rows.payload JSON is not enough if we claim byte-identical replay. JSON parsing can reorder keys or normalize numbers. Keep the original bytes or a path to content-addressed bytes, then derive raw_rows from that. I’d store ingests as archive/blake2b=<file_id>.parquet for parsed rows plus archive/blake2b=<file_id>.raw for exact original input when licensing/privacy allows. If we cannot store original bytes, the CLI should say replay_grade=canonical_only, not pretend it has full audit fidelity.
The other missing failure mode is rule-pack migration. A company will close March with stripe-v1.2.0, then discover in May that application fees were mapped wrong and install stripe-v1.3.0. We need a first-class command that compares rule packs without rewriting history:
recon replay --from stripe-v1.2.0 --to stripe-v1.3.0 --as-of 2026-03-31 --case-type payout_batch
Output should be a diff: cases unchanged, cases newly closed, cases reopened, postings changed by account/currency. That becomes the pull request artifact a controller can review before accepting a new rules package.
For the state machine, I’d avoid one linear lifecycle. Use derived flags from case state and balances:
open: case exists with required evidence missing or no successful match.matched: evidence set satisfies matching rule, but closure invariant has not necessarily passed.variance_pending: matched enough to explain source rows, but clearing residual != 0.exported: sink wrote output and storedexport_digest.closed: exported plus all closure rules pass, or exported plus accepted variance exactly offsets the residual.
A dispute can then be partially reversed without a special state explosion: the case has two linked reversal events expected, one present, one absent. The explain command can say principal_reversed=true, fee_reversed=false, outstanding=-1500 USD minor units. That is clearer than pretending “reversed” is atomic.
The architecture is locking into place with exceptional rigor.
Let's codify the key decisions we've established:
- Deterministic Identity: Zero autoincrement IDs. Every
event_id,match_id,case_id,transition_id, andvariance_idis a content-addressed BLAKE2b hash. - Deterministic Querying: All DuckDB match queries enforce strict deterministic tie-breaker sorting
ORDER BY (value_time, external_id, event_id)with explicit ranking. - Storage & Ingestion: Single-writer safety via content-addressed raw files (
archive/blake2b=<id>.raw) and Parquet batches (archive/blake2b=<id>.parquet), loaded into DuckDB during the merge/match phase. - Two-Level Accounting Model:
- Level 1:
event_posting_templatesgenerate postings strictly per individual child event (charge, fee, refund, dispute, reversal). - Level 2:
case_closure_rulesassert clearing balance invariants (sum(clearing) == 0).
- Level 1:
- State Progression: Derived status (
open->matched->variance_pending->exported->closed). - Rule-Pack Versioning & Diffing:
recon replay --from v1.2.0 --to v1.3.0producing case and posting diffs for review.
Let's now tackle the two hardest operational hurdles to finalize our repository design:
1. Multi-Currency, FX Realization, and the Zero-Sum Invariant
In cross-border transactions (e.g., charge in EUR, processor settlement in USD, merchant operating bank in GBP), three amounts exist at three different timestamps.
To prevent leaking multi-currency imbalance into the ledger:
- Every event records
amount_minor,currency,exponent, plus optionalsettlement_amount_minorandsettlement_currency. - When an event spans currencies, the posting template cannot emit an unbalanced multi-currency journal entry. It must emit two single-currency legs bridged by a dedicated FX clearing/conversion account (
Equity:FXGainLossorAssets:FXClearing), with the explicit exchange rate derived from the processor's settlement statement.
2. Concrete Rule DSL Compilation
Let's specify what a declarative matching rule looks like in YAML and how the compiler translates it to DuckDB SQL.
For example, rules/stripe_payout_batch.yaml:
rule_id: stripe_payout_1toN_v1
version: "1.0.0"
target_case_type: payout_batch
description: "Matches a Stripe payout bank deposit to the set of balance transactions in the payout batch"
predicates:
parent:
source: stripe
kind: payout
children:
source: stripe
kinds: [charge, refund, dispute, fee, adjustment]
join_key: payout_id # matches parent.external_id
invariants:
- name: amount_match
expression: "sum(children.amount_minor) == parent.amount_minor"
- name: currency_match
expression: "all(children.currency == parent.currency)"
The compiler turns this into a deterministic DuckDB CTE query that groups children by payout_id, computes the aggregate sum and count, joins with the parent payout event, and outputs matching candidate sets with deterministic rank.
3. Core Fixture Suite
Let's define the 4 canonical fixtures in fixtures/:
stripe_payout_with_variance: 217 charges, 4 refunds, 1 dispute fee, 13¢ rounding variance.chargeback_lifecycle_3stage: Day 0 charge, Day 20 dispute + fee, Day 45 win + fee reversal across two payout months.cross_currency_eur_usd: EUR 100 customer charge, USD 108.50 payout settlement, USD 2.10 processor fee, EUR 0.40 FX variance.camt053_bank_vs_ledger: CAMT.053 XML statement matching against internal ledger payouts and ACH batches.
Claude, Codex:
How should the CLI subcommand structure and configuration files (openrecon.yaml, accounts.yaml) be structured so that a developer running recon init gets a working workspace immediately?
Before the CLI question, three corrections — one of them undoes a decision that was just codified.
The YAML rule contradicts the two-level split we agreed on ten minutes ago
rules/stripe_payout_batch.yaml puts this in invariants:
- name: amount_match
expression: "sum(children.amount_minor) == parent.amount_minor"
Run that against fixture 1. Children sum to 12,400.00, the payout says 12,400.13, the invariant fails, the rule doesn't fire — and therefore no case is created. You end up with 440 unmatched balance transactions and one unmatched payout sitting in a pile, which is precisely the state a controller was already in before installing us. The whole point of the case object is that the discrepancy has somewhere to live.
Matching predicates group. Closure rules assert. sum(children) == parent is a closure rule and it belongs in case_closure_rules, exactly where the two-level model put it. What the matching rule should say is only "children are the balance transactions whose payout_id equals this payout's external_id" — no arithmetic at all. Same objection kills all(children.currency == parent.currency): that predicate makes fixture 3 unmatchable, since a EUR charge inside a USD payout is the normal case, not an anomaly.
And join_key: payout_id only expresses Pass 1. Fixture 4 has no join key — a CAMT.053 line and an ACH batch share nothing but an amount and a date window. The DSL needs a second predicate form or it can only describe the easy half:
predicates:
left: { source: mercury, kind: credit }
right: { source: internal, kind: ach_batch }
match_on:
- { field: amount_minor, op: eq }
- { field: value_time, op: within, tolerance: 48h }
cardinality: "1:1"
rank_by: [abs_time_delta, event_id] # deterministic tie-break, mandatory
FX gain/loss is not equity
Equity:FXGainLoss is wrong. Transaction-level FX gain or loss hits the income statement — Income:FX:Gain and Expenses:FX:Loss in Beancount's five-root world. The thing that lives in equity is CTA, the cumulative translation adjustment from restating a subsidiary's whole functional-currency financials under ASC 830, and we're nowhere near that. Ship the wrong root and every user's P&L is understated by their FX drift, silently, forever.
The related trap: "the explicit exchange rate derived from the processor's settlement statement." Stripe doesn't hand you a rate on a balance transaction — it hands you a presented amount and a settled amount, and the rate is implied. If you store rate = 1.0850 and recompute, EUR 100.00 × 1.0850 gives you 108.50 today and 108.4999999 the day someone touches it with a float. So: store both legs as integers, never a rate. The rate is a display artifact computed at render time from the two amounts. That also makes fixture 3's 0.40 EUR variance fall out as a real integer residual rather than a rounding argument.
Now the CLI, and the thing recon init has to guarantee
The test for recon init isn't "a workspace exists." It's that these three commands work with no credentials, no bank account, and no network:
$ recon init --with-fixture stripe_payout_with_variance
$ recon match
$ recon cases list
CASE TYPE STATE RESIDUAL
rc_a3f19c… payout_batch variance_pending -0.13 USD
That's the README's first sixty seconds and it's also the acceptance test for the MVP. If it needs a Stripe key, nobody evaluates us.
Workspace layout:
myco-recon/
openrecon.yaml # engineer's file: sources, rule-pack pins, sinks
accounts.yaml # controller's file: chart-of-accounts mapping
rules/ # vendored rule packs, pinned by version
archive/ # content-addressed .raw + .parquet — SYSTEM OF RECORD
workspace.duckdb # derived cache, deletable
exports/
.gitignore # archive/, workspace.duckdb, exports/
Two files rather than one, deliberately: accounts.yaml is reviewed by the person who owns the chart of accounts and changes at a different cadence than source config. Different reviewer, different file, different PR.
The design line I'd insist on: workspace.duckdb is a cache, not a database. recon rebuild reconstructs it byte-identically from archive/ alone. The moment that file becomes precious we have built a database, and every corrupted page, every failed migration, every "can I copy this to my laptop" is our support burden. Keep it disposable and the archive is the only thing that needs backing up.
Subcommands, flat, every read command taking --as-of:
recon init [--with-fixture NAME]
recon ingest <source> <path...> # writes archive/, never touches the db
recon rebuild # archive/ -> workspace.duckdb
recon match [--rule-pack stripe@1.2.0]
recon cases list|show|explain
recon variance accept --case …
recon export beancount --out …
recon replay --from … --to …
recon check # exit code, for CI
recon doctor
recon close --period 2026-03 --cutoff 2026-04-05T00:00:00Z
recon check exit codes need to distinguish two very different things, because a CI job should behave differently for each: 0 everything closed; 1 open or variance_pending cases exist (normal, someone has work to do); 2 a closure invariant is violated with no case explaining it (this is the alarm — money is missing and the engine can't say where); 64 usage/config error. Collapsing 1 and 2 into "nonzero" is how the alarm gets muted by a || true in someone's Makefile.
recon close is the object nobody has named yet. Right now --as-of is a query filter, which means "the March close" is a thing you reconstruct by remembering which flags you typed. A close should be a first-class, content-addressed row that pins: the period, the cutoff record_time, the exact set of archive/ file ids in scope, every rule-pack version used, the resulting case states, and the export_digest of what was written to the ledger. Then recon close verify 2026-03 re-derives all of it and tells you whether it still reproduces. That artifact is what an auditor is actually asking for when they ask "how do you know the books are right", and it costs us one table.
Two smaller things:
Secrets and PII. openrecon.yaml holds ${STRIPE_API_KEY} style references and never a literal. Better still, the MVP shouldn't need API access at all — ingest a file the user already downloaded. And archive/*.raw will contain real IBANs and account numbers, so recon init writes the .gitignore and recon doctor hard-fails if git is tracking anything under archive/. People will otherwise commit a CAMT.053 to a public repo; it has happened to every treasury tool that shipped without that check.
The binary name is a problem. recon is thoroughly taken in the security world — recon-ng, and "recon" as a search term returns pentest tooling for pages. Anyone Googling our error messages lands in reconnaissance frameworks. I'd keep OpenRecon as the project name and ship the binary as something searchable. Not worth spending a turn on, but worth not defaulting into.
The correction on the sample YAML is important. Matching rules should create the case and attach evidence even when arithmetic fails. Arithmetic belongs in closure rules. If the fixture with the 13 cent residual does not produce one clear variance_pending case, the product has failed its own demo.
For the binary name, I would not ship recon. Use openrecon as the canonical command despite the extra characters, and optionally add a short alias later if users ask for it. Searchability matters more than saving five keystrokes, especially for a tool whose errors will be pasted into GitHub issues.
recon init should create a working workspace, but I’d structure it like this:
myco-openrecon/
openrecon.yaml
accounts.yaml
rules.lock
rules/
stripe/1.0.0/*.yaml
bank_csv/1.0.0/*.yaml
archive/
raw/
parquet/
exports/
reports/
workspace.duckdb
.gitignore
I’d add rules.lock, separate from openrecon.yaml, because rule resolution should be reproducible like dependency resolution. openrecon.yaml says “use stripe ^1.0”; rules.lock pins stripe 1.0.3 with package digest, every rule digest, and the normalizer code hash. openrecon replay --from-lock old.lock --to-lock rules.lock is easier to reason about than comparing loose version strings.
Concrete openrecon.yaml:
workspace_version: 1
archive:
raw_dir: archive/raw
parquet_dir: archive/parquet
store_raw_bytes: true
pii_policy: local_only
duckdb:
path: workspace.duckdb
threads: 4
rule_packs:
- name: stripe
constraint: "1.0.x"
- name: bank_csv
constraint: "1.0.x"
sources:
stripe_balance_transactions:
connector: stripe.csv
rule_pack: stripe
account_profile: stripe_default
bank_operating:
connector: bank_csv
rule_pack: bank_csv
account_profile: mercury_operating
sinks:
beancount:
enabled: true
out_dir: exports/beancount
operating_currency: USD
controls:
require_raw_archive: true
forbid_tracked_archive: true
default_as_of: now
auto_accept_variance_minor: 0
Concrete accounts.yaml:
account_profiles:
stripe_default:
clearing: Assets:PaymentProcessors:Stripe:Clearing
fees_processing: Expenses:Financial:ProcessorFees
fees_dispute: Expenses:Financial:DisputeFees
refunds: Income:Sales:Refunds
chargebacks: Expenses:Financial:Chargebacks
variance_rounding: Expenses:Financial:RoundingVariance
fx_gain: Income:FX:Gain
fx_loss: Expenses:FX:Loss
mercury_operating:
bank: Assets:Bank:Mercury:Checking
posting_templates:
stripe.charge:
- debit: stripe_default.clearing
amount: event.amount
- credit: Income:Sales:Gross
amount: event.amount
stripe.processing_fee:
- debit: stripe_default.fees_processing
amount: abs(event.amount)
- credit: stripe_default.clearing
amount: abs(event.amount)
stripe.payout:
- debit: mercury_operating.bank
amount: event.amount
- credit: stripe_default.clearing
amount: event.amount
I would keep expressions deliberately constrained. Amount references can be event.amount, abs(event.amount), or named derived integer fields. No arbitrary Python in accounts.yaml. If the controller file can execute code, it becomes unreviewable.
On close, I agree it needs to be first-class. I’d make the generated close artifact human-readable as well as machine-verifiable:
reports/closes/2026-03/
close.json # content-addressed manifest
case_states.csv # case_id,type,state,residual,export_digest
postings.beancount # exact exported file or digest reference
replay.diff # empty on initial close, populated on later verification drift
The close table can then store only hashes and paths:
CREATE TABLE closes (
close_id VARCHAR PRIMARY KEY,
period VARCHAR NOT NULL,
cutoff_record_time TIMESTAMPTZ NOT NULL,
archive_digest VARCHAR NOT NULL,
rules_lock_digest VARCHAR NOT NULL,
accounts_digest VARCHAR NOT NULL,
case_states_digest VARCHAR NOT NULL,
export_digest VARCHAR NOT NULL,
created_by VARCHAR NOT NULL,
created_at TIMESTAMPTZ NOT NULL
);
One thing I’d change in the subcommands: split match and check from close, but make close run them in a fixed pipeline unless --no-match is passed. The command a controller runs should be boring:
openrecon init --with-fixture stripe_payout_with_variance
openrecon rebuild
openrecon match
openrecon cases list
openrecon cases explain rc_a3f19c
openrecon variance accept --case rc_a3f19c --amount 13 --currency USD --account stripe_default.variance_rounding --reason "processor residual below investigated materiality"
openrecon close --period 2026-03 --cutoff 2026-04-05T00:00:00Z
openrecon close verify 2026-03
For the README demo, the golden path should stop before variance accept and show the open case. That demonstrates the tool finds a real problem instead of smoothing it over.
We have reached complete consensus on the CLI ergonomics, rules.lock determinism model, accounts.yaml constraint bounds, and the first-class openrecon close cryptographic manifest.
Let's nail down the last two operational subsystems required for the repository blueprint:
1. Ingestion Quarantine & Malformed Record Isolation
Real-world bank CSVs and webhook dumps frequently contain duplicate rows, mangled date encodings, or missing required columns.
If an ingestion step crashes halfway through, it leaves a dirty partial state; if it silently skips bad rows, it induces false reconciliation breaks downstream.
I propose OpenRecon implement a Quarantine Architecture:
- Ingestion executes in atomic batches per input file.
- Rows failing parsing or schema validation are written to
archive/quarantine/<file_id>.quarantine.jsonlwith exact line number, raw content, and validation error. - The ingest manifest records
total_rows,valid_events,quarantined_rows. - If
quarantined_rows > 0, the batch is flagged withingest_warning=true. The closure engine refuses to close a period (openrecon close) if any ingested file in the period contains unresolved quarantine items without explicit waiver.
2. The Pluggable Sink Contract (sinks/base.py)
To keep the engine strictly ledger-agnostic, the export layer should consume an immutable ExportBatch dataclass:
@dataclass(frozen=True)
class ExportPosting:
account: str
amount_minor: int
currency: str
exponent: int
value_date: date
narration: str
metadata: dict[str, str] # case_id, rule_id, evidence_digest
@dataclass(frozen=True)
class ExportBatch:
period: str
cutoff_record_time: datetime
cases: list[ReconciliationCase]
postings: list[ExportPosting]
accepted_variances: list[AcceptedVariance]
The Beancount sink (sinks/beancount.py) translates this into:
2026-03-31 * "Stripe Payout Settlement" ^rc_a3f19c
recon_case: "rc_a3f19c"
evidence_digest: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
Assets:Bank:Mercury:Checking 12400.13 USD
Assets:PaymentProcessors:Stripe:Clearing -12400.13 USD
3. Pytest Golden-File Harness (tests/)
Our proof-of-correctness requires a deterministic test runner:
tests/test_fixtures.pyiterates overfixtures/*/.- Each fixture directory contains
input/(raw files),expected_cases.json, andexpected_postings.beancount. - Test runs
openrecon init -> ingest -> match -> closeinto an in-memory DuckDB instance and asserts byte-for-byte equality against expected goldens.
Claude, Codex:
Are there any missing edge cases in the quarantine model or sink interface? If this feels tight, we are ready to start structuring the complete repository blueprint, README, and code implementation manifest for the final deliverable.
The quarantine model and the sink contract both have gaps, but there's a third thing that's more urgent: the golden-file harness as described cannot pass, ever. Let me take that first because it invalidates the proof-of-correctness plan the whole design rests on.
Wall-clock is in the hashes
source_files.ingested_at, case_state_transitions.record_time, closes.created_at, and default_as_of: now are all wall-clock. Run the fixture suite twice and every one of those differs, so "byte-for-byte equality against expected goldens" fails on run two. Worse, it fails nondeterministically — CI goes red on a rerun with no code change, everyone learns to hit retry, and the test suite stops meaning anything within a month.
Two fixes, both needed:
openreconreads a clock override —--record-timeon ingest, and aSOURCE_DATE_EPOCH-style env var the harness sets. Ingest defaultsrecord_timeto the statement's own date when the format carries one (CAMT.053<CreDtTm>, MT940's statement date), falling back to now only for formats that carry nothing.- Wall-clock fields are excluded from every content hash.
created_atcan sit inclose.jsonas a recorded fact; it must not be an input toclose_id.
And this one, which was decided three turns ago and is a bug:
case_id VARCHAR PRIMARY KEY, -- blake2b(case_type || natural_key || first_record_time)
first_record_time in the case id means ingesting the same file into a fresh workspace produces different case ids. openrecon close verify 2026-03 on a rebuilt workspace then reports every case as changed, and workspace.duckdb is a deletable cache stops being true. Drop it — blake2b(case_type || natural_key) is sufficient, because the natural key is by definition the thing that identifies the case.
Quarantine catches the wrong failure
The three examples given aren't one class. Duplicate rows parse fine — they're handled by content-addressing (raw_id = blake2b(file_id || ordinal || bytes)), not by quarantine. And the genuinely dangerous case is the one quarantine structurally cannot see: the row that parses cleanly and is wrong.
Concrete: a bank changes its CSV export and flips the sign convention on debits, or switches 1.234,56 to 1,234.56 between two downloads. Every row parses. Every row validates. quarantined_rows is 0, ingest_warning is false, and you have just booked the entire statement backwards.
What catches that is not row-level validation — it's file-level control totals, and the good news is the formats hand them to us:
- CAMT.053 carries
<Bal>withOPBDopening andCLBDclosing balances, plus<TxsSummy>/<TtlNtries>with entry counts and total credits/debits. - MT940 carries
:60F:opening and:62F:closing. - Stripe's payout reconciliation report carries its own totals per category.
So the ingest contract should be: if the source declares its own totals, ingest recomputes them from the parsed rows and fails the file when they disagree. Not a warning, not a quarantined row — the whole file is refused, because a control-total mismatch means you don't know what you parsed. For sources with no declared totals (a hand-exported CSV), fall back to a delta check against the previous file from the same source: row count, sum, date range, and debit/credit ratio, warning above a configured band. That's the layer that catches the sign flip.
Two smaller problems with quarantine as specified:
It's stored in archive/, which breaks the archive's meaning. archive/ is the immutable system of record; quarantine.jsonl is a derived parse result. Put it there and someone will hand-edit a date to "fix" a row, and replay silently stops reproducing. Quarantine should be recomputed by rebuild from the .raw bytes and live under the disposable side.
A waiver needs the same rigor as variance accept. "Explicit waiver" as written could be a flag on the close command, which means waiving 3 rows in March silently waives 300 in April. A waiver must pin the exact digest of the quarantined set it covers, carry an actor and a reason, and stop applying the moment that set changes.
Sink contract: four holes
-
There is no transaction boundary.
postings: list[ExportPosting]is flat, so the Beancount sink has to guess which postings belong to one entry — and it can't, sincecase_idin metadata isn't enough (one case emits 440 child entries plus a settlement). Add the level:ExportBatch.entries: list[ExportEntry], andExportEntryownsentry_id,date,payee,narration,links,postings. Which also fixes the next one: -
value_dateis on the wrong object. Beancount has one date per transaction, not per posting. Leaving it per-posting lets a sink receive an entry whose two legs disagree on date, and there's no correct rendering of that. -
There is no re-export story, and that's the contract's job, not the sink's. March is exported, then a restatement lands in May. Does the sink append to
2026-03.beancount, rewrite it, or emit a correcting entry? If we don't say, every sink will answer differently and the Beancount and TigerBeetle outputs will diverge. I'd make it a rule of the contract: sinks are write-once per close and never edit a file they previously wrote. A restatement produces a new close, in the new period, containing correcting entries that reference the originalcase_id. That's also just how accountants handle a prior-period adjustment, so it's the behavior a controller expects anyway. -
Rendering integers to decimal must be specified, and narration must be escaped.
Decimal(amount_minor).scaleb(-exponent)— spell it out in the base class, because the first sink that writesamount_minor / 10**exponentreintroduces floats through the back door. And narration comes from bank descriptions, which contain quotes, newlines, and occasionally bytes that aren't valid UTF-8; an unescaped"in a payee name produces a Beancount file that won't parse, and the user's first experience of us is a syntax error in their ledger.
One detail in the example entry: the evidence_digest shown, e3b0c442…, is the SHA-256 of the empty string (truncated to 63 characters). We picked BLAKE2b. Don't ship the empty-string hash as sample data — someone will grep for it, and someone else will write a test that asserts against it.
Agreed on the determinism bug. first_record_time in case_id is wrong; case identity should be stable across rebuilds. I’d define IDs this way:
source_file_id = blake2b(raw_bytes)
raw_row_id = blake2b(source_file_id || ordinal || raw_row_bytes)
event_id = blake2b(raw_row_id || normalizer_id || normalizer_code_hash || canonical_event_json)
case_id = blake2b(case_type || source || natural_key)
match_id = blake2b(rule_id || rule_pack_digest || sorted(event_ids) || match_shape)
close_id = blake2b(period || cutoff_record_time || archive_digest || rules_lock_digest || accounts_digest || export_digest)
created_at, ingested_at, and human actor timestamps are facts, not identity inputs. Fixtures should set OPENRECON_RECORD_TIME=2026-04-05T00:00:00Z and pass explicit cutoffs. No now in tests, no wall-clock in hashes.
The quarantine correction is also right: malformed rows are not the main risk. The main risk is a clean parse of the wrong semantics. I’d split ingest into four stages and make the status names precise:
raw_captured -> parsed -> controls_verified -> promoted
raw_captured: bytes copied to content-addressed storage.parsed: connector produced canonical rows plus derived quarantine output.controls_verified: source-level totals reconcile to parsed rows.promoted: events are eligible for matching.
Only promoted files can feed cases. A file with row parse failures is not promoted unless there is a waiver for the exact quarantine digest. A file with a declared control-total mismatch cannot be waived in the MVP. That sounds strict, but it prevents a user from waiving “3 malformed rows” when the real issue is that every debit sign flipped.
I’d add these tables:
CREATE TABLE ingest_manifests (
source_file_id VARCHAR PRIMARY KEY,
connector VARCHAR NOT NULL,
connector_version VARCHAR NOT NULL,
source_name VARCHAR NOT NULL,
raw_digest VARCHAR NOT NULL,
parsed_digest VARCHAR,
quarantine_digest VARCHAR,
controls_digest VARCHAR,
status VARCHAR NOT NULL, -- raw_captured | parsed | controls_verified | promoted | refused
refusal_reason VARCHAR,
record_time TIMESTAMPTZ NOT NULL
);
CREATE TABLE control_checks (
check_id VARCHAR PRIMARY KEY,
source_file_id VARCHAR NOT NULL,
check_name VARCHAR NOT NULL, -- closing_balance | total_debits | row_count | category_totals
expected_minor BIGINT,
actual_minor BIGINT,
currency CHAR(3),
exponent SMALLINT,
passed BOOLEAN NOT NULL,
evidence_digest VARCHAR NOT NULL
);
CREATE TABLE quarantine_waivers (
waiver_id VARCHAR PRIMARY KEY,
source_file_id VARCHAR NOT NULL,
quarantine_digest VARCHAR NOT NULL,
actor VARCHAR NOT NULL,
reason VARCHAR NOT NULL,
record_time TIMESTAMPTZ NOT NULL
);
Quarantine artifacts should live under workspace/quarantine/ or reports/quarantine/, not archive/, because they are derived. The archive remains raw inputs plus normalized Parquet snapshots if we decide those are source artifacts. Anything a user might be tempted to “fix” manually does not belong in the system-of-record directory.
For sinks, the flat ExportPosting list needs to become an entry graph. I’d make the base contract this:
@dataclass(frozen=True)
class ExportPosting:
account: str
amount_minor: int
currency: str
exponent: int
metadata: Mapping[str, str]
@dataclass(frozen=True)
class ExportEntry:
entry_id: str
value_date: date
flag: Literal["*", "!"]
payee: str | None
narration: str
links: tuple[str, ...] # case ids, source ids
tags: tuple[str, ...]
postings: tuple[ExportPosting, ...]
metadata: Mapping[str, str]
@dataclass(frozen=True)
class ExportBatch:
close_id: str
period: str
cutoff_record_time: datetime
entries: tuple[ExportEntry, ...]
The base class should validate before any sink writes:
- Every entry balances per currency unless the target sink explicitly supports Beancount-style cost lots and the entry uses them correctly.
- Decimal rendering uses
Decimal(amount_minor).scaleb(-exponent), never division. - Payee, narration, metadata keys, and metadata values are escaped by shared library code.
- Entry ordering is deterministic:
(value_date, entry_id). - Sinks are write-once per close; a prior-period correction is a new entry in the new close, not an edit to the old file.
One missing edge case for the sink: negative postings and sign conventions. Connectors normalize sign from the source perspective, but exports need account-perspective postings. A refund event of -6900 USD should not produce a negative debit to sales unless the template says that is acceptable for the sink. The template compiler should normalize to explicit debit/credit direction after evaluating amount sign, and reject an entry where a “debit” expression yields a negative amount unless the template uses signed: true. That catches a large class of backwards-bank-feed errors before they hit Beancount.
For the final blueprint, I’d include a small “threat model” section because it explains why the design is so strict: we are defending against silent mutation, source restatement, nondeterministic matching, sign-convention drift, accidental PII commits, and sink divergence. That will make the repository feel opinionated instead of merely complicated.
The architecture, data contracts, and threat mitigations are completely solid.
Let's review the final consensus:
- Identity & Hash Derivations: Zero wall-clock or sequence numbers in identity. Everything is content-addressed from raw bytes up to
close_id. - 4-Stage Ingestion Promotion:
raw_captured->parsed->controls_verified->promoted. Mismatched control totals hard-block promotion; row parse errors require an explicit cryptographic waiver. - Sink & Accounting Correctness:
ExportEntrygraph with balanced postings per currency,Decimal.scaleb(-exponent)rendering, sign-convention enforcement, and write-once cryptographic close artifacts. - Threat Model: Explicit defenses against restatement drift, parallel join jitter, silent suspense leakage, float precision bugs, and PII leakage.
Let's align on the structure of our final repository blueprint artifact (which I will compile and submit with signal='result' on our closing turn):
Deliverable Structure:
- Executive Summary & Mission: Why OpenRecon exists (the missing evidence/matching layer between rails and ledgers) and what it replaces.
- Threat Model & Invariant Matrix: The 6 core financial/determinism failure modes and their exact technical mitigations.
- Full Repository Layout: Complete tree structure of the repository (
core/,rules/,connectors/,sinks/,cli/,fixtures/,tests/). - Core Data Models & DuckDB Schema: Python dataclasses and SQL DDL for events, cases, transitions, variances, control checks, and closes.
- Declarative Rule & Account DSL: Specification and syntax for
rules/*.yaml,rules.lock,openrecon.yaml, andaccounts.yaml. - CLI Specification & Human Operator Walkthrough: Step-by-step terminal walkthrough with exact command invocations and outputs.
- Canonical Fixture Suite & Test Harness: Detailed breakdown of the 4 gold-standard test fixtures.
- MVP Roadmap & Contribution Guide: Immediate milestone deliverables for v0.1.0 release.
Claude, Codex:
Review this outline. Are there any last structural sections or technical contracts you want included or emphasized before we produce the complete blueprint?
The outline is right on the technical spine and thin in exactly the places that decide whether anyone can actually use or contribute to this. Six additions, roughly in order of how much damage their absence causes.
1. The connector contract is missing, and it's the highest-traffic extension point in the repo. Sinks got a full frozen-dataclass spec. Connectors got a directory. But every contribution this project ever receives will be "here's an adapter for Adyen / Wise / Revolut / my regional bank's weird CSV" — and without a stated contract, each one will invent its own ordering, its own sign convention, and its own idea of record_time. Section 5 or its own section needs:
class Connector(Protocol):
name: str
version: str
def sniff(self, raw: bytes) -> bool: ...
def parse(self, raw: bytes, *, record_time: datetime) -> ParseResult: ...
def controls(self, raw: bytes) -> list[DeclaredControl]: ...
plus the prohibitions, which matter more than the signature: parse makes no network calls, reads no clock, and must be a pure function of (raw, record_time). And a conformance suite every new connector must pass — parse the same bytes twice, get identical raw_row_ids; shuffle nothing; declare controls or explicitly declare none. That's maybe 80 lines of test that turns "we accept adapter PRs" into something reviewable by someone who doesn't know the format.
2. Fixtures must be synthetic and generated, not anonymized. I proposed "an open corpus of anonymized fixtures" early on and I want to walk half of that back. Scrubbed real bank data is one missed field away from publishing somebody's IBAN, and nobody reviewing the PR can verify the scrub was complete — CAMT.053 has free-text <RmtInf> fields that routinely contain names and invoice references. Ship fixtures/<name>/generate.py, committed, deterministic, seeded, with CI regenerating and diffing against the checked-in files. The corpus keeps its value as a shared test target and stops being a liability. Where we want format realism that only a real file provides, use the sample files the standards bodies and vendors already publish under their own terms.
3. License, and the disclaimer that goes with a financial tool. Apache-2.0 rather than MIT, specifically for the patent grant — payments and reconciliation is a patent-dense area and a contributor patent grant is worth having. Then two things in the README that this category always gets asked: OpenRecon produces evidence, it does not constitute a control environment, and it is not accounting advice. A controller will ask "is this SOX-compliant?" The honest answer is that compliance is a property of their process; what we provide is a reproducible artifact that their process can point at. Say that plainly rather than letting people assume.
Related and less obvious: rule packs have a licensing question of their own. A rule pack encoding Stripe's published fee formula is fine. A rule pack that embeds SWIFT's MT940 field specification may not be redistributable, since that spec is licensed. Worth one paragraph in the contribution guide so a well-meaning contributor doesn't paste a spec table into a YAML file.
4. A non-goals section, load-bearing. Without it the issue tracker fills with "add a web UI" and "support 500M rows" within a month, and each one costs a maintainer a polite paragraph. State: not a ledger; not real-time (batch, run after settlement); single-writer, one workspace per legal entity, so multi-entity consolidation is out of scope; no UI in v0.1; and an honest row ceiling — tens of millions of events on a laptop, beyond that talk to us. Naming the ceiling is more credible than implying there isn't one.
5. Schema migration doesn't exist, and that deserves to be stated as a feature. Because workspace.duckdb is a cache, a schema change is workspace_version: 2 plus openrecon rebuild. No migration scripts, no ALTER TABLE, no half-migrated database in a support thread. That falls out of the cache decision for free, and if we don't write it down someone will helpfully add Alembic in month three.
6. Supply chain. Normalizers are Python plugins, so installing a rule pack means executing someone's code against your bank statements. v0.1 should ship first-party packs vendored in-repo, with no remote registry and no auto-install, and the threat model should say so — it's the one attack in that list where the attacker is a person rather than a data-quality accident.
One thing I'd promote within the existing outline: section 6 should show the actual output of openrecon cases explain, not just the invocation. That rendering — here is the payout, here are the 440 children, here is the −0.13, here is which rule matched each side and what evidence hash proves it — is the product. It's the thing someone screenshots into a Slack thread, and it's what a controller judges us on in the first thirty seconds. A blueprint that specifies the schema in full detail and leaves the one human-facing surface as a command name has its emphasis backwards.
And for the roadmap section, I'd replace a feature list with one falsifiable acceptance test for v0.1.0: the four fixtures produce byte-identical output on two machines with different core counts, twice each, and openrecon close verify 2026-03 reproduces its manifest after a full rm workspace.duckdb && openrecon rebuild. If that passes, every determinism claim in the threat model is demonstrated rather than asserted. If it doesn't, nothing else in the blueprint is true yet.
One open question I'd want the blueprint to answer rather than leave implicit: what happens on the second close of the same period? A controller closes March on April 5, then a restatement lands April 20 and they close March again. Do we forbid it, version it (2026-03.r2), or force the correction into April? We've said sinks are write-once per close, which implies the second close is a distinct close of the same period — but nothing yet says whether a period can hold more than one, and the answer changes what close verify means.
The outline should add the connector contract as its own section, not bury it under DSL. It is the biggest contribution surface, and it is where determinism will be lost first.
I’d specify this contract:
class Connector(Protocol):
name: str
version: str
supported_kinds: tuple[str, ...]
def sniff(self, raw: bytes) -> bool: ...
def parse(self, raw: bytes, *, record_time: datetime) -> ParseResult: ...
def declared_controls(self, raw: bytes) -> tuple[DeclaredControl, ...]: ...
Hard rules:
parseis pure: no network, no filesystem except the provided bytes, no clock reads, no random IDs.- Output order must follow input order unless the source has a declared ordering field, in which case the connector must document it and sort explicitly.
- Sign convention must be declared in connector docs and tested with at least one debit and one credit.
- A connector with no control totals must explicitly return
declared_controls=(), so “none available” is a conscious state rather than a forgotten implementation.
The conformance test for every connector should be generic: parse the same bytes twice under the same record_time, assert identical canonical JSON and row IDs; parse under a different record_time, assert only record_time changes; run control checks; run a sign-fixture test. That makes adapter PRs reviewable.
I also agree the fixture corpus should be generated, not anonymized. The repo should commit both fixtures/<name>/generate.py and the generated raw files. CI runs the generator with a fixed seed and fails if the checked-in fixture changes unexpectedly. For vendor realism, link to permissively usable sample files or keep tiny hand-authored format fragments. Do not accept “scrubbed production CAMT” in PRs.
On second closes: allow them, but make them immutable revisions. Forbid overwriting a close. A restatement discovered on April 20 can produce a new March close manifest, but it does not rewrite the April 5 export.
CREATE TABLE close_manifests (
close_id VARCHAR PRIMARY KEY,
period VARCHAR NOT NULL, -- 2026-03
revision_of VARCHAR, -- previous close_id, nullable
discovery_record_time TIMESTAMPTZ NOT NULL,
cutoff_record_time TIMESTAMPTZ NOT NULL,
archive_digest VARCHAR NOT NULL,
rules_lock_digest VARCHAR NOT NULL,
accounts_digest VARCHAR NOT NULL,
export_digest VARCHAR NOT NULL
);
CREATE TABLE close_publications (
publication_id VARCHAR PRIMARY KEY,
close_id VARCHAR NOT NULL,
period VARCHAR NOT NULL,
status VARCHAR NOT NULL, -- draft | accepted | superseded
actor VARCHAR NOT NULL,
reason VARCHAR NOT NULL,
record_time TIMESTAMPTZ NOT NULL
);
openrecon close verify 2026-03 should verify the latest accepted publication by default and accept --close-id for an exact historical artifact. openrecon close list 2026-03 shows r1, r2, and their reasons. The revision label can be presentation-only, derived by ordering accepted publications for a period; it does not need to be part of identity.
For exports, I’d use this rule:
- If the revised close is before ledger publication, the user may export the new close as the period file they actually book.
- If the prior close was already published, the new close emits a correcting export batch dated in the current operational period, with metadata
corrects_close_id=<old>andcorrected_period=2026-03.
That fits write-once sinks and preserves the audit trail. It also avoids pretending the tool knows whether the company is allowed to revise issued financials. OpenRecon can produce the evidence and the correcting entries; the accounting policy decision stays with the user.
The final blueprint should include a sample openrecon cases explain output. Something like:
CASE rc_a3f19c payout_batch variance_pending
Source: stripe payout po_123
As of: 2026-04-05T00:00:00Z
Evidence
settlement: po_123 +12,400.13 USD event ev_91bd...
children: 440 rows +12,400.00 USD digest b2:7ac0...
residual: -0.13 USD
Rules
grouped_by_payout_id stripe.payout_batch@1.0.3 matched
clearing_zero failed: expected 0, actual -13 minor units
Next actions
openrecon cases explain rc_a3f19c --show-children
openrecon variance accept --case rc_a3f19c --amount 13 --currency USD --account stripe_default.variance_rounding --reason <text>
That output is more important than the SQL in the README. Users will trust the project when explain makes an accounting break obvious.
Add the non-goals and licensing sections exactly as load-bearing docs: Apache-2.0, not accounting advice, not SOX compliance by itself, not a ledger, not real-time, no web UI in v0.1, no remote plugin registry in v0.1, workspace DuckDB is a rebuildable cache with no migrations. For supply chain, first-party rule packs only until the project has signing, review policy, and a sandbox story for normalizers. Python plugins reading bank statements are too sensitive to auto-install from arbitrary Git refs.
The precision here is outstanding. The explicit Connector protocol, pure parsing rules, generated fixture seed harness, immutable close revisions, and terminal explain UX complete the core design.
Let's push into two final operational capabilities before we assemble the repository specification:
1. Self-Contained Audit Bundles (openrecon bundle audit)
When an external auditor (or tax authority) reviews a closed period, they will not run our internal development workflow. They need a self-contained, cryptographically verifiable artifact that proves financial integrity offline.
I propose a dedicated subcommand:
openrecon bundle audit --period 2026-03 --out dist/audit-2026-03.zip
The resulting zip archive contains:
manifest.json: Root hash and recursive BLAKE2b checksums of every file in the bundle.rules.lock&accounts.yaml: The exact frozen accounting rules and mapping specs.raw/: Content-addressed raw input files (blake2b=<id>.raw).cases.jsonl&postings.beancount: The complete case graph and ledger export.close_manifest.json: The cryptographic signature and publication record.standalone_verifier.py: A zero-dependency script that loads the raw files, runs the pinned DuckDB matching CTEs, checks all invariants, and outputs a binaryPASS / FAILwith zero external network access.
2. Performance Budget & Resource Bounds (v0.1.0 SLA)
To ensure OpenRecon remains fast enough for interactive CLI usage on a standard developer laptop (e.g. 4-8 cores, 16GB RAM):
- Raw Ingestion: >= 50,000 rows/sec per connector using streaming Python iterators + pyarrow parquet writer.
- Match Compilation & Execution: <= 250ms for a 10,000-child payout batch join in DuckDB.
- Full Period Close & Export: <= 3.0s total runtime for a 100,000-event monthly ledger close.
- Memory Ceiling: <= 512MB RSS regardless of dataset size by maintaining chunked parquet streaming into DuckDB.
Claude, Codex:
Does this audit bundle structure and performance SLA align with your expectations for the v0.1.0 MVP?
The audit bundle is the right instinct and two of its claims aren't true as written. The SLA has one number that's flatly wrong for the format we care most about.
"Zero-dependency standalone_verifier.py" that "runs the pinned DuckDB matching CTEs"
Those two clauses contradict each other. Running the CTEs needs DuckDB, and re-deriving events from raw/ needs the connectors, which means the verifier is the whole application with a different entry point. An auditor who unzips this, reads "zero-dependency," runs it, and gets ModuleNotFoundError: duckdb has just learned something about how carefully we check our claims.
Split it into two levels and name them honestly:
verify --level=integrity— genuinely stdlib-only. Recompute the merkle manifest, confirm every entry balances per currency, confirm each case's stated residual equals the arithmetic over its own postings, confirm each accepted variance exactly offsets the residual it claims to. That catches tampering and arithmetic fraud, needshashlibandjson, and runs on any Python 3.11 an auditor already has. This is the one an auditor will actually run.verify --level=replay— re-parsesraw/and re-runs matching. Requires a pinnedduckdb==X.Y.Zwith a recorded wheel hash. Worth shipping, worth not calling zero-dependency.
The bundle ships PII to a third party, by default, silently
We spent a whole section on "recon doctor hard-fails if git is tracking archive/" and then made the default audit command zip the same bytes and hand them to an outside firm. An auditor is often entitled to source documents, so this isn't wrong — but it must be a decision, not a default. --include-raw should be required, and the command should print what it's about to include: how many files, which sources, and a warning naming the fields those formats carry (CAMT.053's <RmtInf> is free text and routinely holds customer names and invoice references). Without the flag, bundle the digests and let the auditor request the bytes separately.
close_manifest.json doesn't contain a signature, and shouldn't claim to
The bundle spec says "the cryptographic signature and publication record," but nothing in this design signs anything. BLAKE2b proves the bundle wasn't corrupted in transit. It proves nothing about who produced it, and it's no defense against the one adversary an audit bundle exists to consider — the person who made the bundle. Anyone who can edit the numbers can recompute the hashes.
A real signature means key management: where the key lives, who holds it, what happens when the controller leaves. That's a subsystem, not a field, and it does not belong in v0.1. So drop the word. The bundle is content-addressed and reproducible, which is a strong and true claim. If a user wants attestation, let them detach-sign the manifest with minisign or age — tools that already solved key handling — and record the signature file alongside. We should never invent our own.
The bundle is missing its most audit-relevant contents
cases.jsonl and postings.beancount tell an auditor what we booked. The first question they actually ask is what we didn't book and who authorized that. So the bundle needs control_checks, quarantine_waivers with actor and reason, accepted_variances with actor and reason, and the ingest manifests showing which files reached promoted and which were refused. Plus the openrecon version and every connector version — without those, --level=replay can't be reproduced two years later, which is exactly when someone will try.
And the manifest needs to pin its own reproduction rules: files sorted by path, digest algorithm and length stated explicitly, so a verifier written in another language reaches the same root hash. "Recursive BLAKE2b checksums" underdetermines the result — BLAKE2b has a variable digest length.
The SLA
50,000 rows/sec per connector is wrong for XML. For CSV through pyarrow it's conservative. For CAMT.053, where each <Ntry> wraps <NtryDtls><TxDtls> with nested amount, party, and remittance subtrees, lxml.iterparse in Python realistically lands in the low thousands of entries per second, and I'd expect 2,000–5,000 on a laptop. Making that a stated SLA means we either miss it publicly or quietly stop counting XML as a connector. Set it per format: CSV/JSONL ≥ 50k rows/sec, XML formats ≥ 3k entries/sec, and revise both once a benchmark exists.
512MB RSS "regardless of dataset size" fights the 250ms join. DuckDB honors SET memory_limit='512MB' by spilling hash joins to a temp directory — so the memory bound is achievable, but the latency target isn't, once you're spilling. State it as a pair: memory bounded by an explicit configured memory_limit with a named spill directory, and the latency targets holding up to a stated event count, above which you're trading time for the memory ceiling.
And these are targets, not an SLA, until something measures them. An SLA in the v0.1 README with no benchmark behind it is a claim that decays silently. What I'd commit instead: a benchmarks/ directory with a fixture-generated dataset at a stated scale, a runner that prints the four numbers, and a CI job that fails on a regression beyond a band. Then the README quotes measured numbers with the machine spec next to them, and they mean something.
The number that's missing is the one users feel. Throughput and close time are batch concerns that run once a month while someone gets coffee. The controller's actual experience is cases list, then cases explain rc_…, then explain --show-children, over and over for an hour. If explain re-derives the evidence graph and takes three seconds, the tool feels broken no matter how fast the close was. Budget the interactive path separately and tightly: cases list and cases explain under 200ms on a warm workspace, which likely means the case residual and evidence digest are materialized at match time rather than computed per-invocation.
One thing I'd add to the bundle that costs nothing: a README.txt at its root, addressed to a human who has never heard of this project, saying what the bundle is, what command produced it, and the two verify invocations. Every audit bundle format I've seen that omitted that got opened, not understood, and forwarded back with a request for a PDF.
The audit bundle should be included, but the wording has to stay honest: two verification modes, raw inclusion explicit, no claims of built-in signatures.
I’d spec the command like this:
openrecon bundle audit --period 2026-03 --out dist/openrecon-audit-2026-03.zip
openrecon bundle audit --period 2026-03 --include-raw --out dist/openrecon-audit-2026-03-with-raw.zip
Without --include-raw, the bundle contains digests, manifests, case graph, controls, variances, waivers, exports, and verifier metadata. With --include-raw, it also includes source bytes and prints a preflight warning:
This bundle will include 12 raw files from sources: stripe_balance_transactions, mercury_operating.
Raw CAMT/CSV files may contain customer names, account numbers, IBANs, invoice references, and free-text remittance fields.
Continue only if the recipient is authorized to receive source financial records.
Bundle layout:
audit-2026-03/
README.txt
manifest.json
close_manifest.json
rules.lock
accounts.yaml
openrecon.yaml
cases.jsonl
case_links.jsonl
case_state_transitions.jsonl
control_checks.jsonl
ingest_manifests.jsonl
quarantine_waivers.jsonl
accepted_variances.jsonl
postings.beancount
verifier/
verify_integrity.py
requirements-replay.txt
wheels.lock
raw/ # only with --include-raw
manifest.json should define the hash procedure so another implementation can reproduce it:
{
"manifest_version": 1,
"digest": { "algorithm": "blake2b", "digest_size_bytes": 32, "encoding": "hex" },
"path_order": "bytewise_utf8_ascending",
"line_endings": "lf",
"root_digest": "...",
"files": [
{ "path": "cases.jsonl", "size": 9182, "digest": "..." }
]
}
Call the stdlib script verify_integrity.py, not standalone_verifier.py, because it verifies the artifact rather than replaying the full engine. It checks file digests, per-entry balance, residual arithmetic, accepted variance offsets, close manifest references, and that every non-closed case is represented in cases.jsonl. Replay verification can be a normal CLI command or a packaged environment:
python verifier/verify_integrity.py manifest.json
openrecon bundle verify --level replay audit-2026-03/ --require-wheel-hashes
The performance section should be “benchmarked targets,” not an SLA. v0.1 should ship benchmarks/ with generated datasets and the README should print measured numbers with a machine profile. Proposed targets:
CSV/JSONL ingest: >= 50,000 rows/sec on generated 1M-row fixture
CAMT.053/MT940 XML ingest: >= 3,000 entries/sec using streaming parse
10k-child payout match: <= 250ms when workspace fits configured memory limit
100k-event monthly close: <= 3s on reference laptop, excluding raw XML parse
cases list warm query: <= 200ms for 100k events / 5k cases
cases explain warm query: <= 200ms without --show-children, <= 750ms with 10k children
RSS: bounded by DuckDB memory_limit plus Python streaming buffers
The interactive budgets are worth emphasizing. A controller will tolerate a slow close more readily than a slow investigation loop. To hit those budgets, match should materialize a case_summaries table and an evidence_rollups table during the batch run:
CREATE TABLE case_summaries (
case_id VARCHAR PRIMARY KEY,
state VARCHAR NOT NULL,
residual_minor BIGINT,
currency CHAR(3),
exponent SMALLINT,
evidence_digest VARCHAR NOT NULL,
last_record_time TIMESTAMPTZ NOT NULL
);
CREATE TABLE evidence_rollups (
case_id VARCHAR PRIMARY KEY,
settlement_count BIGINT NOT NULL,
child_count BIGINT NOT NULL,
matched_count BIGINT NOT NULL,
quarantined_count BIGINT NOT NULL,
accepted_variance_count BIGINT NOT NULL,
digest VARCHAR NOT NULL
);
cases explain --show-children can still page through raw children, but the default explain should not recompute the world.
One last technical contract for the blueprint: redaction for human reports. The audit bundle with raw files is one mode, but cases explain, reports/closes/*, and GitHub issue examples should default to redacted metadata. Bank narrations and remittance fields need a shared redactor with deterministic tokens:
ACME CUSTOMER 1234 -> party_7f31
IBAN DE89370400440532013000 -> iban_2b91
invoice INV-88213 -> invoice_53ad
Use keyed HMAC or workspace-local salt stored outside git, not raw hashes, so common IBANs cannot be dictionary-attacked from public issue text. That belongs in the threat model next to PII leakage and audit bundles.
The audit bundle specification, benchmark targets, and HMAC-based PII redaction layer are completely aligned. The materialized case_summaries and evidence_rollups tables are essential for keeping cases list and cases explain snappy on 100k+ event ledgers.
Let's explore one final critical developer experience aspect: Rule Debugging & CTE Introspection.
When a developer or finance engineer is authoring or debugging a custom rule (e.g. rules/adyen/2.1.0/ic_plus_plus.yaml) and finds that 12 out of 500 settlement batches fail to match:
- Running raw SQL joins against DuckDB manually is tedious and loses the context of canonical IDs and evidence rollups.
- If
openrecon matchis a black box, users will be tempted to widen tolerances dangerously.
I propose a dedicated rule introspection tool:
openrecon debug rule <rule_id> --case <case_id_or_external_id>
This command:
- Executes the rule's compiled DuckDB CTEs with intermediate inspection hooks.
- Prints a structured trace showing:
- Candidate Selection: Which rows matched the parent and child predicates.
- Join Evaluation: Which candidate children were paired with the parent.
- Invariant Assertion Breakdown: For each failed invariant, shows the exact calculated vs expected values (e.g.,
amount_match: sum(children) = 10,400.12 USD, parent = 10,400.15 USD, diff = -0.03 USD). - Dropped/Competing Candidates: Identifies if any child row was claimed by a competing higher-rank rule or fell outside the settlement timestamp window.
Claude, Codex:
How should the rule compiler structure intermediate CTE views to enable this introspection without introducing runtime overhead during normal openrecon match execution?
The framing of the question contains the trap: "without introducing runtime overhead during normal match" invites a compiler with a debug=True branch that emits different SQL. Do that and debug output can disagree with what match actually did — the money version of "it works in the debugger." Someone will spend a day trusting a trace that describes a query the engine never ran.
One compiled artifact, filtered, never a variant. The compiler emits a fixed chain of named CTEs per rule:
WITH left_candidates AS (…),
right_candidates AS (…),
pairs AS (…), -- cartesian within the grouping key
scored AS (…), -- per-predicate deltas, computed always
ranked AS (…), -- rank_by, deterministic tie-break
selected AS (SELECT * FROM ranked WHERE rk = 1 AND all_predicates_pass)
SELECT * FROM selected;
Normal match reads selected. Debug runs the same text with a source-level predicate bound (WHERE natural_key = ?) and reads each stage in turn. DuckDB pushes that filter down through the chain, so debugging one case touches roughly one case's worth of rows. The overhead in normal mode is zero because CTEs are inlined, not materialized — no stage is written anywhere unless debug asks for it.
The guarantee needs a test, not a convention: assert that the CTE body hash is byte-identical between the two invocations, and add a golden test asserting debug's selected rows equal the persisted matches for the same case. If the compiler ever grows a debug branch, that test dies.
Notice that scored computes deltas unconditionally. That's deliberate. You need the amount delta and the time delta to rank candidates anyway, so keeping them in the always-on path costs nothing and means the near-miss information exists whether or not anyone asked for it.
The output that matters isn't the failed invariant
"12 of 500 batches fail to match" — knowing which invariant failed is nearly useless, because the answer is always "the one that failed." What a finance engineer needs is the near-miss: what was the closest candidate, and by how much, along each dimension separately.
UNMATCHED parent mercury:credit ev_4c1a… +10,400.15 USD 2026-03-14T09:02Z
best candidates (rule bank_ach_1to1@2.1.0)
rank event amount_delta time_delta key verdict
1 ev_9f02… -0.03 USD +71h — OUT_OF_WINDOW (tolerance 48h)
2 ev_31bd… -0.03 USD +73h — OUT_OF_WINDOW
3 ev_77ac… +412.00 USD +2h — AMOUNT_MISMATCH
12 of 500 parents share this shape: amount within 5 minor units, time 68-76h.
That last line is the actual finding. A cluster at 68–76 hours against a 48-hour window doesn't mean the tolerance is too tight; it means this bank posts on T+3 and the rule was written for T+1. You cannot see that from "invariant failed," and you can see it immediately from a delta histogram.
Which brings up the guardrail, because visibility alone makes the tolerance problem worse
Show someone that 12 batches would match at 96 hours and they will set the window to 96 hours. The number that stops a bad change is the blast radius, and it comes from the same query with the relaxed predicate:
$ openrecon debug rule bank_ach_1to1 --relax window=96h --dry-run
newly matched: 12 parents
newly ambiguous: 47 parents now have >1 rank-1 candidate ← this is the cost
newly reclaimed: 3 children currently claimed by adyen_settlement@1.4.0
47 new ambiguities is a much clearer "no" than any warning text we could write. And --dry-run should emit a proposed rule diff, never apply one, so the workflow lands where the existing machinery already is: debug → proposed diff → version bump → replay --from-lock --to-lock → review the case diff → merge. Debug becomes a PR generator rather than a footgun.
Two gaps this exposes
Rejections are never persisted. "Dropped/competing candidates" was in the requirements from the first mention of the case evidence graph, but no table ever landed for it, so debug would have to recompute — and a recomputation under a newer rule pack won't describe what happened at match time. Write them during match:
CREATE TABLE match_rejections (
rejection_id VARCHAR PRIMARY KEY, -- blake2b(rule_id || rule_pack_digest || candidate_digest)
rule_id VARCHAR NOT NULL,
parent_event_id VARCHAR NOT NULL,
candidate_event_id VARCHAR NOT NULL,
reason_code VARCHAR NOT NULL, -- LOST_RANK | OUT_OF_WINDOW | AMOUNT_MISMATCH | ALREADY_CLAIMED
amount_delta_minor BIGINT,
time_delta_seconds BIGINT,
rank INTEGER,
record_time TIMESTAMPTZ NOT NULL
);
Bounded to the top k=5 per unmatched parent, configurable. Unbounded it's O(n²) on the pairs relation and will dwarf every other table in the workspace. That's a real ceiling and the docs should say so: rejections are a debugging aid with a fixed depth, not a complete record of everything the engine considered.
Rule ordering is currently implicit, and that's a determinism bug in the same family as the parallel hash join. "Already claimed by a competing higher-rank rule" presumes rules run in a defined order with earlier claims winning — but nothing has specified where that order comes from. If it's directory iteration, it differs between NTFS and ext4, which means the same workspace matches differently on the developer's laptop and the CI container. rules.lock needs an explicit integer priority per rule, resolved at lock time and part of the lock digest. Ties on priority are an error at lock time, not a coin flip at match time.
One compiled artifact is the right rule, and I’d make it enforceable at the API level: the compiler returns a CompiledRule object with named stages and one SQL string. match and debug are only different consumers of that object.
@dataclass(frozen=True)
class CompiledRule:
rule_id: str
rule_pack_digest: str
priority: int
sql: str
stages: tuple[str, ...] # left_candidates, right_candidates, pairs, scored, ranked, selected
cte_body_digest: str
Normal execution:
WITH ...same ctes...
INSERT INTO matches
SELECT ... FROM selected;
Debug execution:
WITH ...same ctes...
SELECT * FROM scored WHERE parent_event_id = ? ORDER BY rank LIMIT ?;
Same sql, same cte_body_digest, different terminal statement. The golden test should assert both that digest equality holds and that debug selected --case X matches persisted match_members for X under the same rules.lock.
I’d be careful with “scored computes deltas unconditionally costs nothing.” It costs nothing for amount and time deltas because ranking needs them. It is not true for heavier diagnostics like normalized counterparty similarity, regex captures, or tokenized remittance comparison. The DSL should distinguish score_fields from debug_fields:
rank_by:
- abs(amount_delta_minor)
- abs(time_delta_seconds)
- event_id
debug_fields:
- normalized_counterparty
- remittance_tokens
rank_by fields are always computed. debug_fields can be projected by the same CTE only when the rule already computes them for predicates, or produced by a separate explainer function over the bounded candidate set. That avoids making every production match pay for verbose human diagnostics.
Persisting top-k rejections is worth it, but the table needs to tie to the rule-pack digest and case as well as parent/candidate. I’d use:
CREATE TABLE match_rejections (
rejection_id VARCHAR PRIMARY KEY,
case_id VARCHAR,
rule_id VARCHAR NOT NULL,
rule_pack_digest VARCHAR NOT NULL,
parent_event_id VARCHAR NOT NULL,
candidate_event_id VARCHAR NOT NULL,
candidate_digest VARCHAR NOT NULL,
reason_code VARCHAR NOT NULL,
amount_delta_minor BIGINT,
currency CHAR(3),
exponent SMALLINT,
time_delta_seconds BIGINT,
rank INTEGER NOT NULL,
record_time TIMESTAMPTZ NOT NULL
);
rejection_id should exclude record_time, same determinism rule as everywhere else:
blake2b(rule_id || rule_pack_digest || parent_event_id || candidate_event_id || reason_code || rank)
Top-k should be default 5, max maybe 50, and recorded in rules.lock or openrecon.yaml so debug depth is reproducible. If a user asks for deeper diagnostics, they run openrecon debug rule ... --recompute --top-k 100 and the output is explicitly marked non-persisted.
Rule priority absolutely belongs in rules.lock, and ties should fail lock generation. I’d also add a claim ledger so “already claimed” is not inferred from final matches:
CREATE TABLE event_claims (
claim_id VARCHAR PRIMARY KEY,
event_id VARCHAR NOT NULL,
case_id VARCHAR NOT NULL,
match_id VARCHAR NOT NULL,
rule_id VARCHAR NOT NULL,
rule_pack_digest VARCHAR NOT NULL,
priority INTEGER NOT NULL,
claim_scope VARCHAR NOT NULL, -- exclusive | shared
record_time TIMESTAMPTZ NOT NULL
);
Some events must be shared. A charge can belong to its original sale case and later be linked to a dispute case. A payout child should probably be exclusive within payout matching. If every claim is globally exclusive, lifecycle links become a fight against the matcher. The rule spec should declare claim_scope and claim_namespace, for example stripe_payout_membership vs stripe_dispute_lifecycle.
Debug should show three views, not one wall of output:
openrecon debug rule stripe.payout_batch --case rc_a3f19c --stage scored
openrecon debug rule stripe.payout_batch --case rc_a3f19c --near-misses
openrecon debug rule bank_ach_1to1 --relax window=96h --dry-run
The --relax path must never write matches. It should produce a proposed YAML diff plus a blast-radius table:
proposed diff: rules/bank_csv/1.0.0/ach_1to1.yaml
newly matched: 12
newly ambiguous: 47
exclusive claim conflicts: 3
cases changing state: 9 open -> matched, 3 variance_pending -> closed
That turns debugging into a rules PR workflow instead of a production mutation. The final blueprint should include this because it explains how users extend rules safely without widening tolerances blind.
Every major subsystem of OpenRecon has now been designed and stress-tested:
- Scope & Position: Ledger-agnostic evidence, matching, and case-resolution engine in Python + DuckDB.
- Deterministic Identity: Universal content-addressed BLAKE2b hashing; zero wall-clock/sequence numbers.
- Deterministic Matching: Single compiled CTE pipeline with deterministic tie-breaker sorting, explicit priority ordering, and scoped event claims (
exclusivevsshared). - 4-Stage Ingestion:
raw_captured->parsed->controls_verified->promoted, with quarantined rows and cryptographic waivers. - Two-Level Accounting: Event-level posting templates generating balanced entries per currency, and case-level closure invariant rules.
- Immutable Close Lineage: Cryptographic close manifests (
close_manifests), immutable revision publications (close_publications), and correcting delta exports for published periods. - Offline Audit Bundles: Self-contained zip packages with
verify_integrity.pyand HMAC-keyed PII redaction for reports and public issues. - Rule Debugging & Safety:
CompiledRuleCTE introspection, persisted top-k rejections, and dry-run blast radius analysis (--relax --dry-run).
As we head toward the final turn to publish the deliverable, let's do a final review of the core data schema and API contracts to ensure zero inconsistencies.
Claude, Codex:
Are there any remaining edge cases or ambiguities in the data models or contracts before I write and publish the complete GitHub repository specification as the final deliverable?
Yes — six, and the first two would produce wrong numbers in production, not just confusion.
1. There is no read rule for superseded events, which means restatement doesn't actually work yet
We put supersedes_id on events in the second message and never wrote down how to read past it. case_events is keyed (case_id, event_id, role), so when Stripe amends a balance transaction, both the original and the amendment are members of the case. Every sum() over that case now double-counts. The 13¢ residual becomes a 12,400.13 residual.
The missing rule, which belongs in the blueprint as its own paragraph because every query depends on it:
A case's evidence set at time T is, for each supersession chain, the single event with the greatest
record_time ≤ Tthat is not superseded by another event withrecord_time ≤ T. Every aggregate — residual, rollup, closure invariant, export — reads that set, never the raw membership.
In practice that's one view (events_as_of) that every rule and every rollup selects from, and a lint that fails if any compiled SQL references the events table directly. Without the lint someone writes FROM events in a closure rule in month two and nobody notices until a restatement lands.
2. Nobody has mentioned timezones, and ExportEntry.value_date is a date
value_time is TIMESTAMPTZ. Beancount dates are naive calendar dates. Converting one to the other requires a timezone, and if we default to UTC we've made a silent policy decision that's wrong for most users: a charge at 2026-03-31T23:30Z books to March for a UTC company, to March 31 local for a New York company — and to April 1 for one in Auckland. Period boundaries are exactly where this bites, which means it bites at close, every month, on the transactions that matter most.
So: reporting_timezone is a required field in openrecon.yaml, it has no default, and it's an input to the digest that the close pins. And --cutoff 2026-04-05T00:00:00Z should probably accept a local time and record the resolved instant, because a controller thinks in their own calendar and will get the UTC offset wrong at least once.
3. evidence_digest means three different things and is spelled one way
It appears on reconciliation_cases, matches, case_state_transitions, accepted_variances, control_checks, and as digest on evidence_rollups. On the immutable case row it cannot mean "digest of the current evidence set" — the case row never changes and evidence keeps arriving, so it's stale the moment the second file lands. Rename that one opening_evidence_digest and define the other five explicitly, or verify_integrity.py will compare things that were never the same function.
4. case_summaries and evidence_rollups are mutable tables in an append-only design
That's fine — they're materializations for the 200ms interactive budget. But they must be declared derived: rebuilt by openrecon rebuild, never an input to any content hash, and specifically evidence_rollups.digest must not be what the audit bundle verifies against. If the verifier checks a cache, we've made the cache load-bearing and the "delete workspace.duckdb freely" guarantee is gone.
5. currency CHAR(3) contradicts the reason we added exponent
The exponent column exists partly because USDC needs 6 decimal places. USDC is not three characters. Neither is any Beancount commodity like AAPL. Either widen to VARCHAR with a validation rule, or state plainly in non-goals that v0.1 is ISO-4217 only and drop the crypto justification for exponent (JPY at 0 and KWD at 3 justify it on their own). Right now the schema says one thing and the rationale says another.
6. Rule priority inside a rule pack makes multi-pack installs unlockable
Ties fail lock generation — correct. But if priority is declared inside a pack, then a Stripe pack and an Adyen pack written by different people will both pick 100, and every workspace that installs both fails to lock with no recourse except forking a vendored file. Priority is a property of the workspace's rule composition, not of a rule. Put it in openrecon.yaml as an ordered list of pack names, derive per-rule integers from that ordering at lock time, and let a pack declare only intra-pack ordering.
Two smaller ones worth a line each in the spec rather than a section:
accepted_variances.account_key resolves through accounts.yaml, which changes. The close pins accounts_digest, so a re-export of March resolves correctly — but only if resolution always happens against the close's pinned accounts file rather than the working-tree one. Say that explicitly; the obvious implementation reads the file on disk.
A bundle built without --include-raw cannot be replay-verified, since replay re-parses raw bytes. openrecon bundle verify --level replay against a digest-only bundle should fail with that exact message rather than a missing-file traceback, and README.txt should say which level the bundle it sits in supports.
One genuine ambiguity I'd rather flag than paper over: case_id = blake2b(case_type || source || natural_key) assumes one source per case, but the CAMT.053 fixture is inherently two-sided — a bank line and an internal ACH batch. It works if source means "the anchoring source the rule declares," and it breaks if anyone reads it as "where this case's evidence came from." Whichever we mean, the blueprint should say it in the field comment, because the second reading is the more natural one and it produces a case id that changes depending on which side got ingested first.
Those six should all be folded into the final schema before it is written. The restatement read rule is the biggest one: if queries do not consistently use an as-of current event set, append-only history turns into double-counting.
I’d define one canonical view and require every generated rule to target it:
CREATE VIEW events_as_of AS
SELECT e.*
FROM events e
WHERE e.record_time <= current_setting('openrecon.as_of')::TIMESTAMPTZ
AND NOT EXISTS (
SELECT 1
FROM events newer
WHERE newer.supersedes_id = e.event_id
AND newer.record_time <= current_setting('openrecon.as_of')::TIMESTAMPTZ
);
If DuckDB session settings are awkward, make it a macro/table function instead:
SELECT * FROM events_as_of(TIMESTAMPTZ '2026-04-05T00:00:00Z')
The compiler should reject rule SQL containing FROM events or JOIN events outside the implementation of that view/function. Same for closure rules and export queries. That lint is not optional.
On timezone: make reporting_timezone required in openrecon.yaml, and use IANA names only:
entity:
name: MyCo US, Inc.
reporting_timezone: America/New_York
operating_currency: USD
ExportEntry.value_date is derived by converting value_time into reporting_timezone. Period close commands should allow local input:
openrecon close --period 2026-03 --cutoff-local 2026-04-05T17:00:00
The manifest records both:
{
"cutoff_local": "2026-04-05T17:00:00",
"reporting_timezone": "America/New_York",
"cutoff_record_time_utc": "2026-04-05T21:00:00Z"
}
That prevents the recurring “UTC close date” error.
Rename the digests so they say what they mean:
reconciliation_cases.opening_evidence_digest
matches.match_evidence_digest
case_state_transitions.transition_evidence_digest
accepted_variances.variance_evidence_digest
control_checks.control_evidence_digest
evidence_rollups.cache_digest
case_summaries and evidence_rollups are derived cache tables. The blueprint should state that they are excluded from close identity, audit identity, and replay verification. They can be rebuilt and compared as a performance check, not as audit evidence.
For currency, I’d choose ISO-4217 only in v0.1. Use currency VARCHAR NOT NULL plus a validator rather than CHAR(3), because it costs nothing and avoids schema churn, but document that v0.1 accepts ISO-4217 codes only. exponent still matters for JPY, KWD, BHD, and CLF. Crypto and securities can be a later commodity model with explicit non-goal status.
Rule ordering should be composition-level, not pack-level. The pack declares internal order; the workspace declares pack order:
rule_order:
- stripe
- adyen
- bank_csv
rules.lock expands that to resolved priorities:
rules:
- id: stripe.payout_batch
pack: stripe
priority: 1000
intra_pack_order: 10
- id: adyen.settlement_batch
pack: adyen
priority: 2000
intra_pack_order: 10
Ties in the resolved lock fail. Users can reorder packs without forking a third-party rule file, and the lock digest captures the decision.
For accepted_variances.account_key, resolution must always happen against the close-pinned accounts_digest. The stored variance keeps the symbolic key plus the resolved account at close time:
account_key VARCHAR NOT NULL,
resolved_account VARCHAR NOT NULL,
accounts_digest VARCHAR NOT NULL
If the current working accounts.yaml changes, close verify --close-id old still resolves against the archived close bundle, not the live file.
For digest-only audit bundles, verify --level replay should fail cleanly:
FAIL: replay verification requires raw source files. This bundle was created without --include-raw.
Supported level: integrity
On case_id, define source as the anchoring source chosen by the rule, not all evidence sources. I’d rename it to avoid misreading:
anchor_source VARCHAR NOT NULL,
natural_key VARCHAR NOT NULL,
case_id = blake2b(case_type || anchor_source || natural_key)
For a CAMT bank-vs-internal ACH case, the rule decides the anchor. If the bank statement line has a stable end-to-end ID, use anchor_source=bank and that ID. If not, use a deterministic synthetic natural key from the rule’s selected anchor side: date, amount, currency, statement id, ordinal, and source file digest. Do not let ingestion order decide it.
I’d add one final acceptance-test clause: ingest the same two-sided fixture in both orders, bank then internal and internal then bank, and assert identical case IDs, states, and exports. That catches the anchor ambiguity immediately.
Every single refinement has been accepted and incorporated:
events_as_ofMacro Enforcement: All matching, closure, and export queries MUST read fromevents_as_of(as_of_timestamp)to prevent restatement double-counting; rawFROM eventsqueries outside this macro are rejected by the rule compiler.- Deterministic Timezones:
reporting_timezoneis mandatory (e.g.America/New_York), andopenrecon closerecordscutoff_local,reporting_timezone, andcutoff_record_time_utc. - Explicit Digest Naming: Distinct columns for
opening_evidence_digest,match_evidence_digest,transition_evidence_digest,variance_evidence_digest,control_evidence_digest, andcache_digest(with cache tables explicitly excluded from audit/close identity). - Anchor Invariance:
case_id = blake2b(case_type || anchor_source || natural_key)with explicit anchor selection by the rule, and bidirectional ingestion order invariance tests (bank then internalvsinternal then bank). - Historical Account Pinning:
accepted_variancesrecordsaccount_key,resolved_account, andaccounts_digestso future account renames never mutate historical closes. - Integrity vs Replay Verification:
openrecon bundle verifycleanly distinguishes--level integrity(hash/arithmetic check) from--level replay(which requires--include-raw).
Our alignment is 100% complete across architecture, threat models, data schemas, CLI surface, and testing.
On my next (and final) turn, I will publish the complete, buildable OpenRecon Repository Specification & Blueprint with signal='result', covering:
- Problem domain & threat model
- Full directory layout and module responsibilities
- Complete DuckDB SQL DDL and Python dataclasses
- Connector protocol & rules DSL
- CLI syntax & operator workflows
- Generated fixture suite & CI test harnesses
- Contribution guide & v0.1.0 roadmap
Claude, Codex: Stand by for the final deliverable.
Before you write it, one thing nobody in this room has said once, and I think it's the most useful contribution left: we have designed roughly nine months of work and called it v0.1.
Count the subsystems: four-stage ingest with control verification, a rule compiler with lint enforcement, a claim ledger with scopes and namespaces, bitemporal read semantics, immutable close revisions with publication states, audit bundles with two verification levels, HMAC redaction, rejection persistence, blast-radius analysis, a benchmark suite with CI gates. Every one earned its place in the conversation. Collectively they describe a product a funded team ships in a year.
If the spec presents all of it as the plan, the repo reads as vaporware on day one — an enormous README with openrecon/ containing three stub files. That's the specific way ambitious open-source projects die: not from a bad design, from a design nobody can start.
So I'd ask the blueprint to mark every subsystem with a shipping tier, and to make v0.1 embarrassingly small:
| Subsystem | Tier |
|---|---|
events, matches, cases, case_events, case_state_transitions |
v0.1 |
| Stripe balance-transaction CSV connector | v0.1 |
| Beancount sink, write-once | v0.1 |
Content-addressed ids, events_as_of, frozen clock |
v0.1 |
init / ingest / rebuild / match / cases list / cases explain / export / check |
v0.1 |
| Fixture 1 + golden harness + the two-machine determinism test | v0.1 |
variance accept, accepted_variances |
v0.2 |
| Control totals, quarantine, waivers | v0.2 |
| Bank CSV connector, CAMT.053 | v0.2 |
close, close verify, close revisions |
v0.3 |
replay, rules.lock, rule packs as distributable units |
v0.3 |
debug rule, rejections, --relax --dry-run |
v0.4 |
| Audit bundles, redaction, benchmarks-as-gate | v0.5 |
The test is whether a competent person can ship the v0.1 row in a few focused weeks. That row can, and it already demonstrates the thesis — ingest a real payout, find the 13¢, explain it, export the rest. Everything below it is credible because the top row exists, and worthless without it. The design work we did isn't wasted by the cut; it's what stops v0.1 from making a decision that v0.3 can't live with. That's the entire value of designing ahead.
One mechanical suggestion: put the tier in the spec as a column, and mirror it in the repo as GitHub milestones with the same names. Then the roadmap section is a link rather than prose that goes stale in a month.
The disagreement that survived, since the result is supposed to name one:
controls.auto_accept_variance_minor is configurable. I argued in my first message that auto-accepting variance under a threshold is how a 13¢ bug becomes a $130,000 bug, and the compromise reached — default 0, and changing it bumps the rule-pack version so old runs still reproduce — makes the change auditable but doesn't make it hard. Someone under close pressure at 9pm on April 5 will set it to 500 and every rounding break in the period disappears silently, reproducibly, with a version bump nobody reads.
I'd still cut the knob entirely for v0.1 and force variance accept per case. What would settle it is evidence rather than argument: ship without it, and see whether real users file an issue asking for it. If ten batches a month each need a manual 13¢ acceptance, they'll say so and they'll be right. If nobody asks, we never build the footgun. Absence is cheap to reverse; a threshold that's been in a release for a year is not.
And a second, smaller one I'll own rather than pretend is settled: this room added a subsystem on every single turn and never once removed one. That's a real property of how we worked, and the spec would be more honest for saying that the design is deliberately maximal and the tier table is where it gets disciplined.
Last practical note. The README's first screen decides whether anyone reads the rest, and right now the blueprint's section 1 is "Executive Summary & Mission." What belongs above the fold instead:
- One sentence on what it is — reconciles payment-processor settlements against your bank and your ledger, and shows its work.
- The three commands from
init --with-fixturethroughcases list, with the real terminal output showingvariance_pending -0.13 USD. - One sentence on what it is not: not a ledger, not accounting advice, not real-time.
- The
explainoutput, in full.
The threat model and the DDL are what make an evaluator trust the project, but only after the first screen has made them want to. Put them at position 2 and 4 in the document and they'll never be reached.
The tiering correction is necessary. The design is valuable, but if the blueprint presents all of it as v0.1, it will look like an architecture essay instead of a repo someone can clone and improve.
I’d make the final deliverable explicitly two-layered:
- Repository blueprint: the long-lived design constraints that prevent painting ourselves into a corner.
- Shipping plan: the smallest milestone that proves the tool works.
For v0.1, I’d cut even harder than the proposed table in one place: no close, no variance accept, no bank connector, no audit bundle, no rule-pack distribution. The first release should do one thing end-to-end:
openrecon init --with-fixture stripe_payout_with_variance
openrecon ingest stripe fixtures/stripe_payout_with_variance/input/stripe_balance_transactions.csv
openrecon rebuild
openrecon match
openrecon cases list
openrecon cases explain rc_a3f19c
openrecon export beancount --out exports/stripe.beancount
openrecon check
Expected behavior:
CASE TYPE STATE RESIDUAL
rc_a3f19c payout_batch variance_pending -0.13 USD
export should export only closed/matched entries and leave the variance case marked unresolved. If that means the demo does not produce a perfect month close, that is fine. The thesis is “find and explain the break,” not “finish the accounting workflow.” A first release that refuses to hide 13 cents is more credible than one that includes a premature acceptance mechanism.
So I agree with removing controls.auto_accept_variance_minor entirely for v0.1. I’d go further: leave auto-accept out of the core schema until real usage proves it belongs. Manual variance accept in v0.2 is enough. If a future version adds auto-accept, it should be a rule that creates explicit accepted_variances rows with actor=policy:<policy_id>, never a config knob that makes the case vanish.
The v0.1 schema can be small while preserving compatibility:
events
matches
match_members
reconciliation_cases
case_events
case_state_transitions
case_summaries # derived cache
evidence_rollups # derived cache
Include events_as_of, stable IDs, required reporting_timezone, currency VARCHAR with ISO-4217 validation, and generated fixture tests now. Those are foundations. Defer the tables for controls, waivers, closes, accepted variances, rejections, event claims, bundles, and publications until their commands exist.
The README first screen should be product proof, not positioning. Suggested top:
# OpenRecon
OpenRecon reconciles payment-processor settlements against ledger-ready postings and shows its work.
```bash
openrecon init --with-fixture stripe_payout_with_variance
openrecon match
openrecon cases list
CASE TYPE STATE RESIDUAL
rc_a3f19c payout_batch variance_pending -0.13 USD
OpenRecon is not a ledger, not accounting advice, and not real-time infrastructure.
Then show `cases explain` in full before the threat model. The DDL can come after users understand what the tool does.
For the facilitator’s final result, I’d explicitly name the scope discipline as an agreed correction: “The design is intentionally broader than v0.1; the roadmap tiers prevent the initial repo from becoming vaporware.” That is not a footnote. It is the difference between an open-source project and a whitepaper.
OpenRecon: Open-Source Financial Reconciliation & Evidence Engine
Repository Specification & Architectural Blueprint
Executive Summary & Problem Domain
Modern payment processors (Stripe, Adyen), bank statement feeds (CAMT.053, MT940, CSV), and internal transactional databases record financial movements in incompatible shapes, granularities, and timeframes. Existing open-source tools either solve basic personal budgeting (Actual Budget, Firefly III) or high-frequency trading simulation (LEAN), leaving a gaping void for developer-first, deterministic settlement reconciliation and financial invariant auditing.
OpenRecon provides the missing ingestion, matching, case-resolution, and cryptographic evidence layer between transaction rails and general ledgers (Beancount, plain SQL, TigerBeetle). It operates as a local-first Python library and CLI backed by embedded DuckDB.
Agreement & Architectural Consensus
- Scope & Separation of Concerns: OpenRecon is an evidence and reconciliation engine, not a ledger. It ingests settlement feeds, compiles declarative matching rules to DuckDB SQL, models exceptions as Reconciliation Cases, and emits balanced, immutable entries to pluggable accounting sinks.
- Deterministic Identity: Zero autoincrement sequence numbers, zero wall-clock timestamps in hashes. All identifiers are content-addressed BLAKE2b digests:
source_file_id = blake2b(raw_bytes) raw_row_id = blake2b(source_file_id || ordinal || raw_row_bytes) event_id = blake2b(raw_row_id || normalizer_id || normalizer_code_hash || canonical_event_json) case_id = blake2b(case_type || anchor_source || natural_key) match_id = blake2b(rule_id || rule_pack_digest || sorted(event_ids) || match_shape) close_id = blake2b(period || cutoff_record_time_utc || archive_digest || rules_lock_digest || accounts_digest || export_digest) - Bitemporality & Restatement Safety: Every event carries
value_time(economic occurrence) andrecord_time(system acquisition). All matching, closure, and export operations execute strictly against theevents_as_of(cutoff_record_time)macro/view, preventing restatement double-counting. - Two-Level Accounting Rule:
event_posting_templates: Generate balanced accounting entries per canonical child event (charge, fee, refund, dispute, reversal).case_closure_rules: Assert clearing balance invariants over the case graph (sum(clearing) == 0).
- Scope Discipline & Tiered Shipping Plan:
- v0.1.0 (Core Engine & Golden Path Demo): Single-rail Stripe CSV ingest, 1:N payout batch matching,
events_as_ofview, case state machine,openrecon cases explain, Beancount export, and synthetic golden-file tests. Auto-accept variance is strictly excluded. - v0.2.0 (Accounting Controls & Close Lineage):
openrecon variance accept,openrecon closewith cryptographic manifests, and correcting delta exports. - v0.3.0 (Multi-Rail Ingest & Quarantine): Bank CSV/CAMT.053 connectors, 4-stage ingestion promotion (
raw_captured -> parsed -> controls_verified -> promoted), and quarantine isolation. - v0.4.0 (Compliance & Bundling): Offline audit bundles (
openrecon bundle audit), keyed HMAC PII redaction, and dry-run rule relaxation diffs.
- v0.1.0 (Core Engine & Golden Path Demo): Single-rail Stripe CSV ingest, 1:N payout batch matching,
Threat Model & Defensive Invariant Matrix
| Threat / Failure Mode | Root Cause in Conventional Systems | OpenRecon Architectural Defense |
|---|---|---|
| Restatement Drift | Mutating database rows when a processor issues an amended balance transaction. | Append-only event store with supersedes_id. All reads query events_as_of(record_time). |
| Non-Deterministic Matching | Multi-threaded parallel hash joins in DuckDB yielding unstable candidate order across machines. | Explicit deterministic tie-breaker sorting: ORDER BY (value_time, external_id, event_id) with explicit rank. |
| Silent Suspense Leakage | Sweeping small rounding differences into automatic suspense accounts without human oversight. | Strict hard-fail on non-zero clearing residuals. Exceptions remain open as variance_pending cases. |
| Float Precision Corruption | Performing financial arithmetic on floating-point data types. | Integer minor units (amount_minor) + currency exponent. Formatting uses Decimal(amount_minor).scaleb(-exponent). |
| Sign-Convention Inversion | Backward debits/credits resulting from ambiguous source CSV definitions. | Source-agnostic sign normalization in connectors + template validation rejecting negative debits. |
| PII & Data Leakage | Exporting unredacted bank statement narrations to issue trackers and audit packages. | Keyed HMAC redaction for terminal outputs and optional --include-raw flag with preflight authorization warnings. |
Repository Layout
openrecon/
├── pyproject.toml
├── openrecon.yaml
├── accounts.yaml
├── rules.lock
├── openrecon/
│ ├── __init__.py
│ ├── cli/
│ │ ├── __init__.py
│ │ ├── main.py
│ │ ├── cases.py
│ │ ├── explain.py
│ │ ├── export.py
│ │ ├── match.py
│ │ └── verify.py
│ ├── core/
│ │ ├── __init__.py
│ │ ├── events.py
│ │ ├── cases.py
│ │ ├── hashing.py
│ │ ├── db.py
│ │ └── types.py
│ ├── connectors/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── stripe_csv.py
│ │ └── bank_csv.py
│ ├── rules/
│ │ ├── __init__.py
│ │ ├── compiler.py
│ │ ├── engine.py
│ │ └── resolver.py
│ ├── sinks/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── beancount.py
│ └── redaction/
│ ├── __init__.py
│ └── hmac_mask.py
├── fixtures/
│ ├── stripe_payout_with_variance/
│ │ ├── generate.py
│ │ ├── input/
│ │ ├── expected_cases.json
│ │ └── expected_postings.beancount
│ ├── chargeback_lifecycle_3stage/
│ ├── cross_currency_eur_usd/
│ └── camt053_bank_vs_ledger/
└── tests/
├── conftest.py
├── test_connectors.py
├── test_compiler.py
├── test_determinism.py
├── test_fixtures.py
└── test_sinks.py
Core Data Models & DuckDB DDL
Database Schema (openrecon/core/db.py)
-- Canonical Bitemporal Events
CREATE TABLE events (
event_id VARCHAR PRIMARY KEY, -- blake2b(raw_row_id || normalizer_id || code_hash || canonical_json)
source_file_id VARCHAR NOT NULL, -- blake2b(raw_bytes)
raw_row_id VARCHAR NOT NULL,
source VARCHAR NOT NULL, -- stripe | mercury | adyen
external_id VARCHAR, -- bt_1P..., ch_3M...
kind VARCHAR NOT NULL, -- charge | fee | refund | dispute | payout | reversal
amount_minor BIGINT NOT NULL,
currency VARCHAR NOT NULL, -- ISO-4217 code
exponent SMALLINT NOT NULL, -- 2 for USD/EUR, 0 for JPY, 3 for KWD
value_time TIMESTAMPTZ NOT NULL, -- Economic occurrence
record_time TIMESTAMPTZ NOT NULL, -- Ingestion / system acquisition
supersedes_id VARCHAR, -- Restatement linkage
normalizer_id VARCHAR NOT NULL,
code_hash VARCHAR NOT NULL
);
-- Canonical As-Of View
CREATE VIEW events_as_of AS
SELECT e.*
FROM events e
WHERE e.record_time <= current_setting('openrecon.as_of')::TIMESTAMPTZ
AND NOT EXISTS (
SELECT 1
FROM events newer
WHERE newer.supersedes_id = e.event_id
AND newer.record_time <= current_setting('openrecon.as_of')::TIMESTAMPTZ
);
-- Reconciliation Cases
CREATE TABLE reconciliation_cases (
case_id VARCHAR PRIMARY KEY, -- blake2b(case_type || anchor_source || natural_key)
case_type VARCHAR NOT NULL, -- payout_batch | dispute | bank_transfer
anchor_source VARCHAR NOT NULL,
natural_key VARCHAR NOT NULL, -- e.g. po_123, dp_456
currency VARCHAR NOT NULL,
exponent SMALLINT NOT NULL,
opened_record_time TIMESTAMPTZ NOT NULL,
rule_pack_digest VARCHAR NOT NULL,
opening_evidence_digest VARCHAR NOT NULL
);
CREATE TABLE case_events (
case_id VARCHAR NOT NULL,
event_id VARCHAR NOT NULL,
role VARCHAR NOT NULL, -- settlement | child | fee | dispute | reversal
record_time TIMESTAMPTZ NOT NULL,
PRIMARY KEY (case_id, event_id, role)
);
CREATE TABLE matches (
match_id VARCHAR PRIMARY KEY, -- blake2b(rule_id || rule_pack_digest || sorted_event_ids || match_shape)
rule_id VARCHAR NOT NULL,
rule_pack_digest VARCHAR NOT NULL,
reason_code VARCHAR NOT NULL, -- EXACT_ID | FEE_TOLERANCE | BATCH_SUM
match_evidence_digest VARCHAR NOT NULL,
record_time TIMESTAMPTZ NOT NULL,
supersedes_match_id VARCHAR
);
CREATE TABLE match_members (
match_id VARCHAR NOT NULL,
event_id VARCHAR NOT NULL,
side VARCHAR NOT NULL, -- internal | external
PRIMARY KEY (match_id, event_id)
);
CREATE TABLE case_state_transitions (
transition_id VARCHAR PRIMARY KEY,
case_id VARCHAR NOT NULL,
from_state VARCHAR,
to_state VARCHAR NOT NULL, -- open | matched | variance_pending | exported | closed
reason_code VARCHAR NOT NULL,
actor VARCHAR NOT NULL, -- system | user:<id>
transition_evidence_digest VARCHAR NOT NULL,
record_time TIMESTAMPTZ NOT NULL
);
-- Materialized Query Acceleration (Excluded from Audit & Close Digests)
CREATE TABLE case_summaries (
case_id VARCHAR PRIMARY KEY,
state VARCHAR NOT NULL,
residual_minor BIGINT NOT NULL,
currency VARCHAR NOT NULL,
exponent SMALLINT NOT NULL,
cache_digest VARCHAR NOT NULL,
last_record_time TIMESTAMPTZ NOT NULL
);
Core Python Domain Models (openrecon/core/types.py)
from dataclasses import dataclass
from datetime import date, datetime
from typing import Mapping, Sequence, Literal
@dataclass(frozen=True)
class CanonicalEvent:
event_id: str
source_file_id: str
raw_row_id: str
source: str
external_id: str | None
kind: str
amount_minor: int
currency: str
exponent: int
value_time: datetime
record_time: datetime
supersedes_id: str | None
normalizer_id: str
code_hash: str
@dataclass(frozen=True)
class ExportPosting:
account: str
amount_minor: int
currency: str
exponent: int
metadata: Mapping[str, str]
@dataclass(frozen=True)
class ExportEntry:
entry_id: str
value_date: date
flag: Literal["*", "!"]
payee: str | None
narration: str
links: tuple[str, ...]
tags: tuple[str, ...]
postings: tuple[ExportPosting, ...]
metadata: Mapping[str, str]
@dataclass(frozen=True)
class ExportBatch:
period: str
cutoff_record_time_utc: datetime
reporting_timezone: str
entries: tuple[ExportEntry, ...]
Configuration & Rules DSL Specification
Workspace Configuration (openrecon.yaml)
workspace_version: 1
entity:
name: "Acme SaaS US, Inc."
reporting_timezone: "America/New_York"
operating_currency: "USD"
duckdb:
path: "workspace.duckdb"
memory_limit: "512MB"
threads: 4
rule_order:
- stripe
- bank_csv
sources:
stripe_balance_transactions:
connector: stripe.csv
rule_pack: stripe
account_profile: stripe_default
mercury_checking:
connector: bank_csv
rule_pack: bank_csv
account_profile: mercury_operating
sinks:
beancount:
enabled: true
out_dir: exports/beancount
Declarative Rule Specification (rules/stripe/1.0.0/payout_batch.yaml)
rule_id: stripe.payout_batch
version: "1.0.0"
target_case_type: payout_batch
description: "Matches a Stripe payout settlement deposit to child balance transaction events"
predicates:
anchor:
source: stripe
kind: payout
children:
source: stripe
kinds: [charge, refund, dispute, fee, adjustment]
join_key: payout_id
rank_by:
- abs(amount_delta_minor)
- abs(time_delta_seconds)
- event_id
closure_asserts:
- name: clearing_zero
expression: "sum(children.amount_minor) == anchor.amount_minor"
Operator Walkthrough & Terminal UX
1. Initialize Workspace with Benchmark Fixture
$ openrecon init --with-fixture stripe_payout_with_variance
Initialized OpenRecon workspace in ./myco-recon
Loaded fixture: stripe_payout_with_variance (217 charges, 4 refunds, 1 dispute fee, 1 payout)
2. Execute Matching & Review Open Cases
$ openrecon match
Compiling rules from rules.lock (digest: 8b2f91a...)...
Executing match pipeline against events_as_of(2026-04-05T00:00:00Z)...
Matched 222 events across 1 case.
$ openrecon cases list
CASE TYPE STATE RESIDUAL LAST_EVENT
rc_a3f19c payout_batch variance_pending -0.13 USD 2026-03-31T23:59:00Z
3. Inspect the Evidence Graph
$ openrecon cases explain rc_a3f19c
================================================================================
CASE rc_a3f19c [payout_batch] STATE: variance_pending
Anchor: Stripe Payout po_123 (value_time: 2026-03-31 17:00:00 America/New_York)
================================================================================
Evidence Summary:
Settlement (po_123): +12,400.13 USD (event: ev_91bd2a...)
Children (222 events): +12,400.00 USD (digest: b2:7ac019...)
------------------------------------------------------------------------------
Residual Imbalance: -0.13 USD (Clearing account does not close to 0)
Closure Rule Invariants:
[PASS] rule_id_match: stripe.payout_batch@1.0.0
[FAIL] clearing_zero: expected 0, actual -13 minor units
Next Actions:
$ openrecon cases explain rc_a3f19c --show-children
$ openrecon variance accept --case rc_a3f19c --amount 13 --currency USD --account stripe_default.variance_rounding --reason "processor rounding diff"
4. Export Ledger Entries
$ openrecon export beancount --out exports/2026-03.beancount
Exported 222 balanced postings to exports/2026-03.beancount.
Notice: Case rc_a3f19c remains in variance_pending state; exported with flag '!'.
Canonical Fixture Suite
| Fixture Directory | Scenario & Edge Case Tested | Verification Criteria |
|---|---|---|
stripe_payout_with_variance |
217 charges, 4 refunds, 1 dispute fee, 13¢ rounding residual. | Identifies 1 variance_pending case with exact -0.13 USD residual. |
chargeback_lifecycle_3stage |
Day 0 charge ($100), Day 20 dispute + fee ($100+$15), Day 45 win + partial reversal. | Preserves Month 1 closed books; generates delta postings in Month 2. |
cross_currency_eur_usd |
EUR 100 customer charge, USD 108.50 settlement, USD 2.10 fee. | Bridges multi-currency legs through explicit Assets:FXClearing account. |
camt053_bank_vs_ledger |
CAMT.053 XML statement paired with internal ACH ledger. | Ingestion order invariance: identical cases regardless of load sequence. |
Implementation Milestones & Roadmap
- v0.1.0 (MVP Foundation):
- Core Python engine, DuckDB client, and schema DDL.
- Pure
StripeCSVConnectorwith streaming CSV parser. - YAML rule compiler for 1:N batch join queries.
openrecon init,match,cases list,cases explain,export beancount.- Pytest golden harness with
stripe_payout_with_variance.
- v0.2.0 (Audit & Lineage):
reconciliation_casesclosure state machine.openrecon variance acceptandaccepted_variancestable.openrecon closewith cryptographicclose_manifest.json.
- v0.3.0 (Connectors & Controls):
- Bank CSV & CAMT.053 XML streaming parser.
- 4-stage ingestion promotion pipeline and quarantine file routing.
- Bidirectional ingestion order invariance tests.
- v0.4.0 (Audit Bundles & DX):
openrecon bundle auditzip generator withverify_integrity.py.- HMAC keyed metadata redactor for terminal logs and bug reports.
openrecon debug rule --relax --dry-runblast-radius analyzer.
License & Project Governance
- License: Apache-2.0
- Disclaimer: OpenRecon is an operational evidence tool and does not constitute formal accounting advice or autonomous SOX certification.