Skip to main content
GoldenMatch uses YAML config files with Pydantic validation. Every section is optional — GoldenMatch auto-configures what you leave out.

Full YAML reference

Matchkeys

Three matchkey types:

Transforms

Applied to field values before scoring. Refdata transforms are auto-prepended by the controller when a column name matches the relevant pattern AND its profiled col_type agrees. See Reference Data.

Scorers

Cross-encoder reranking

Add rerank: true to a weighted matchkey to re-score borderline pairs with a cross-encoder model:

Blocking

Set auto_select: true to auto-pick the best blocking key by histogram analysis. Set auto_suggest: true to get blocking suggestions when no keys are specified.

Golden rules

Eight merge strategies for building canonical records: Set a default strategy and override per field:

Lock-step field groups

By default GoldenMatch picks the best value for each column independently. That can produce “Frankenstein” records where, say, a street address comes from one source while the city and zip come from another. field_groups prevents this by pinning a set of columns to a single winning source record.
The winner is chosen by the group’s strategy applied across the group’s columns as a unit. All columns in the group then take their values from that one record — including that record’s nulls. There is no column-by-column fallback within a group (strict lock-step). Two constraints the config loader enforces:
  • Columns must be disjoint across groups (a column can belong to at most one group).
  • A column in a field_groups entry cannot also have its own field_rules entry.
Auto-detection (optional, default off). Set field_group_detection: true in golden_rules, or export GOLDENMATCH_FIELD_GROUP_SURVIVORSHIP=1, to let auto-config propose groups. The heuristic is conservative — it targets well-known correlated patterns (mailing address, person name, contact block) and optionally uses a GoldenCheck domain pack for infermap-fed detection. Explicitly declared field_groups are always honored regardless of whether the flag is set.
Correlated survivorship resolves on the driver (the in-memory golden builder). The Ray Phase-5 streaming pipeline (GOLDENMATCH_DISTRIBUTED_PIPELINE=2) and the Sail backend reject configs that include field_groups or conditional field_rules with a clear error. Run those configs on the in-memory path.
Vectorized resolution (automatic). The in-memory golden builder resolves field_groups, scalar fields, allow_fill, anchor, and conditional field_rules through a vectorized Polars path when building plain golden records (no per-field provenance requested). It is selected automatically when every strategy in the config is one the vectorized path reproduces exactly; any config using a custom:, confidence_majority, majority_vote, or unanimous_or_null strategy — or one requesting per-field provenance — falls back to the per-cluster path. Output is byte-identical either way (values and confidences); the only difference is speed. No configuration is required. The richer per-field source provenance surfaced by save_lineage / explain --cluster is built on the per-cluster path.

Surfacing the group decision

When a config uses field_groups (or conditional/validated field_rules), GoldenMatch surfaces the survivorship decision in three places:
  • Lineage JSON (save_lineage / goldenmatch lineage --output-dir ...): a top-level golden_records array, one entry per cluster, each with a groups array (name, columns, winner_row_id, winner_source, strategy, tie, confidence) and a human-readable audit string. Example: street, city, zip promoted together from record 7 via most_complete (group 'mailing_address').
  • CLI goldenmatch explain --cluster <id>: the output gains a Survivorship: block with the same audit lines (group promotions and any conditional/validated field notes).
  • MCP lineage tool: the result includes the same golden_records section so agents can read which columns were promoted lock-step and from which source.
For plain configs with no field_groups and no conditional field_rules, output is byte-identical to prior behavior — none of these surfaces add anything.

anchor strategy and allow_fill

strategy: anchor selects the winning record by first restricting candidates to those where the named anchor column is non-null, then applying most_complete within that subset. If no record has the anchor present, the group falls back to plain most_complete. Use this when one column (such as a plan or product ID) anchors the rest of the group’s fields.
allow_fill: true relaxes strict lock-step for the winning record’s null cells. After the winner is selected, each null cell in the winner is back-filled from the best other record (chosen by the group’s strategy) that has a value for that cell. The lineage audit records a <group>: <col> back-filled from record N line for every filled cell, and the filled map in the group provenance tracks which row supplied each value. Off by default; when off, the winner’s nulls are kept as-is.

Conditional and validated field rules

A field_rules entry can be a list of clauses instead of a single strategy. The first clause whose when: predicate is satisfied wins; the last clause must have no when: and acts as the default.
when: predicate syntax. The predicate is a safe boolean mini-language, not Python eval. Supported operators: == != < <= > >= in not in and or not. Names resolve to already-resolved golden fields. Literals: list/tuple, numeric, string, boolean. Unsupported: function calls, attribute access, indexing, arithmetic. If a predicate references a field that is absent or None for a given cluster, the predicate is treated as unsatisfied (that clause is skipped) and evaluation continues to the next clause. validate:. An optional candidate filter applied before the merge strategy. Candidates that fail the validator are dropped from consideration before the strategy runs (fail-open: an unknown validator name drops nothing). Only nanp / phone are friendly aliases; every other validator is referenced by its full GoldenFlow transform name: Lineage. The audit trail records how each field’s value was chosen. For a field group the entry reads like street, city, state, zip promoted together from record 7 via most_complete (group 'mailing_address'). For a conditional rule it reads like phone used most_recent because state in ['CA', 'NY'].

Standardization

Map column names to standardizer functions:

Validation

Rule types: regex, min_length, max_length, not_null, in_set, format. Actions: flag (mark but keep), "null" (set to null — quote it so YAML keeps the string; bare null parses as None and fails validation), quarantine (remove from matching).

Environment variables

The table below covers the security/sandbox variables most relevant to a YAML config. For the complete, verified reference of every GOLDENMATCH_* variable — the native gate, backend selection, the distributed pipeline, perf opt-ins, and when to set or avoid each — see Tuning & opt-ins.

Path sandbox

GOLDENMATCH_ALLOWED_ROOT activates an opt-in path containment layer for deployed or shared instances. When set, safe_path() resolves every incoming file argument with Path.resolve(), checks for NUL bytes, and then asserts the resolved path falls under the root. Paths that fail this check are rejected before any I/O occurs.
Unset (the default) means no containment check — any accessible path is valid. This is the correct default for a local-first desktop tool.
The Railway goldenmatch-mcp service should set GOLDENMATCH_ALLOWED_ROOT=/data to scope the MCP tools to the mounted data volume. See MCP Server.

Settings persistence

  • Global: ~/.goldenmatch/settings.yaml — output mode, default model, API keys
  • Project: .goldenmatch.yaml — column mappings, thresholds, blocking config
Settings tuned in the TUI can be saved to the project file. Next run picks them up automatically.

Programmatic config

Or auto-generate from data:

Verification (v1.5.0)

auto_configure_df runs preflight at the end of config generation — 6 checks that auto-repair missing domain-extracted columns, drop useless-cardinality exact matchkeys, flag oversized blocks, demote remote-asset scorers, and cap low-confidence weights. Unrepairable issues raise ConfigValidationError; the full report is attached to the exception as err.report. The pipeline runs postflight after scoring and before clustering — 4 signals (score histogram + bimodality, blocking recall, cluster sizes + bottleneck pairs, threshold-band overlap) that can auto-nudge the threshold on clear bimodality and attach the report to DedupeResult.postflight_report / MatchResult.postflight_report. Two new kwargs on auto_configure_df:
The preflight report is available on the returned config (underscore is private-by-convention but stable across v1.5.x):
See the Verification section in the Python API docs for the full preflight / postflight signatures and the PostflightSignals schema.

Learning Memory (v1.6.0)

The optional memory: section enables persistent corrections. Once a steward, agent, or LLM decides a pair, that decision is stored, re-anchored across row reorders by record_hash, and applied automatically on every subsequent dedupe_df / match_df call. After 10+ corrections accumulate against a matchkey, the learner adjusts that matchkey’s threshold for the next run. Off by default; enable via the YAML block above or config.memory.enabled = True. Full guide: Learning Memory.

Throughput config

The optional throughput: section (or the throughput= kwarg on dedupe_df) enables the sketch-then-verify near-duplicate tier. Off by default; enabling it does not affect any run where the kwarg is None. Full reference: Throughput tier.
ThroughputConfig is re-exported from the top level. The throughput= kwarg also accepts True (uses defaults), a float (sets recall_target), or None (off, byte-identical to today).