Full YAML reference
Matchkeys
Three matchkey types:Transforms
Applied to field values before scoring.
Refdata transforms are auto-prepended by the controller when a column name matches the relevant pattern AND its profiled
col_type agrees. See Reference Data.
Scorers
Cross-encoder reranking
Addrerank: true to a weighted matchkey to re-score borderline pairs with a cross-encoder model:
Blocking
Set
auto_select: true to auto-pick the best blocking key by histogram analysis. Set auto_suggest: true to get blocking suggestions when no keys are specified.
Golden rules
Eight merge strategies for building canonical records:
Set a default strategy and override per field:
Lock-step field groups
By default GoldenMatch picks the best value for each column independently. That can produce “Frankenstein” records where, say, a street address comes from one source while the city and zip come from another.field_groups prevents this by pinning a set of columns to a single winning source record.
strategy applied across the group’s columns as a unit. All columns in the group then take their values from that one record — including that record’s nulls. There is no column-by-column fallback within a group (strict lock-step).
Two constraints the config loader enforces:
- Columns must be disjoint across groups (a column can belong to at most one group).
- A column in a
field_groupsentry cannot also have its ownfield_rulesentry.
field_group_detection: true in golden_rules, or export GOLDENMATCH_FIELD_GROUP_SURVIVORSHIP=1, to let auto-config propose groups. The heuristic is conservative — it targets well-known correlated patterns (mailing address, person name, contact block) and optionally uses a GoldenCheck domain pack for infermap-fed detection. Explicitly declared field_groups are always honored regardless of whether the flag is set.
Correlated survivorship resolves on the driver (the in-memory golden builder). The Ray Phase-5 streaming pipeline (
GOLDENMATCH_DISTRIBUTED_PIPELINE=2) and the Sail backend reject configs that include field_groups or conditional field_rules with a clear error. Run those configs on the in-memory path.Vectorized resolution (automatic). The in-memory golden builder resolves
field_groups, scalar fields, allow_fill, anchor, and conditional field_rules through a vectorized Polars path when building plain golden records (no per-field provenance requested). It is selected automatically when every strategy in the config is one the vectorized path reproduces exactly; any config using a custom:, confidence_majority, majority_vote, or unanimous_or_null strategy — or one requesting per-field provenance — falls back to the per-cluster path. Output is byte-identical either way (values and confidences); the only difference is speed. No configuration is required. The richer per-field source provenance surfaced by save_lineage / explain --cluster is built on the per-cluster path.Surfacing the group decision
When a config usesfield_groups (or conditional/validated field_rules), GoldenMatch surfaces the survivorship decision in three places:
- Lineage JSON (
save_lineage/goldenmatch lineage --output-dir ...): a top-levelgolden_recordsarray, one entry per cluster, each with agroupsarray (name, columns, winner_row_id, winner_source, strategy, tie, confidence) and a human-readableauditstring. Example:street, city, zip promoted together from record 7 via most_complete (group 'mailing_address'). - CLI
goldenmatch explain --cluster <id>: the output gains aSurvivorship:block with the same audit lines (group promotions and any conditional/validated field notes). - MCP
lineagetool: the result includes the samegolden_recordssection so agents can read which columns were promoted lock-step and from which source.
field_groups and no conditional field_rules, output is byte-identical to prior behavior — none of these surfaces add anything.
anchor strategy and allow_fill
strategy: anchor selects the winning record by first restricting candidates to those where the named anchor column is non-null, then applying most_complete within that subset. If no record has the anchor present, the group falls back to plain most_complete. Use this when one column (such as a plan or product ID) anchors the rest of the group’s fields.
allow_fill: true relaxes strict lock-step for the winning record’s null cells. After the winner is selected, each null cell in the winner is back-filled from the best other record (chosen by the group’s strategy) that has a value for that cell. The lineage audit records a <group>: <col> back-filled from record N line for every filled cell, and the filled map in the group provenance tracks which row supplied each value. Off by default; when off, the winner’s nulls are kept as-is.
Conditional and validated field rules
Afield_rules entry can be a list of clauses instead of a single strategy. The first clause whose when: predicate is satisfied wins; the last clause must have no when: and acts as the default.
when: predicate syntax. The predicate is a safe boolean mini-language, not Python eval. Supported operators: == != < <= > >= in not in and or not. Names resolve to already-resolved golden fields. Literals: list/tuple, numeric, string, boolean. Unsupported: function calls, attribute access, indexing, arithmetic. If a predicate references a field that is absent or None for a given cluster, the predicate is treated as unsatisfied (that clause is skipped) and evaluation continues to the next clause.
validate:. An optional candidate filter applied before the merge strategy. Candidates that fail the validator are dropped from consideration before the strategy runs (fail-open: an unknown validator name drops nothing). Only nanp / phone are friendly aliases; every other validator is referenced by its full GoldenFlow transform name:
Lineage. The audit trail records how each field’s value was chosen. For a field group the entry reads like
street, city, state, zip promoted together from record 7 via most_complete (group 'mailing_address'). For a conditional rule it reads like phone used most_recent because state in ['CA', 'NY'].
Standardization
Map column names to standardizer functions:Validation
regex, min_length, max_length, not_null, in_set, format.
Actions: flag (mark but keep), "null" (set to null — quote it so YAML keeps the string; bare null parses as None and fails validation), quarantine (remove from matching).
Environment variables
The table below covers the security/sandbox variables most relevant to a YAML config. For the complete, verified reference of everyGOLDENMATCH_* variable — the native gate, backend selection, the distributed pipeline, perf opt-ins, and when to set or avoid each — see Tuning & opt-ins.
Path sandbox
GOLDENMATCH_ALLOWED_ROOT activates an opt-in path containment layer
for deployed or shared instances. When set, safe_path() resolves every
incoming file argument with Path.resolve(), checks for NUL bytes, and
then asserts the resolved path falls under the root. Paths that fail
this check are rejected before any I/O occurs.
The Railway
goldenmatch-mcp service should set
GOLDENMATCH_ALLOWED_ROOT=/data to scope the MCP tools to the mounted
data volume. See MCP Server.Settings persistence
- Global:
~/.goldenmatch/settings.yaml— output mode, default model, API keys - Project:
.goldenmatch.yaml— column mappings, thresholds, blocking config
Programmatic config
Verification (v1.5.0)
auto_configure_df runs preflight at the end of config generation — 6 checks that auto-repair missing domain-extracted columns, drop useless-cardinality exact matchkeys, flag oversized blocks, demote remote-asset scorers, and cap low-confidence weights. Unrepairable issues raise ConfigValidationError; the full report is attached to the exception as err.report.
The pipeline runs postflight after scoring and before clustering — 4 signals (score histogram + bimodality, blocking recall, cluster sizes + bottleneck pairs, threshold-band overlap) that can auto-nudge the threshold on clear bimodality and attach the report to DedupeResult.postflight_report / MatchResult.postflight_report.
Two new kwargs on auto_configure_df:
preflight / postflight signatures and the PostflightSignals schema.
Learning Memory (v1.6.0)
The optionalmemory: section enables persistent corrections. Once a steward, agent, or LLM decides a pair, that decision is stored, re-anchored across row reorders by record_hash, and applied automatically on every subsequent dedupe_df / match_df call. After 10+ corrections accumulate against a matchkey, the learner adjusts that matchkey’s threshold for the next run. Off by default; enable via the YAML block above or config.memory.enabled = True.
Full guide: Learning Memory.
Throughput config
The optionalthroughput: section (or the throughput= kwarg on dedupe_df) enables the
sketch-then-verify near-duplicate tier. Off by default; enabling it does not affect any run
where the kwarg is None. Full reference: Throughput tier.
ThroughputConfig is re-exported from the top level. The throughput= kwarg also accepts True
(uses defaults), a float (sets recall_target), or None (off, byte-identical to today).