Tuning & opt-ins - Golden Suite

This is the one place that lists every runtime knob GoldenMatch reads — native acceleration, backend selection, the distributed pipeline, the prep/scoring perf opt-ins, and the full GOLDENMATCH_* environment-variable table. For each, you get what it does, its default, and when to set it (or leave it alone). If you are tuning a large run and just want the answer, jump to Recommended setups. If something ran far slower than you expected, read Gotchas that cost a day first — every item there is a real footgun.

Most GoldenMatch knobs are environment variables, not config fields, because they switch a code path (native vs Python, in-memory vs distributed) rather than describe your data. Data-shape settings — matchkeys, blocking keys, golden rules — live in Configuration. This page is the operational layer on top of that.

Recommended setups

You almost never need to set anything for correctness — auto-config and the planner choose sane defaults. These blocks are about speed at scale and reproducibility, not behavior.

Local / interactive (up to ~500K rows)

Set nothing. Zero-config is the supported path. If you want the compiled kernels for a measurable speed-up on the scoring/clustering hot loops, install the optional native wheel:

pip install goldenmatch[native]

With the wheel present, the default GOLDENMATCH_NATIVE=auto already runs every component that has a kernel natively (native is the reference; pure-Python is the lossy fallback). You do not need to set GOLDENMATCH_NATIVE=1.

Single box at scale (1M–200M rows, 16-core / 64 GB class)

# Make a missing/broken native wheel a LOUD error instead of a silent slow fallback.
export GOLDENMATCH_NATIVE=1
# Turn on the Polars-native address-normalize fast path (skips the per-row Python plugin).
export GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE=1
# Reproducible runs: don't read cross-run auto-config memory.
export GOLDENMATCH_AUTOCONFIG_MEMORY=0
# These two are already ON by default; set them explicitly in a bench harness for the record.
export GOLDENMATCH_BUCKET_SLIM_PROJECTION=1
export GOLDENMATCH_GOLDEN_SLIM_MULTIDF=1
# Windows only: skip the Polars CPU probe that can hang at import.
# export POLARS_SKIP_CPU_CHECK=1

The planner picks backend=bucket for this lane automatically when the native kernel is enabled. This is the exact env used by the quality-invariant scale ladder (1K → 100M), so it is the validated single-box recipe.

Setting backend="ray" does NOT distribute the run. On its own it falls back to an in-memory path that materializes the whole frame on the driver. Real distribution needs GOLDENMATCH_DISTRIBUTED_PIPELINE=2 plus a Ray Dataset input — see Distributed pipeline and Gotchas. At single-box scale, prefer bucket.

Distributed (Ray cluster, 50M+ rows)

pip install goldenmatch[ray]
export GOLDENMATCH_DISTRIBUTED_PIPELINE=2     # Phase 5 streaming — the real distributed path
export GOLDENMATCH_ENABLE_DISTRIBUTED_RAY=1   # let the planner pick ray for backend selection
export GOLDENMATCH_NATIVE=1
# Multi-node only: a SHARED scratch the WCC checkpoints round-by-round (node-local breaks cross-node reads).
export GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH=gs://your-bucket/rc_scratch

Phase 5 has a hard contract: a Ray Dataset input with a global __row_id__ column, and it writes golden records to output_path rather than returning an in-memory clusters dict. The recall-complete path (block-shuffle scoring + randomized-contraction WCC) is default-on and was validated end-to-end at 100M rows on a real 5-node GCP cluster (9.2 min, 20M clusters recovered exactly, 0.36 GB driver RSS). See Distributed pipeline for the full contract. At single-box scale, prefer bucket — only reach for a Ray cluster when the data genuinely won’t fit one box.

Native acceleration

Rust is the reference implementation (as of the 2026-07 reference-mode change). The Rust/PyO3 kernel is the source of truth for GoldenMatch’s hot-path primitives; pure-Python is the lossy fallback used only when a native wheel isn’t installed (the wheel ships as pip install goldenmatch[native]). A central gate (core/_native_loader.py) selects per component. GOLDENMATCH_NATIVE (core/_native_loader.py):

Value	Behavior
`auto` (default, or unset)	Run native wherever the component’s kernel symbol exists on the wheel — native is the reference. Pure-Python is the fallback for a missing wheel/symbol.
`1`	Require native. Raises `RuntimeError` at the first kernel call if the wheel isn’t importable. Use this on scale/bench runs so a missing wheel fails loudly instead of running 100x+ slower in the Python fallback.
`0`	Force the pure-Python fallback everywhere. Use for parity debugging or if you suspect a kernel bug.

Under auto, native runs for every component that has a kernel symbol — clustering, block_scoring, pairs, featurize, hashing, autoconfig, sketch, pprl_bloom, and the multimodal perceptual hashes. Each cleared bit-/ULP-level parity against the Python reference, so the fallback is output-identical (golden-vector verified). (The former per-component allowlist, _GATED_ON, is retained in the loader as sign-off documentation but no longer governs auto.) The sketch component covers both MinHash/LSH (#1081) and SimHash-over-embeddings (#1090) band hashing (~29x faster; the single source of truth across Python/Rust/TS, so semantic blocking runs on the Rust sketch-core kernel by default). Fellegi-Sunter block scoring is native-authoritative by default as of goldenmatch 2.6.0 (GOLDENMATCH_FS_NATIVE=0 opts out to the reproducible numpy path); it is scoped to the probabilistic path only. The autoconfig component is the shared, pyo3-free decision core (goldenmatch-autoconfig-core) behind auto-config — the planner, column classifier (with the adaptive 1 − 1/√n identifier-cardinality floor), the sample→full pair-count extrapolation (the corrected ratio² + Chao1 estimate), and the adaptive decision thresholds (the sparse-match floor and the per-type exact-matchkey floor). It runs native under auto and is byte-identical to the pure-Python oracle (golden-vector parity across Python / native / wasm-TS). The same crate compiles to WebAssembly so the TypeScript surface consumes the identical decision logic — one source of truth, zero cross-surface drift. When to set =1: any large or benchmarked run where a silent Python fallback would be a disaster. The cost of a missing wheel is enormous (the scoring kernel is the dominant stage), and auto hides it. When to leave it auto: normal local use. If the wheel is installed you already get native; if it isn’t, you get working Python.

`GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE`

Default off. When set to 1, address-normalize transforms run as a Polars-native expression instead of the per-row Python refdata plugin. On address-heavy data at scale the per-row plugin is a real bottleneck in the prep stage; the native path removes it.

Set it (=1) when you have address columns and are running at scale. Output is parity-tested on representative inputs.
Leave it off for small runs, or if you hit an address-canonicalization edge case (it is opt-in precisely because one integration-level cluster-count drift hasn’t been fully root-caused).

`GOLDENMATCH_NATIVE_RAYON_MIN_PAIRS`

Default 20000000. Candidate-pair count per kernel call below which the native block-scorer runs in the calling thread instead of fanning out to rayon. This exists to avoid a futex-park stall observed on some 8-core Linux CPUs (issue #688). The Python caller already parallelizes across buckets, so the sequential path loses nothing on small calls. Leave it at the default unless you are reproducing #688; 0 forces rayon always, a huge value forces sequential always.

`GOLDENMATCH_NATIVE_SKETCH_RAYON_MIN_ROWS`

Default 10000. Row count below which the native MinHash/LSH batch sketcher (#1081) runs in the calling thread instead of fanning out to rayon — the same #688 calling-thread-vs-fanout guard, applied to the sketch kernel. The SimHash batch sketcher (#1090) shares this threshold. Relevant whenever the sketch kernel runs native — which is the default under auto since #1090 (set GOLDENMATCH_NATIVE=0 to force the Python path), or under GOLDENMATCH_NATIVE=1. 0 forces rayon always, a huge value forces sequential always.

Backend selection

Same pipeline, different execution engine. Auto-config picks one from your row count; you can override.

Backend	Range	How it works
`polars` (default)	< 500K rows	In-memory Polars. Fastest per record.
`bucket`	1M–200M, single box	Per-bucket native scoring. The validated single-box path on 16c/64GB. Planner picks it when native is enabled.
`chunked`	5M+, memory-constrained box	Streaming reader + cross-chunk join + block-keyed index. The 4c/16GB fallback.
`duckdb`	500K–50M	Out-of-core pair store, spills to disk so RAM isn’t the ceiling.
`ray`	≥ 50M, distributed	Per-block remote tasks. See the distribution caveat below — `backend="ray"` alone does not stream.

Force one in config or on the CLI:

config = gm.GoldenMatchConfig(backend="duckdb")

goldenmatch dedupe big.csv --backend ray
goldenmatch dedupe big.csv --chunked

Auto-selection knobs:

Variable	Default	Effect
`GOLDENMATCH_PLANNER_BUCKET`	`1`	The v3 planner may pick `bucket`. Set `0` to opt the bucket backend out.
`GOLDENMATCH_DUCKDB_SCORE_DB`	unset (in-memory)	Path for the DuckDB pair store to spill to disk. Point it at scratch with room when running DuckDB at 50M+.

The old single-threshold GOLDENMATCH_AUTOCONFIG_BACKEND / GOLDENMATCH_AUTOCONFIG_BACKEND_THRESHOLD env overrides were removed in v2.0. Backend selection is now the v3 planner’s job — it decides from (ComplexityProfile, RuntimeProfile). To force a backend, set it explicitly on GoldenMatchConfig(backend=...) or the CLI --backend.

ANN blocker backend (embedding/semantic blocking)

The embedding ANN blocker (ANNBlocker) resolves one of three inner-product backends per index build: native HNSW → FAISS → numpy. Native HNSW is the optional goldenmatch-hnsw wheel — a pure-Rust IndexHNSWFlat with zero C dependencies (no FAISS); FAISS (goldenmatch[embeddings]) is exact IndexFlatIP; numpy is the always-available all-pairs fallback. Scores are the raw inner product on the HNSW/FAISS paths and cosine on numpy — identical when vectors are L2-normalized (the default embedder output). In auto mode HNSW is chosen only where it wins: a large corpus and a small neighbor count per probe. Below that gate, brute force is both faster and exact, so small indexes (and the retrieve-all pattern behind the persistent VectorIndex) stay on the exact path unchanged.

Variable	Default	Effect
`GOLDENMATCH_ANN_BACKEND`	`auto`	Force a backend: `auto` (size-gated HNSW→FAISS→numpy), `hnsw` (native, ignores the size gate), `faiss`, or `numpy`. Degrades gracefully when the named library is absent.
`GOLDENMATCH_ANN_HNSW_MIN`	`4096`	In `auto`, the minimum corpus size for HNSW. Below it, the exact backend is used (HNSW’s win is asymptotic).
`GOLDENMATCH_ANN_HNSW_MAX_K`	`512`	In `auto`, the maximum `top_k` for HNSW. Above it (retrieve-nearly-all), the exact backend is used instead.
`GOLDENMATCH_ANN_HNSW_M`	`16`	HNSW graph degree (`M`): max neighbors per node on upper layers (`2M` on layer 0). Higher = better recall, more memory.
`GOLDENMATCH_ANN_HNSW_EF_CONSTRUCTION`	`200`	Candidate-list size during index build. Higher = better graph quality, slower build.
`GOLDENMATCH_ANN_HNSW_EF_SEARCH`	`64`	Candidate-list size during search (raised to at least `top_k`, capped at the corpus size). Higher = better recall, slower query.

goldenmatch-hnsw is an optional wheel, installed separately (pip install goldenmatch-hnsw) exactly like FAISS. Without it, the ANN blocker uses FAISS if present, else the numpy fallback — no behavior change.

Distributed pipeline

This is the knob most people get wrong, so read the contract before you run a 100M job. GOLDENMATCH_DISTRIBUTED_PIPELINE selects how run_dedupe_pipeline_distributed (the Ray Dataset entry point) executes:

Value	Phase	What actually happens
unset (default)	Phase 2 “cheat-line”	Calls `take_all()` on the input and runs `dedupe_df()` in memory on the driver. Back-compat default. Does not distribute scoring.
`1`	Phase 4 scaffold	Clustering/golden stages are Ray-ready, but auto-config and scoring still materialize on the driver. Kept for compatibility.
`2`	Phase 5 streaming	The real distributed path: scoring → clustering → golden → write all stay Ray Datasets. No `take_all()` on the input.

Phase 5 (=2) contract — different from in-memory dedupe:

Input must carry a global __row_id__ column. Without it, each partition synthesizes local IDs that collide and the weakly-connected-components step merges unrelated clusters.
Output is written to output_path as parquet golden records. It does not return an in-memory clusters dict the way dedupe_df does.
It still materializes two things on the driver: the ~5K-row auto-config sample, and cluster aggregates for golden-field synthesis. These are the last “cheat-lines” being retired.

GOLDENMATCH_ENABLE_DISTRIBUTED_RAY (default 0) is separate: it only lets the v3 planner pick ray during backend selection. It does not turn on the streaming pipeline. Explicit backend="ray" always works without it. The planner stopped auto-picking ray after the Distributed Plan v1 stack failed the 5M kill criterion against the bucket backend, so ray is opt-in. Recall-complete clustering at scale (#844, shipped 2026-06-11). Phase 5’s default path now block-shuffles scoring so true duplicates co-locate, then runs a relational randomized-contraction weakly-connected-components pass that clusters correctly across input-partition boundaries. This replaced two earlier WCC implementations that died at 100M (a driver head-wedge and a Ray streaming-executor deadlock). It is default-on and validated end-to-end at 100M (9.2 min, 20M clusters exact). On a multi-node cluster the WCC checkpoints each round to GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH, which must be a shared path (e.g. gs://...) — a node-local scratch breaks the cross-node parquet reads and now raises rather than silently under-merging.

Only scoring needs distribution at this scale; clustering usually doesn’t. The scored pair set is small (~(id_a, id_b, score) per match), so at 100M rows it’s ~1.8 GB — it fits one node. Clustering therefore routes to the in-memory scipy connected-components pass (seconds) automatically whenever the pairs fit driver RAM, and only falls back to the distributed WCC (a multi-hour tail) when they genuinely don’t. You don’t need to set GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD for this — it’s the default.

Correlated survivorship (field_groups and conditional field_rules) does not run on the distributed or Sail paths. The golden builder that resolves lock-step groups and conditional predicates runs on the driver. If your config includes field_groups or list-form field_rules, the Ray Phase-5 streaming pipeline (GOLDENMATCH_DISTRIBUTED_PIPELINE=2) and the Sail backend will refuse with a clear error rather than silently mis-merging. Run those configs on the in-memory path.

Distributed sub-knobs (only relevant on a real Ray cluster):

Variable	Default	Effect
`GOLDENMATCH_DISTRIBUTED_BLOCK_SHUFFLE`	`1` (on)	Block-shuffle scoring so duplicates co-locate before clustering. The recall-complete path. `0` restores the legacy per-partition `local_cc` (under-merges across partitions at scale).
`GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH`	unset	Scratch directory the randomized-contraction WCC checkpoints to. Must be shared (e.g. `gs://...`) on a multi-node cluster or the run raises.
`GOLDENMATCH_DISTRIBUTED_WCC`	`two_phase`	WCC algorithm for `build_clusters_distributed`’s own callers (the at-scale Phase-5 path forces `randomized_contraction` and ignores this). `two_phase` is chain-safe; avoid `pointer_jump` (can deadlock).
`GOLDENMATCH_DISTRIBUTED_WCC_SEED`	unset	Seed for the randomized-contraction affine hash (reproducible cluster IDs across runs).
`GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD`	unset (memory-aware)	Pair count above which clustering goes distributed. Unset (default): memory-aware — clustering stays on the fast driver-side scipy.csgraph path whenever the scored pair set fits available driver RAM, and only distributes when it genuinely won’t. Set an explicit integer to force a pair-count boundary either way.
`GOLDENMATCH_DISTRIBUTED_GOLDEN_THRESHOLD`	`5000000`	Multi-member row count above which golden-record building goes distributed.
`GOLDENMATCH_DISTRIBUTED_SCORE_NUM_CPUS`	`2`	CPUs requested per Ray scoring task.
`GOLDENMATCH_DISTRIBUTED_SCORE_PROJECT`	`1` (on)	Project to scoring-relevant columns before the block-shuffle so it moves narrow blocks, not full records (#957). Output-invariant. `0` disables.
`GOLDENMATCH_DISTRIBUTED_SHUFFLE_PARTS`	`min(256, cpu×4)`	Block-shuffle partition count. Raise it to shrink per-partition blocks so more `_score` tasks fit the object-store budget (relieves `_score` `ResourceBudget` backpressure). Co-location stays correct at any value.
`GOLDENMATCH_DISTRIBUTED_OP_RESERVATION`	unset (Ray default ~0.5)	Ray Data per-op object-store reservation ratio. Lower it (e.g. `0.2`) to hand the running `_score` op more object-store budget when it’s `ResourceBudget`-backpressured below CPU capacity (idle workers, object store not full). Tune at scale.
`GOLDENMATCH_DISTRIBUTED_SCORE_CONCURRENCY`	unset	Explicit `concurrency=` on the `_score` map_batches. Pins the task-pool size once blocks are narrow; the object-store reservation/shuffle-parts knobs above govern the actual `ResourceBudget` cap.

If the distributed score stage leaves workers idle (MapBatches(_score): … [backpressured:tasks(ResourceBudget)] with the object store well under capacity), _score is reservation-capped, not CPU-bound. Lower GOLDENMATCH_DISTRIBUTED_OP_RESERVATION (→ 0.2) and/or raise GOLDENMATCH_DISTRIBUTED_SHUFFLE_PARTS, then re-measure — the optimal pair is cluster-shaped. (Shrinking Ray Data’s target_max_block_size is not a safe lever: _score keeps each co-located partition whole via batch_size=None, so a split block would split a group and under-score.)

You bring the Ray cluster — GoldenMatch ships the pipeline, not the bootstrap. The GCP recipe is in docs/distributed-ray-cluster-setup.md. At single-box scale the bucket backend is validated through 200M and is simpler — use it unless the data won’t fit one box.

Fast path at scale (distribute only what doesn’t fit)

The single most expensive distributed-tuning mistake is distributing a stage that fits in memory. “The job is big” does not mean “every stage must be distributed.” Ask it per stage:

Stage	Fits one node at 100M?	Fast path	Slow trap
Scoring	No — 100M rows	Distributed across workers, native kernel on (`GOLDENMATCH_NATIVE=1` + `goldenmatch-native` installed on every node)	pure-Python (cluster installs plain `goldenmatch`); ~6 of 80 CPU if the shuffle blocks are oversized
Clustering (WCC)	Yes — the scored pair set is small (~1.76 GB / 110M edges at 100M)	In-memory driver-side `scipy.csgraph` connected-components (~60s)	the distributed randomized-contraction WCC — its O(log N) `gs://`-checkpoint rounds are a multi-hour tail when forced on a pair set that fits
Golden	depends	distributed groupby, or skip it if you only need cluster membership	building golden you then discard

Concrete: don’t force the distributed WCC on a pair set that fits. By default GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD is unset and the routing is memory-aware — clustering stays on the fast in-memory scipy.csgraph path whenever the scored pair set fits available driver RAM, and only distributes when it genuinely won’t. So at 100M (~110M pairs / ~1.76 GB) the default keeps clustering in-memory on a 64–256 GB head. Set an explicit integer only if you want to force a hard pair-count boundary either way:

export GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD=2000000000  # force in-memory WCC below 2B pairs

Setting it to 0 does the opposite — it forces the slow distributed WCC on everything. Recover a result without re-running. The Phase-5 connected-components clustering is independent of the WCC algorithm, so a completed run’s cluster assignments (written to gs:// via the assignments_output_path hook) are the result. If a run finishes but the output is lost, or you want to re-score, download the assignments and score them against your ground truth on a 64 GB box — no cluster, no re-run. The 100M group_by for that scoring OOMs a 16 GB box, so run it on a large runner, not your laptop.

Prep & scoring perf opt-ins

These trade nothing for speed/memory at scale and are safe to leave at their defaults. Two are already on.

Variable	Default	Effect	When to change
`GOLDENMATCH_BUCKET_SLIM_PROJECTION`	`1` (on)	After bucket scoring, drop columns downstream stages don’t need (smaller per-bucket frames). Measured −3.8 GB peak RSS at 10M, output invariant.	Set `0` only if a custom downstream consumer needs a column it drops.
`GOLDENMATCH_GOLDEN_SLIM_MULTIDF`	`1` (on)	Slim internal columns before the multi-DataFrame golden build.	Rarely; `0` restores the wider frame.
`GOLDENMATCH_IDENTITY_BATCH_FINGERPRINT`	`1` (on)	Compute no-PK identity hashes in one Arrow batch instead of per row (2.6x on the fingerprint step).	`0` restores per-row, for parity debugging.
`GOLDENMATCH_IDENTITY_WRITE_PIPELINE`	`1` (on)	psycopg pipeline mode for the per-record identity write path (`resolve_clusters` absorb/merge). Streams `upsert`/`emit_event`/`add_edge` statements without a per-statement network round-trip — the dominant cost of a re-resolve against a remote Postgres (Cloud SQL), where a single transaction removed the fsync but not the round-trips. Postgres-only; no-op on SQLite.	`0` restores per-statement writes, for parity debugging.
`GOLDENMATCH_ENSEMBLE_KERNEL`	`1` (on)	The `ensemble` scorer rides the bucket fast path (per-pair + the float64 `_score_block_vec` vec lane + the native `score_one` id 12 arrow kernel), measured 1.47× faster end-to-end on Febrl3 with byte-identical recall. `0` (`off`/`false`) is the kill-switch → the historical float32 `find_fuzzy_matches` decline path. The earlier default-on attempt regressed on name-scorer matchkeys (`name_freq_weighted_jw` / `given_name_aliased_jw` — neither vec nor native, so they ran the O(N²) per-pair Python loop, 19× slower than `find_fuzzy_matches`); that is now fixed at the source by a perf guard in `_resolve_fast_path` that declines the fast path whenever a field would force per-pair Python.	Leave on. Set `0` only to reproduce pre-2026-07-21 output byte-for-byte. See `docs/superpowers/specs/2026-07-21-ensemble-kernel-measurement.md`.

Diagnostic-only — these can SLOW the run:

Variable	Default	Effect
`GOLDENMATCH_BUCKET_DEBUG`	`0`	Print per-bucket prep / kernel / post-filter timing for `backend=bucket`. Output-invariant, near-zero cost, but it’s a debug aid, not a speed-up.
`GOLDENMATCH_PREP_STAGED_COLLECT`	`0`	Collect prepared records in stages (memory-management experiment). Can be slower; leave off unless diagnosing prep RSS.
`GOLDENMATCH_STANDARDIZE_STAGED`	`0`	Staged standardization pipeline (experimental). Same caveat.

Fused kernel auto-routing

The pipeline can route covered slow-path runs to the fused Arrow-native kernels, byte-identically, with no config change. There are two seams with very different maturity: golden routing is a broad default-on win; match routing is a narrow capacity-survival path that fires only under memory pressure.

Variable	Default	Effect	When to change
`GOLDENMATCH_GOLDEN_FUSED`	on	Golden seam auto-routing (default-on, byte-identical). Tries the fused Arrow-native golden kernel on every covered slow-path config, reaching golden output at ~2x lower peak RSS. Declines to the classic builder (unchanged output) when the config is uncovered, the native kernel is absent, full `ClusterProvenance` lineage is requested, or the run is fast-path-eligible. Surfaced on `DedupeResult.golden_fused_used`.	`0` / `false` / `off` is the kill-switch (forces the classic golden builder everywhere).
`GOLDENMATCH_MATCH_FUSED`	on	Match seam auto-routing — a narrow capacity-survival path. A controller post-step short-circuits the whole match stage to the fused match kernel under a memory-pressure gate AND a narrow config profile: `auto_split=False`, no identity/adaptive/memory/llm_boost/confidence_majority/full-provenance, not across-files-only, and a covered matchkey — weighted (single-key or multi-pass) or Fellegi-Sunter (single-key or multi-pass; the FS path fits EM inside the short-circuit and was wired in #1892). Zero-config auto-config commits `auto_split=True` and explicit configs bypass the controller, so it stays dormant unless you configure that artifact-free profile and hit the RSS gate. When it fires it is a capacity mode that sheds `scored_pairs`/`review_pairs` + cluster-confidence/lineage to survive; cluster membership + golden stay byte-identical. Marked `DedupeResult.match_fused_capacity_mode` + telemetry `rule_name` `+fused_match_post_step`. Measured 2–19× leaner peak RSS on covered FS shapes.	`0` / `false` / `off` is the kill-switch.
`GOLDENMATCH_FUSED_PRESSURE_FRACTION`	`0.65`	Fraction of available RAM the estimated classic-path peak RSS must exceed before match routing’s pressure gate fires. Only relevant to the pressure-gated match seam.	Lower to make the capacity path trip earlier; raise toward `1.0` to make it trip only under extreme pressure.
`GOLDENMATCH_FUSED_RSS_SCALE` / `_BYTES_PER_PAIR` / `_BYTES_PER_CELL` / `_BLOCK_CONCURRENCY`	`0.763` / `64` / `40` / `4`	Calibration coefficients for the est-peak-RSS model that drives the match pressure gate. Defaults are fit against measured runs.	Only when re-calibrating the est-RSS model against a new box/workload; leave alone otherwise.

Auto-config & planning

Variable	Default	Effect	When to set
`GOLDENMATCH_AUTOCONFIG_MEMORY`	`1` (on)	Read/write cross-run auto-config memory at `~/.goldenmatch/autoconfig_memory.db`.	Set `0` in CI or any run that must be reproducible and not influenced by prior runs.
`GOLDENMATCH_PLANNING_EFFORT`	`normal`	Iteration budget for auto-config: `fast` / `normal` / `thinking` / `einstein`. Higher = more thorough, slower; `thinking`+ measure real full-frame pair counts.	`fast` to speed up config on huge inputs; `thinking` when auto-config picks a poor blocking key. `normal` is byte-identical to the historical default.
`GOLDENMATCH_CONFIG_LINT`	`warn`	Pre-flight config linter on the resolved config (zero-config or explicit). `warn` logs data-shape findings and runs; `strict` raises `ConfigLintError` on an error finding before the run; `off` skips. Findings attach to `DedupeResult.lint_findings`.	`strict` in CI/production to fail fast on a degenerate config (e.g. a blocking key that will OOM). `off` to silence.
`GOLDENMATCH_AUTOCONFIG_LLM`	`0`	Let an LLM propose a config when heuristics are exhausted on a RED profile. Needs `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`.	Opt in only when heuristic auto-config can’t find a healthy config and you accept the API cost.
`GOLDENMATCH_AUTOCONFIG_INDICATOR_BUDGET`	unset	Set `fast` to skip the expensive complexity indicators.	Speeding up config on very large inputs.
`GOLDENMATCH_AUTOCONFIG_FORCE_INCLUDE` / `_FORCE_EXCLUDE`	unset	Comma-separated columns to force into / out of auto-config analysis.	Rescue a column auto-config drops, or exclude a noisy one, without writing a full explicit config.
`GOLDENMATCH_FIELD_GROUP_SURVIVORSHIP`	`0` (off)	`1` enables auto-detection of correlated field groups for survivorship. When on, auto-config proposes lock-step field groups (address, person name, contact) so related columns are always promoted from the same source record. Explicitly declared `field_groups` in YAML are always honored regardless of this flag. Off by default — output is byte-identical to prior behavior when unset. Group promotions are visible in the lineage `golden_records` section, `explain --cluster`, and the MCP `lineage` tool (see Surfacing the group decision).	Set when you want auto-config to detect field groups without writing explicit `field_groups` YAML. Equivalent to `field_group_detection: true` in the `golden_rules` config block.
`GOLDENMATCH_PERCEPTUAL_AUTOCONFIG`	`0` (off)	`1` enables multimodal-ER media-as-evidence wiring (ADR 0022). When on, auto-config detects columns holding fixed-width hex perceptual hashes (a 16-char image pHash from `core.perceptual.phash_hex`, a 96-char rotation/crop-aware radial-variance profile from `radial_hex`, or a multi-word audio fingerprint from `audio_fp_hex`) and appends a `phash` / `radial` / `audio_fp` matchkey, plus `perceptual` banded-hamming-LSH blocking for an image column when the committed config has nothing else to block on (the rotation-aligned `radial` feature gets no LSH blocking). Additive + idempotent + fail-open; output is byte-identical to prior behavior when unset.	Set when a dataset carries a precomputed perceptual-hash column you want treated as a match feature without writing explicit matchkey YAML.
`GOLDENMATCH_AUTOCONFIG_ROUTE_PROBABILISTIC`	`0` (off)	`1` lets zero-config auto-config route a probabilistic-shaped dataset to the Fellegi-Sunter path instead of the default exact+weighted matchkeys. The trigger fires when the emitted config has no surviving exact matchkey backed by a strong-identity column (`identifier` / `email` / `phone`) and ≥2 fuzzy fields — i.e. there’s no clean key carrying the dedup, so EM-weighted comparison wins. It delegates to `auto_configure_probabilistic_df` (which also diversifies blocking onto orthogonal stable keys). A behavior change, so default-off pending a broader regression sweep; measured lift on biographical/PII data (historical_50k F1 0.466 → 0.83, recall 0.39 → 0.75) with no regression on datasets that retain a strong key.	Opt in on no-strong-identifier, error-heavy datasets (historical/biographical records, dirty PII) where exact matching under-recalls. Leave off when your data has a reliable id/email/phone.

If auto-config raises ControllerNotConfidentError, that’s by design at ≥100K rows on a RED profile — it’s refusing to run a noise-result dedupe. See Passing an explicit config for recovery (the in-repo docs/explicit-config.md has the failing-sub-profile table). There are also a number of experimental blocking and probabilistic knobs (GOLDENMATCH_BLOCKING_*, GOLDENMATCH_FS_*, GOLDENMATCH_NOISE_AWARE_*, GOLDENMATCH_QUALITY_AWARE_BLOCKING, GOLDENMATCH_FD_NEGATIVE_EVIDENCE). These are research/tuning levers, default-safe, and listed in the full table below. Most users never touch them.

Config-suggestion (healer) knobs

These tune the healer — the review_config config-suggestion loop, which is now wired into the default dedupe_df pipeline as an advisory surface (a free controller signal gates a cheap attach; see GOLDENMATCH_SUGGEST_ON_DEDUPE). The healer needs goldenmatch[native] and degrades gracefully without it. You almost never need to change these — the defaults are the values the suggester gym selected.

Variable	Default	Effect	When to set
`GOLDENMATCH_SUGGEST_ON_DEDUPE`	on	Master switch for the default-pipeline healer surface (Python `dedupe_df` / CLI / servers and the TS/WASM `dedupe(rows, { ... })`): attaches advisory `suggestions` when a free trigger fires. Set `0` to disable the trigger + attach entirely (the explicit `suggest=`/`heal=` opt-ins and `review_config` still work).	Set `0` to opt out of the advisory default-run surface.
`GOLDENMATCH_HEAL_STEP_CAP`	`5`	Max healer iterations — the coarsest cost lever (each step is a full dedupe plus up to `SUGGEST_MAX_VERIFY` verification re-runs). Clamped to >= 1; invalid input falls back to the default.	Lower to cap healer cost on large inputs; raise to allow more repair steps.
`GOLDENMATCH_HEAL_MIN_HEALTH_GAIN`	unset (disabled)	Marginal-gain early stop (#1404): stop the heal loop once an iteration’s cluster-health gain falls below this float. Unset = run up to `HEAL_STEP_CAP`; `0.0` stops on the first non-improving step.	Set a small positive value to stop early once suggestions stop paying off.
`GOLDENMATCH_SUGGEST_MAX_VERIFY`	`8`	Max suggestions verified by a full pipeline re-run before the remaining tail passes through unverified. Each verified candidate is one extra re-run. Values `< 1` are clamped to `1`.	Lower to cut healer cost; raise to verify more candidates end-to-end.
`GOLDENMATCH_SUGGEST_HEALTH`	`cohesion`	Self-verify health proxy the gate uses to keep only non-worsening suggestions. `cohesion` (precision-sensitive `min_edge` × saturating coverage) is the default since 2026-06; `legacy` restores the older `matched_rate × avg_conf − HHI` proxy.	`legacy` only to reproduce pre-2026-06 suggestion behavior. The bake-off showed `cohesion` lifts live recovery 0.15 → 0.54 with no real-pair net-negatives.
`GOLDENMATCH_SUGGEST_COHESION`	`min_edge`	Cohesion statistic: `min_edge`, `mean_bottomk_edge`, or `edge_below_cutoff_fraction`.	Leave alone; `min_edge` won the bake-off.
`GOLDENMATCH_SUGGEST_COVERAGE_CAP`	`0.50`	Coverage-saturation cap for the cohesion proxy (matched-rate at which coverage saturates to 1.0).	Leave alone; `0.50` is the safe operating point — tighter caps let a real-pair net-negative through.
`GOLDENMATCH_SUGGEST_VERIFY`	on	The self-verify gate. Set `0` to return raw kernel suggestions unfiltered (no no-net-negative guarantee).	Diagnostics only — to see what the kernel would propose before the gate.
`GOLDENMATCH_SUGGEST_FULL_DIST`	`0` (off)	Source the kernel’s score distribution from a threshold-0 diagnostic run so the dip (`lower_threshold`) rule sees the full pre-threshold tail.	Set `1` when you want the dip rule to fire on configs whose threshold is well above the score valley.
`GOLDENMATCH_SUGGEST_RECALL_MIN_SHED`	`0.02`	Min recall the tighten-key rule must shed before it will even consider vetoing a matchkey.	Leave alone; raising it makes the recall-shed guard more conservative.
`GOLDENMATCH_SUGGEST_RECALL_COH_ABS`	`0.15`	Max cohesion gain that still counts as “small” when weighing a recall shed.	Leave alone; the gym-selected operating point.
`GOLDENMATCH_SUGGEST_RECALL_RATIO`	`2.0`	Min cohesion-gain-per-recall-shed ratio required to justify shedding recall.	Leave alone; higher = demands more cohesion payoff per unit recall lost.

Servers, security & storage

When you expose any server (MCP, REST, A2A, web UI) on a non-loopback host, it is fail-closed: it refuses to start without the matching bearer token.

Variable	Used by	Notes
`GOLDENMATCH_MCP_TOKEN`	MCP server	Required on non-loopback binds. `/.well-known/` card stays public.
`GOLDENMATCH_MCP_ALLOW_PUBLIC`	MCP server	Opt-out of the fail-closed default: set `1`/`true` to intentionally run an open, unauthenticated public MCP server (e.g. a showcase deployment) without `GOLDENMATCH_MCP_TOKEN`. Unset/`0` = fail closed (a non-loopback bind without a token is refused). A set `GOLDENMATCH_MCP_TOKEN` still takes precedence.
`GOLDENMATCH_API_TOKEN`	REST API	Required on non-loopback binds (all except `/health`).
`GOLDENMATCH_AGENT_TOKEN`	A2A agent	Required on non-loopback binds (except `/health` + agent card).
`GOLDENMATCH_WEB_TOKEN`	Web UI	Required on non-loopback binds (except `/api/v1/healthz`).
`GOLDENMATCH_API_CORS_ORIGINS`	REST API	Comma-separated allow-list; replaces wildcard CORS.
`GOLDENMATCH_ALLOWED_ROOT`	All file-taking surfaces	Opt-in path sandbox. When set, every user-supplied path must resolve under this root or the call is rejected. Recommended for deployed/shared instances (e.g. the Railway MCP server sets it to `/data`). Unset = no containment (correct local-first default).
`GOLDENMATCH_MCP_MAX_UPLOAD_BYTES`	MCP server (inline upload)	Max decoded size for `file_content` / `upload_dataset` inline uploads. Default `67108864` (64 MB). Over the cap the call is rejected — pass a public `http(s)://` URL as `file_path` for larger datasets.
`GOLDENMATCH_MCP_UPLOAD_TTL`	MCP server (inline upload)	Age in seconds after which an inline-uploaded temp file is reaped. Default `86400` (24 h). Uploaded paths are ephemeral scratch; re-upload if you need one older than this.
`GOLDENMATCH_MCP_SESSION_MAX`	MCP server (session state)	Max number of per-MCP-session dedupe/match runs kept in memory so the stateful tools (`list_clusters`, `get_cluster`, `get_golden_record`, `explain_match`, `evaluate`, `export_results`, `match_record`, `find_duplicates`) can read a run made earlier in the same session. Default `64`; least-recently-used sessions evict past the cap.
`GOLDENMATCH_MCP_SESSION_TTL`	MCP server (session state)	Seconds a stored session run stays readable (anchored to when the run was made, not last access). Default `3600` (1 h); past it the stateful tools return a clean “no run loaded” error until you re-run `agent_deduplicate`/`match_sources`.

Storage / persistence:

Variable	Default	Effect
`GOLDENMATCH_DATABASE_URL` / `GOLDENMATCH_IDENTITY_DSN`	unset	Postgres/SQLite connection for the identity graph + review queue / Alembic migrations.
`GOLDENMATCH_PREPARED_RECORD_STORE_DIR`	unset	Directory for the disk-backed prepared-record store.
`GOLDENMATCH_PREPARED_RECORD_STORE_PERSIST`	`0`	`1` keeps the prepared-record store across calls instead of cleaning it up.
`GOLDENMATCH_VECTOR_INDEX_DIR`	`.goldenmatch_vectors`	Default on-disk directory for the persistent semantic `VectorIndex` (embeddings + records + manifest) when a path isn’t passed explicitly. The index survives across runs/processes so embeddings aren’t rebuilt every run.

Product analytics (opt-in): Anonymous usage analytics — off by default. Emits only when GOLDENMATCH_ANALYTICS is on and a POSTHOG_API_KEY is set, so a user’s machine stays silent unless they opt in (the project’s own hosted services set the flag server-side). PII-free by construction: only coarse scalars (scale bands, backend, version) ever leave, never record values, column names, or paths. Full collection list + guarantees in packages/python/goldenmatch/ANALYTICS.md.

Variable	Default	Effect
`GOLDENMATCH_ANALYTICS`	`0` (off)	`1`/`true`/`on` opts in to anonymous, PII-free usage events. Off = nothing is collected or sent.
`POSTHOG_API_KEY`	unset	Destination PostHog project key. Required for any event to send; absent = silent no-op.
`POSTHOG_HOST`	`https://us.i.posthog.com`	PostHog ingestion host override.

Gotchas that cost a day

Read these before a big run. Each is a real trap that produces a slow or wrong result with no error.

backend='ray' ran in-memory and never used the cluster

backend="ray" (or dedupe_df(backend="ray")) alone runs the Phase 2 cheat-line: it collects the whole frame to the driver and dedupes in memory. The cluster sits idle. To actually stream across workers you need GOLDENMATCH_DISTRIBUTED_PIPELINE=2, a Ray Dataset input with a global __row_id__, and you read golden output from output_path — not the return value. On a multi-node cluster also set a shared GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH (e.g. gs://...). For single-box scale, use bucket, not ray.

It ran 100x slower than the docs because native silently fell back to Python

Under the default GOLDENMATCH_NATIVE=auto, if the native wheel isn’t importable (not installed, ABI mismatch, stale build) the scoring kernel silently runs in Python — orders of magnitude slower. On any run where speed matters, set GOLDENMATCH_NATIVE=1 so a missing wheel raises instead of crawling. Confirm with python -c "import goldenmatch_native" or check goldenmatch.core._native_loader.native_available().

The prep stage dominated wall on address-heavy data

The per-row Python address-normalize plugin is a real bottleneck at scale. Set GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE=1 to use the Polars-native expression instead. It’s off by default, so you have to opt in.

A distributed 100M run took hours instead of minutes

The scored pair set is small (~1.76 GB at 100M) and fits one node. By default GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD is unset and routing is memory-aware — clustering stays in-memory (scipy.csgraph, ~60s) whenever the pair set fits driver RAM, so a 100M run should already keep clustering on the fast path. The multi-hour tail happens if something forces the distributed WCC (whose gs://-checkpoint rounds are slow): check you haven’t set the threshold to 0 (forces the slow path on everything) or to a value below your pair count. See Fast path at scale. Distribute the scoring (the rows don’t fit), not the clustering (the pairs do).

Results changed between runs for no reason

Cross-run auto-config memory (GOLDENMATCH_AUTOCONFIG_MEMORY=1, the default) lets a prior run influence the next. For reproducible benchmarks set GOLDENMATCH_AUTOCONFIG_MEMORY=0.

Python hung at import on Windows

Polars runs a WMI CPU probe at import that can hang on some Windows boxes. Set POLARS_SKIP_CPU_CHECK=1. (Add PYTHONIOENCODING=utf-8 if non-ASCII output mangles the console.)

Throughput tier (sketch-then-verify)

New in #1083, builds on #1080. An opt-in high-throughput path for large text corpora where you care about near-duplicate recall but can trade precision for speed. Instead of the full fuzzy/Fellegi-Sunter pipeline, it blocks with MinHash/LSH (lsh) or SimHash (simhash) and confirms candidate pairs by cheap sketch distance before any field-level scoring. Think: training data dedup, web crawl dedup, paper-corpus cleaning.

Enabling it

Pass throughput= to dedupe_df:

import goldenmatch as gm

# Float: sets recall_target (0 to 1). Adjusts LSH banding.
result = gm.dedupe_df(df, throughput=0.95)

# True: uses the default recall_target (0.95).
result = gm.dedupe_df(df, throughput=True)

# None (default): the throughput tier is off. Byte-identical to today.
result = gm.dedupe_df(df)

# ThroughputConfig: full control.
from goldenmatch import ThroughputConfig
result = gm.dedupe_df(df, throughput=ThroughputConfig(
    recall_target=0.90,
    similarity_threshold=0.75,
))

Or via environment variables:

Variable	Default	Effect
`GOLDENMATCH_THROUGHPUT`	`0` (off)	`1` enables the throughput tier with default settings.
`GOLDENMATCH_THROUGHPUT_RECALL`	`0.95`	LSH recall target (tunes banding).
`GOLDENMATCH_THROUGHPUT_SIMILARITY`	unset	Override the near-dup similarity threshold (Jaccard default `0.8`; cosine default `0.85`).

What it does

Auto-selects blocking strategy. Picks simhash (cosine/semantic) when an embedder is reachable, otherwise falls back to lsh (MinHash/Jaccard on shingles). Uses the longest text column in the frame.
Blocks with sketch buckets. Records that share at least one LSH band become candidates. The recall_target controls the banding split via the LSH probability curve.
Verifies by sketch distance only. Pairs that clear similarity_threshold are kept; the field-level fuzzy/FS scorer does not run.
Raises ThroughputNotApplicableError when the frame has no text column. Fix: pass an explicit BlockingConfig to direct it to a column.

Posture and honest reporting

The throughput tier does not measure F1 or precision. DedupeResult.throughput_posture (and the controller telemetry throughput block) carry the metrics the tier can honestly report:

Key	Meaning
`metric`	`"jaccard"` (LSH) or `"cosine"` (SimHash)
`recall_target`	The recall target you passed (tunes banding)
`similarity_threshold`	The sketch-distance threshold applied at verify
`bands`	LSH band count selected for the recall target
`rows_per_band`	Rows per band
`expected_recall`	LSH-theoretic recall at the banding: `1 - (1 - t^r)^b` where `t = similarity_threshold`, `r = rows_per_band`, `b = bands`. This is an analytic bound, not measured on your data.
`reduction_ratio`	`1 - candidate_pairs / possible_pairs`. Observed.
`candidate_pairs`	Pairs that passed blocking.
`verified_pairs`	Pairs that passed sketch-distance verification.
`notes`	Human-readable posture string.

expected_recall is the LSH-theoretic probability that a pair at exactly similarity_threshold Jaccard/cosine collides in at least one band. It is not a measured F1 or recall against ground truth. For pairs above threshold the curve is tighter; for pairs below it, collision probability drops rapidly. This is an honest bound, not a precision measurement.

Measured throughput (benchmark)

The bench-corpus-dedup harness (scripts/bench_corpus_dedup/, .github/workflows/bench-corpus-dedup.yml) measures the tier end-to-end against a real public corpus, with injected ground-truth near-dups so recall is measurable. On a slice of FineWeb (web crawl), the zero-config dedupe_df(df, throughput=…) path measured:

docs	docs/sec	MB/sec	measured recall
14,000	275	0.84	0.44
70,000	1,192	3.64	0.43

docs/sec rises with scale because the wall is dominated by a fixed ~50s of zero-config auto-config at these sizes; the sketch dedup itself is ~9s at 70k (≈7,800 docs/sec raw). Measured recall ≈ 0.43 is the honest LSH recall over a mix of exact / partial-overlap / paraphrase near-dups — the paraphrase and partial dups often do not collide in the LSH bands. (Contrast expected_recall above: the per-pair collision probability at the similarity threshold, not a recall over a real corpus — which is exactly why the bench measures recall directly.) A deterministic per-PR CI perf gate (throughput-gate job, via scripts/bench_corpus_dedup/throughput_perf_gate.py) guards regression on the machine-independent cost — candidate pairs, reduction ratio, and measured recall on a vendored offline corpus — against a committed baseline, so a change can’t quietly blow up the candidate count or drop recall.

Scale note: the published numbers are the validated 14k–70k range. 100k+ corpora currently bottleneck on the survivorship/golden stage; lifting that ceiling (and a datatrove head-to-head) are tracked follow-ups.

Scope

The throughput tier is single-node only. Distributed throughput (#1084) and the product surface (#1085) are follow-on issues. Default-off (throughput=None) is byte-identical to today in every metric.

Configuration precedence

When the same setting is reachable multiple ways, the more specific wins:

Explicit API / CLI argument (dedupe_df(..., backend=...), --backend) — highest.
Config field (GoldenMatchConfig(backend=...), YAML).
Environment variable (GOLDENMATCH_*).
Built-in default (auto-config / planner choice) — lowest.

So an env var sets the default behavior for a process, a config field overrides it for a run, and an explicit call argument overrides it for one call.

Full environment-variable reference

Every GOLDENMATCH_* variable the package reads (plus the one GOLDEN_* exception, GOLDEN_DIAGNOSTICS), grouped by who should care. Defaults verified against source. Most users only ever touch the recommended setups.

Everyday & scale

Variable	Default	Values	Effect
`GOLDENMATCH_NATIVE`	`auto`	`auto` / `1` / `0`	Native kernel gate. `1` requires native (raises if missing); `0` forces Python.
`GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE`	off	`1`	Polars-native address-normalize fast path.
`GOLDENMATCH_FRAME`	`arrow`	`arrow`/`polars`	Frame backend. `arrow` is the DEFAULT since v3.0.0 (measured ~36% faster end-to-end on the 100K zero-config A/B): file ingest via pyarrow, the whole engine runs on the Frame seam, and result frames are `pyarrow.Table`. Since v3.1.0 polars is an OPTIONAL dependency and the polars-free install is the FAST configuration (measured head-to-head at 500K: 7.11s polars-free vs 7.55s polars-present, identical outputs — the Rust fused kernels carry the hot paths). `polars` here requires `pip install 'goldenmatch[polars]'` (a clear error names the extra otherwise); the extra is a COMPATIBILITY surface (this classic lane, the kernel-absent golden fast replay, goldencheck cell-quality weighting), not an accelerator. Output-equivalence enforced by a differential harness + frozen fixtures.
`GOLDENMATCH_FRAME_LANE`	`1` (on)	`0`/`1`	Arrow “frame lane” for the output-writing pipeline (keeps the spine arrow through the final collect). `0` restores the classic polars shim. Ignored when `GOLDENMATCH_COLUMNAR_PIPELINE=1`.
`GOLDENMATCH_AUTOCONFIG_MEMORY`	`1`	`0`/`1`	Cross-run auto-config memory. `0` for reproducible/CI runs.
`GOLDENMATCH_PLANNING_EFFORT`	`normal`	`fast`/`normal`/`thinking`/`einstein`	Auto-config iteration budget.
`GOLDENMATCH_DISTRIBUTED_PIPELINE`	unset	`1`/`2`	Distributed execution phase. `2` = real streaming (needs `__row_id__`, writes to `output_path`).
`GOLDENMATCH_ENABLE_DISTRIBUTED_RAY`	`0`	`0`/`1`	Let the planner pick `ray`. Does not enable streaming on its own.
`GOLDENMATCH_PLANNER_BUCKET`	`1`	`0`/`1`	Allow the planner to pick `bucket`.
`GOLDENMATCH_BUCKET_DEFAULT`	`1` (on)	`0`/`off`/`false`/`no`	Default in-memory bucket FS route on runs up to 750K rows (the #1798-safe path). Opt out to force the legacy per-block path. Distinct from `GOLDENMATCH_FS_DEFAULT_BUCKET`, which routes the FS bucket path itself.
`GOLDENMATCH_ATTRIBUTE_DEMOTION`	`1` (on)	`0`/`1`	Demote an EXACT matchkey on a shared group/list/facility value (shared clinic phone, campaign id, facility NPI) when its large shared-value groups do not co-agree on the person name. `0` restores the pre-2.7.0 behavior.
`GOLDENMATCH_DISCRIMINATIVE_VETO`	`1` (on)	`0`/`1`	The #1351 exact-key discriminative-power veto (demote a shared-locality exact key). `0` disables it.
`GOLDENMATCH_DISCRIMINATIVE_TAU`	`0.5`	float [0,1]	Co-agreement floor for the discriminative veto: an exact key whose large shared-value groups co-agree on the person name below this fraction is vetoed. Out-of-range values fall back to the default.
`GOLDENMATCH_TF_NAME_WEIGHTING`	`1` (on)	`0`/`1`	Data-driven per-dataset name-frequency downweight on `name_freq_weighted_jw` fields. `0` restores the static-census behavior.
`GOLDENMATCH_BASE_STORE`	`auto`	`auto`/`memory`/`lance`	Candidate base store for the ANN `match_one` path. `auto` picks Lance only above ~2M rows when installed.
`GOLDENMATCH_DUCKDB_SCORE_DB`	unset	path	DuckDB pair-store spill path.
`GOLDENMATCH_NATIVE_RAYON_MIN_PAIRS`	`20000000`	int	Native kernel sequential-vs-rayon threshold.
`GOLDENMATCH_NATIVE_RAYON_MIN_BLOOM_ROWS`	`10000`	int	Native PPRL bloom-filter sequential-vs-rayon threshold (row count). `0` = always parallel; a very large value = always sequential. Sibling of `_RAYON_MIN_PAIRS` (the #688-style futex-park guard).
`GOLDENMATCH_CONFIG_LINT`	`warn`	`warn`/`strict`/`off`	Pre-flight config linter: warn-and-run, refuse on error, or skip.
`GOLDENMATCH_THROUGHPUT`	`0` (off)	`0`/`1`	Enable the sketch-then-verify throughput tier. Equivalent to `dedupe_df(throughput=True)`.
`GOLDENMATCH_THROUGHPUT_RECALL`	`0.95`	float	LSH recall target; tunes the band/row split.
`GOLDENMATCH_THROUGHPUT_SIMILARITY`	unset	float	Override the near-dup similarity threshold (Jaccard `0.8` / cosine `0.85` when unset).
`GOLDENMATCH_AUTO_SEMANTIC_BLOCKING`	on	`0`/`1`	Auto-enable SimHash-over-embeddings blocking for text-heavy data (#1090). Default-on; a no-op when no embedder is reachable. `0` disables.
`GOLDENMATCH_SEMANTIC_BLOCKING_THRESHOLD`	`0.6`	float (0,1)	Recall threshold for auto semantic blocking; lower = more bands = higher recall. Drives the SimHash band/row split.

Distributed sub-knobs

Variable	Default	Effect
`GOLDENMATCH_DISTRIBUTED_BLOCK_SHUFFLE`	`1` (on)	Block-shuffle scoring (recall-complete path). `0` = legacy `local_cc`.
`GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH`	unset	Randomized-contraction WCC scratch. Must be shared (`gs://...`) on multi-node.
`GOLDENMATCH_DISTRIBUTED_WCC`	`two_phase`	WCC for `build_clusters_distributed`’s own callers (Phase-5-at-scale forces `randomized_contraction`). Avoid `pointer_jump`.
`GOLDENMATCH_DISTRIBUTED_WCC_SEED`	unset	Randomized-contraction hash seed (reproducible cluster IDs).
`GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD`	unset (memory-aware)	Pair count → distributed clustering. Unset default routes by driver-RAM fit; set an integer to force a hard boundary.
`GOLDENMATCH_DISTRIBUTED_GOLDEN_THRESHOLD`	`5000000`	Row count → distributed golden.
`GOLDENMATCH_DISTRIBUTED_SCORE_NUM_CPUS`	`2`	CPUs per Ray scoring worker.

Perf opt-ins & diagnostics

Variable	Default	Effect
`GOLDENMATCH_BUCKET_SLIM_PROJECTION`	`1` (on)	Drop unneeded columns after bucket scoring.
`GOLDENMATCH_GOLDEN_SLIM_MULTIDF`	`1` (on)	Slim internal columns before golden build.
`GOLDENMATCH_GOLDEN_FUSED`	on	Golden-seam fused auto-routing (default-on, byte-identical, ~2x lower peak RSS). `0` kills it. See Fused kernel auto-routing.
`GOLDENMATCH_MATCH_FUSED`	on	Match-seam fused auto-routing (weighted + Fellegi-Sunter) - a narrow capacity-survival path that fires only under memory pressure on an artifact-free config. `0` kills it. See Fused kernel auto-routing.
`GOLDENMATCH_FUSED_PRESSURE_FRACTION`	`0.65`	RAM-fraction pressure gate for the match-seam routing.
`GOLDENMATCH_FUSED_RSS_SCALE` / `_BYTES_PER_PAIR` / `_BYTES_PER_CELL` / `_BLOCK_CONCURRENCY`	`0.763` / `64` / `40` / `4`	est-peak-RSS calibration coefficients for the match pressure gate.
`GOLDENMATCH_IDENTITY_BATCH_FINGERPRINT`	`1` (on)	Batch no-PK identity hashing.
`GOLDENMATCH_IDENTITY_WRITE_PIPELINE`	`1` (on)	Pipeline the per-record identity writes on remote Postgres (#1912).
`GOLDENMATCH_BUCKET_DEBUG`	`0`	Per-bucket timing breakdown (diagnostic).
`GOLDEN_DIAGNOSTICS`	`1` (on)	Anomaly diagnostics (note: no `MATCH` in the name). On states that are probably a GoldenMatch bug — a native wheel-skew slow path, an unexpected `dedupe_df`/`match_df` crash, the config linter itself crashing, or a broken native install — emit a warning with a prefilled GitHub issue URL plus a PII-safe environment report. Never fires on expected fallbacks or user errors, and sends nothing anywhere (a better error message, not telemetry). `0` disables.
`GOLDENMATCH_PREP_STAGED_COLLECT`	`0`	Staged prep collect (experimental, may slow).
`GOLDENMATCH_STANDARDIZE_STAGED`	`0`	Staged standardization (experimental, may slow).
`GOLDENMATCH_BUCKET_VEC_MIN` / `_MAX`	`32` / `2000`	Block-size band for the vectorized float64 NxN cdist path in bucket scoring.
Fellegi-Sunter routing & scoring knobs (`GOLDENMATCH_FS_*`)	—	Have their own home: see Probabilistic / Fellegi-Sunter for the path-routing map + the complete `FS_*` table.
`GOLDENMATCH_BENCH_DUMP_PAIRS`	unset	Dump per-stage pairs for benchmarking.
`GOLDENMATCH_SKIP_MATCH_LOG` / `GOLDENMATCH_MATCH_LOG_FLUSH_PAIRS`	—	Streaming match-log controls.

Auto-config & blocking (advanced)

Variable	Default	Effect
`GOLDENMATCH_AUTOCONFIG_LLM`	`0`	LLM config fallback (needs API key).
`GOLDENMATCH_AUTOCONFIG_INDICATOR_BUDGET`	unset	`fast` skips expensive indicators.
`GOLDENMATCH_AUTOCONFIG_ARROW_NATIVE`	`1` (on)	Keep auto-config on the arrow-native path instead of coercing the arrow input to polars. Auto-forced off (warn-once per feature) for the sub-features that still assume a polars target (throughput tier, cross-source match). `0` = always coerce arrow→polars.
`GOLDENMATCH_AUTOCONFIG_FORCE_INCLUDE` / `_FORCE_EXCLUDE`	unset	Force columns into/out of analysis.
`GOLDENMATCH_FIELD_GROUP_SURVIVORSHIP`	`0` (off)	Auto-detect correlated field groups for survivorship. Equivalent to `field_group_detection: true` in `golden_rules`. Off = byte-identical to prior behavior. Group promotions are visible in the lineage `golden_records` section, `explain --cluster`, and the MCP `lineage` tool.
`GOLDENMATCH_AUTOCONFIG_SAMPLE_STRATEGY`	stratified	`random` forces random sampling.
`GOLDENMATCH_AUTOCONFIG_ZERO_LABEL_COMMIT`	`1`	Zero-label commit tiebreaker. `0` = legacy.
`GOLDENMATCH_AUTOCONFIG_ZERO_LABEL_STABILITY`	`0`	Perturbation re-runs (slow, experimental).
`GOLDENMATCH_MULTISOURCE_AUTOCONFIG`	`1` (on)	Multi-source-aware auto-config. `0`/`false`/`disabled` = legacy single-source.
`GOLDENMATCH_BLOCKING_PAIRS_PER_ROW`	`50`	Candidate-pairs-per-row budget the blocking selector targets.
`GOLDENMATCH_BLOCKING_MAX_RATIO`	`0.95`	Skip blocking keys above this cardinality ratio.
`GOLDENMATCH_BLOCKING_MIN_RATIO`	`0.01`	Minimum cardinality ratio for a blocking key.
`GOLDENMATCH_BLOCKING_CARDINALITY_SCALER`	Chao1	`observed` uses sample-observed cardinality.
`GOLDENMATCH_BLOCKING_DEGENERATE_THRESHOLD` / `_MAX_AVG_BLOCK_SIZE`	`0.95` / `10000`	Degenerate-blocking guard.
`GOLDENMATCH_BLOCKING_PRUNE_PASSES` / `_PASS_MIN_WEAKPOS`	`0` / `1`	Weak-positive multi-pass pruning.
`GOLDENMATCH_QUALITY_AWARE_BLOCKING`	`0`	Add fuzzy passes for noisy columns (GoldenCheck door #1).
`GOLDENMATCH_FD_NEGATIVE_EVIDENCE`	`0`	FD-driven negative evidence (GoldenCheck door #3).
`GOLDENMATCH_FACILITY_NAME_NE`	`0`	On company/location-mode data (shared phone/address/company demoted because the sharers don’t co-agree on the person name), add the person full name (first+last) as `token_sort` negative evidence so distinct colleagues at one facility don’t fuse. Opt-in; only fires when facility mode is detected, so person datasets (e.g. Febrl) are unaffected even when set.
`GOLDENMATCH_QUALITY_GATED_REVIEW`	`0`	Downgrade auto-merge on low-quality records.

Probabilistic Fellegi-Sunter (advanced)

A type: probabilistic matchkey trains EM once, then its candidate pairs are scored by one of four paths, chosen automatically — you rarely set these by hand, but this is the map when you need to:

bucket (default) — memory-bounded field-hash buckets; the #1798-safe route for static/multi_pass blocking on the default backend.
external-blocks — memory-bounded scorer for strategy-generated candidates (lsh/ann/learned/canopy/sorted_neighborhood) that buckets can’t re-derive from field hashes.
batched — the legacy per-block path. Reached by GOLDENMATCH_FS_DEFAULT_BUCKET=0 or an explicit non-default backend. Eager build_blocks, so memory grows with dataset size.
fused — a single Arrow-native FFI call (block + score + dedup + cluster) under est-RSS pressure on an artifact-free config; 2–19× leaner peak RSS. See Fused kernel auto-routing.

Within whichever path is chosen, the per-block math runs on the native Rust FS kernel by default (GOLDENMATCH_FS_NATIVE), falling back to the vectorized numpy scorer (GOLDENMATCH_FS_VECTORIZED) and then the scalar loop. Every knob below is an escape hatch or an advanced tuning lever — the defaults are the recommended path. Path routing

Variable	Default	Effect
`GOLDENMATCH_FS_DEFAULT_BUCKET`	`1` (on)	Route `static`/`multi_pass` FS on the default backend to the memory-bounded bucket scorer (the #1798 OOM fix). `0` = legacy batched path (eager `build_blocks`, memory grows with N) — the literal “legacy” hatch; also disables the external-blocks route.
`GOLDENMATCH_MATCH_FUSED`	on	Whole-stage fused match kernel (weighted and FS). A covered FS run routes here under est-RSS pressure on an artifact-free config. `0`/`false`/`off` kill-switch. See Fused kernel auto-routing.
`GOLDENMATCH_AUTOCONFIG_ROUTE_PROBABILISTIC`	`0`	Route zero-config auto-config to the probabilistic (FS) matchkey builder instead of the weighted one.
`GOLDENMATCH_FS_ROUTE_MIN_ROWS`	`500`	Small-N floor on FS routing: a probabilistic-shaped dataset with fewer rows stays on the robust weighted path (FS EM is data-starved at small N — it commits a RED config and its discrete-level comparison under-merges fuzzy-close variants). `0` disables the floor (route purely by shape). Every dataset that validated the FS-default win sits far above the floor, so routing there is unchanged; only tiny corpora (KG entity sets, small demos) fall back.

Scoring engine (native vs numpy vs scalar)

Variable	Default	Effect
`GOLDENMATCH_FS_NATIVE`	on (`auto`)	Native Rust FS block scorer — the reference/answer under `auto`. NE-bearing matchkeys score natively when the wheel advertises `FS_SUPPORTS_NE` (`goldenmatch-native` ≥ 0.1.15). `0`/`false`/`off` = the reproducible numpy fallback.
`GOLDENMATCH_FS_BUCKET_NATIVE`	`1` (on)	Use the batched native scorer inside the bucket route. `0` = the per-block native loop (byte-identical parity hatch). Only gates the batched call; per-block nativeness still follows `GOLDENMATCH_FS_NATIVE`.
`GOLDENMATCH_FS_VECTORIZED`	on	Vectorized (`rapidfuzz.cdist` N×N) block scoring (~9×). `0` = scalar per-pair loop. String scorers only — `embedding`/`record_embedding` are matrix-only and always run vectorized.
`GOLDENMATCH_FS_BATCH_ROWS`	`256`	Small-block coalescing cap for the batched vectorized scorer (amortizes per-call FFI/marshal overhead). Sweep knee; raising past it stops helping.
`GOLDENMATCH_FS_VEC_MAX_ELEMS`	`50000000`	Max N×N matrix elements before a block is split — the vectorized-scorer memory guard against a single mega-block.
`GOLDENMATCH_FS_WORKERS`	`min(16, cpu)`	Thread-pool size for the batched FS scorer (both kernels release the GIL). `1` forces the deterministic sequential reference loop.
`GOLDENMATCH_DISTRIBUTED_FS_TRAIN_ROWS`	`200000`	EM training-sample row cap on the distributed FS path.

Model & math

Variable	Default	Effect
`GOLDENMATCH_FS_AUTOCONFIG_V2`	`1` (on)	FS auto-config v2 comparison-set + blocking curation. `0` = legacy field set.
`GOLDENMATCH_FS_DOMAIN_COMPARATORS`	`0` (off)	Admit `date` columns with the magnitude-aware `date_diff` scorer (day-distance bands: a full-year DOB gap is a weak partial) instead of `levenshtein`. Scale-neutral (just a new scorer — same blocks/pairs/EM/clustering). Default off is byte-identical. Spec: `docs/superpowers/specs/2026-07-23-fs-domain-comparators-design.md`.
`GOLDENMATCH_FS_TF_ADJUSTMENT`	`0` (off)	Winkler term-frequency adjustment on skewed categorical fields (name/string/geo/zip): an exact agreement on a rare value out-weights one on a common value (“Zelinski” > “Smith”). Targets the over-merge regime (historical_50k precision below Splink). Stays on the native FS path (scale-safe); default off is byte-identical.
`GOLDENMATCH_FS_STRIP_HONORIFICS`	`0` (off)	Append the `strip_honorifics` transform to name comparison fields, so a title/rank token leaked into a name field (“Sir”, “Baronet”, “Bt.”) stops carrying match weight (a honorific-only value becomes missing, not an empty-string agreement). Reaches the over-merge that `GOLDENMATCH_FS_TF_ADJUSTMENT` can’t on historical_50k (F1 +0.0108, precision +0.0245 in the A/B). Regnal numerals are kept. Default off is byte-identical.
`GOLDENMATCH_FS_CALIBRATED`	`linear`	`posterior` = true match-probability calibration (frontier-neutral — ranks pairs identically, so it can’t change F1; it makes the score a real probability with a meaningful 0.5 boundary).
`GOLDENMATCH_FS_MONOTONIC`	`warn`	Non-monotonic EM-weight handling. `warn` (default) = detect + warn, don’t modify (Splink posture). `enforce`/`1`/`on` = isotonic (PAV) repair. `0`/`off` = silent (no detection).
`GOLDENMATCH_FS_MISSING`	per-dataset	Missing-value semantics (#1846). `unobserved` = textbook FS (a missing value is absence of evidence); `disagree` = evidence against a match. Env overrides `mk.missing`; by default auto-config picks per-dataset from profiled null rates.
`GOLDENMATCH_TF_NAME_WEIGHTING`	`1` (on)	Data-driven term-frequency down-weighting for `name_freq_weighted_jw` fields (shared with the weighted path). `0` = static-census behavior.
`GOLDENMATCH_FS_EM_SAMPLE_ROWS`	`100000`	Cap on the frame rows used to build EM-training blocks on the bucket route (scoring re-partitions internally, so only EM training sees the sample). Bounds the FS memory peak at scale; byte-identical F1 when `height <= cap`. `0` restores full-frame EM.
`GOLDENMATCH_FS_EM_BLOCK_SLIM`	`1` (on)	Project the EM-training block frame to `__row_id__` + the blocking-key columns before `build_blocks` (EM reads only those). Cuts the EM `build_blocks` memory peak; byte-identical output. `0` restores full-width blocks (the parity oracle).
`GOLDENMATCH_FS_EM_AGG_BLOCKS`	`1` (on)	Build the EM-training blocks as compact `int64` row-id arrays via one `group_by().agg()` per pass (no per-block frames) — supersedes the block-slim lever, killing the per-block-object floor. Byte-identical output on `static`/`multi_pass`. `0` restores the frame path.
`GOLDENMATCH_FS_OUT_OF_CORE`	`0` (off)	Opt-in single-box out-of-core FS path for datasets past the ~40M in-memory wall: the prepared frame spills to a DuckDB file and blocks stream one group at a time (bounded peak). Reached via `dedupe_to_parquet(..., out_dir=...)` on a single-probabilistic-matchkey `static`/`multi_pass` config, or the scoring branch inside `dedupe_df`. Byte-parity with the in-memory route absent oversized blocks.
`GOLDENMATCH_FS_OOC_ARROW_CLUSTER`	`1` (on)	Within the out-of-core path, keep the scored pairs as an Arrow `PAIR_STREAM` table (never `list[tuple]`), dedup them with the native `dedup_pairs_arrow` kernel, and cluster with the Rust `build_clusters_arrow_native` Union-Find — so pairs never enter Python as objects and no `dict[int,dict]` is built. `0` restores the `list[tuple]` + Python `build_clusters` path (rollback). Distinct from the in-memory `GOLDENMATCH_FS_ARROW_STREAM`.
`GOLDENMATCH_FS_OOC_DEBUG`	`0` (off)	Print a per-phase progress line for the out-of-core FS scorer (load+index, and per blocking pass: block-map build time, scan+score wall, block-row count), so a long-running ≥25M streaming leg shows live progress instead of a blank spinner. Output-invariant.
`GOLDENMATCH_FS_MAX_PASS_PAIRS`	`300000000`	Global total candidate-pair budget for probabilistic-blocking auto-config. Bounds the emitted passes so a low-cardinality pass can’t compound into a megablock (the OOM/over-generation gate).
`GOLDENMATCH_FS_ARROW_STREAM`	`0` (off)	Opt the eligible single-matchkey FS bucket dedupe path into incremental per-bucket Arrow pair-stream accumulation (banks the exclude-set + scoring-accumulation peak). Byte-identical clusters.
`GOLDENMATCH_FS_COLUMNAR_CLUSTER`	`0` (off)	B2c (#1811): thread the Arrow pair stream STRAIGHT to the columnar cluster path (`build_clusters_columnar` over `_columnar_pairs_df`) on the eligible single-FS-matchkey in-memory bucket dedupe, so the driver-resident `all_pairs` Python `list[tuple]` is never built during scoring → clustering — the accumulator that runs to hundreds of millions of tuples before clustering starts at 14M (the late-stage OOM of #1811). The in-memory analogue of `GOLDENMATCH_FS_OOC_ARROW_CLUSTER`; superset of `GOLDENMATCH_FS_ARROW_STREAM` (also drops `matched_pairs`). Clusters are NOT byte-identical (the FS bucket pipeline is ~0.1%-nondeterministic run-to-run regardless). The post-cluster `scored_pairs` list is deduped columnar + shed above `GOLDENMATCH_FS_SCORED_PAIRS_MAX` (#2006).
`GOLDENMATCH_FS_SCORED_PAIRS_MAX`	`50000000`	B2c (#2006): cap on how many deduped pairs the FS columnar-cluster path materializes into `DedupeResult.scored_pairs`. Above the cap the driver-resident `list[tuple]` is SHED (empty list + `scored_pairs_shed=True`) — the last O(pairs) accumulator #1811 left; `clusters`/`golden`/`dupes`/`unique` are unaffected, only the steward-facing raw pair list drops, and never silently (the marker). `0` disables the cap (always materialize). Scoped to `GOLDENMATCH_FS_COLUMNAR_CLUSTER`; the weighted columnar lane is unchanged.
`GOLDENMATCH_FS_REQUIRE_POSITIVE_EVIDENCE`	`1` (on)	Require strictly positive net evidence (summed match weight `W > 0`) to link a pair in linear mode — drops `W <= 0` pairs the min-max normalization would otherwise map onto a score `>= 0.5`, killing the scale-growing FS over-merge. `0` restores the legacy emit-at-neutral behavior. Linear-only (posterior calibration already folds the prior into the 0.5 Bayes cut); the wasm/DuckDB/Postgres surfaces pass `false`.

Other advanced (scorer / golden / cluster — not FS-specific)

Variable	Default	Effect
`GOLDENMATCH_NOISE_AWARE_SCORERS`	`1` (on)	Auto-upgrade `token_sort`→`jaro_winkler` on noisy text. `0` = legacy.
`GOLDENMATCH_NOISE_AWARE_TARGET`	`jaro_winkler`	Override the noise-aware target scorer.
`GOLDENMATCH_GOLDEN_STRATEGY_STRICT`	`0`	Strict golden survivorship.
`GOLDENMATCH_GOLDEN_TUNER_MIN_CORRECTIONS` / `GOLDENMATCH_NE_TUNER_MIN_CORRECTIONS`	—	Memory re-tuning floors.
`GOLDENMATCH_CLUSTER_SPLIT_EDGE_BUDGET`	∝ n_rows	Oversized-cluster auto-split work budget.
`GOLDENMATCH_COLUMNAR_PIPELINE`	`0`	Columnar pipeline experiment (Phase A opt-in).

Servers, security, storage, LLM/embedding

Variable	Default	Effect
`GOLDENMATCH_MCP_TOKEN` / `_API_TOKEN` / `_AGENT_TOKEN` / `_WEB_TOKEN`	unset	Bearer tokens; required on non-loopback binds.
`GOLDENMATCH_API_CORS_ORIGINS`	unset	REST CORS allow-list.
`GOLDENMATCH_ALLOWED_ROOT`	unset	Opt-in path sandbox root.
`GOLDENMATCH_DATABASE_URL` / `GOLDENMATCH_IDENTITY_DSN`	unset	DB connection / migration DSN.
`GOLDENMATCH_SAIL_IDENTITY_ID_SCHEME`	`h1`	Identity record-id scheme on the Sail (Spark Connect) tier.
`GOLDENMATCH_PREPARED_RECORD_STORE_DIR` / `_PERSIST`	unset / `0`	Prepared-record store path / persistence.
`GOLDENMATCH_LLM_MODEL`	provider default	Override the LLM scorer model.
`GOLDENMATCH_EMBEDDING_PROVIDER` / `GOLDENMATCH_INHOUSE_MODEL`	unset	Embedding provider / in-house model id.
`GOLDENMATCH_GPU_MODE` / `_GPU_ENDPOINT` / `_GPU_API_KEY`	auto	GPU embedding mode / remote endpoint / key.
`GOLDENMATCH_LLAMA_GGUF`	unset	Path to a local GGUF embedding model for the `model="llama"` / `"llama:/path.gguf"` backend — offline embeddings, no cloud/torch (`pip install goldenmatch[llama]`). Benchmarks: `scripts/bench_llama_embedder.py`, `scripts/bench_llama_product_matching.py`.
`GOLDENMATCH_LLM_BASE_URL`	`https://api.openai.com/v1`	OpenAI-compatible base URL for the LLM scorer — point at a local llama.cpp / llamafile server for offline, free LLM judging (also honors `OPENAI_BASE_URL`).
`GOLDENMATCH_WEAKNESS_LLM`	`0` (off)	Opt-in: let the `config_weaknesses` tool phrase its one-paragraph `summary_plain` with an LLM (needs `OPENAI_API_KEY`/`ANTHROPIC_API_KEY`). Only a compact structured digest of the findings (ids/severities/short labels) is sent — never raw rows or data. The structured findings themselves are always deterministic and never depend on the LLM; off = deterministic template summary.
`GOLDENMATCH_WEAKNESS_LLM_MODEL`	`gpt-4o-mini` (OpenAI) / `claude-3-5-haiku-latest` (Anthropic)	Model id for the `GOLDENMATCH_WEAKNESS_LLM` summary, per detected provider.
`GOLDENMATCH_UDF_IMPORTS`	unset	Snowflake UDF import asset directory.
`GOLDENMATCH_SYNC_STREAMING_THRESHOLD` / `_STREAMING_BLOCK_WORKERS`	`500000` / auto	Postgres streaming-store thresholds.
`GOLDENMATCH_ANALYTICS`	`0` (off)	Opt-in anonymous, PII-free usage analytics to PostHog (needs `POSTHOG_API_KEY`). Off = nothing collected.

Not a GOLDENMATCH_* var but relevant: set OPENAI_API_KEY or ANTHROPIC_API_KEY for LLM scoring and the LLM auto-config fallback, and POLARS_SKIP_CPU_CHECK=1 on Windows to skip the Polars import-time CPU probe.

Backends and scale

The measured numbers behind each backend choice.

Scale envelope

The block-size failure modes that matter more than the backend.

Distributed routing

The planner decides scoring, clustering, and golden routing per stage, keyed off driver RAM (not cluster total). The lint_routing / explain_routing MCP tools surface these decisions and flag overrides that force a slow path; a slow-path override at scale refuses unless allow_slow_path=true.

Single box

With no Ray cluster connected, every stage runs in-memory / in-process. There is nothing to distribute to, so the projections below do not apply.

Driver-RAM projection

When a cluster is present, each materializing stage distributes only when its working set exceeds the driver-RAM budget (available_ram_gb × 0.6):

Clustering / WCC distributes when the projected edge set (estimated_pair_count × 16 bytes) exceeds the budget. At 100M rows the ~1.76 GB edge set fits, so WCC runs in-memory (scipy) — forcing the distributed path there is a multi-hour tail.
Scoring and golden distribute when the row-frame estimate (n_rows × 512 bytes) exceeds the budget. The same data therefore distributes scoring on a 48 GB-driver cluster but keeps it in-memory on a 256 GB box.

Overrides

distributed_routing.* config pins and the legacy GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD env var are honored but linted. CLUSTERING_THRESHOLD=0 forces distributed WCC regardless of the projection; at scale the linter flags it ERROR and the runtime refuses unless allow_slow_path=true.

​Recommended setups

​Local / interactive (up to ~500K rows)

​Single box at scale (1M–200M rows, 16-core / 64 GB class)

​Distributed (Ray cluster, 50M+ rows)

​Native acceleration

​GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE

​GOLDENMATCH_NATIVE_RAYON_MIN_PAIRS

​GOLDENMATCH_NATIVE_SKETCH_RAYON_MIN_ROWS

​Backend selection

​ANN blocker backend (embedding/semantic blocking)

​Distributed pipeline

​Fast path at scale (distribute only what doesn’t fit)

​Prep & scoring perf opt-ins

​Fused kernel auto-routing

​Auto-config & planning

​Config-suggestion (healer) knobs

​Servers, security & storage

​Gotchas that cost a day

​Throughput tier (sketch-then-verify)

​Enabling it

​What it does

​Posture and honest reporting

​Measured throughput (benchmark)

​Scope

​Configuration precedence

​Full environment-variable reference

​Everyday & scale

​Distributed sub-knobs

​Perf opt-ins & diagnostics

​Auto-config & blocking (advanced)

​Probabilistic Fellegi-Sunter (advanced)

​Servers, security, storage, LLM/embedding

Backends and scale

Scale envelope

​Distributed routing

​Single box

​Driver-RAM projection

​Overrides

Recommended setups

Local / interactive (up to ~500K rows)

Single box at scale (1M–200M rows, 16-core / 64 GB class)

Distributed (Ray cluster, 50M+ rows)

Native acceleration

`GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE`

`GOLDENMATCH_NATIVE_RAYON_MIN_PAIRS`

`GOLDENMATCH_NATIVE_SKETCH_RAYON_MIN_ROWS`

Backend selection

ANN blocker backend (embedding/semantic blocking)

Distributed pipeline

Fast path at scale (distribute only what doesn’t fit)

Prep & scoring perf opt-ins

Fused kernel auto-routing

Auto-config & planning

Config-suggestion (healer) knobs

Servers, security & storage

Gotchas that cost a day

Throughput tier (sketch-then-verify)

Enabling it

What it does

Posture and honest reporting

Measured throughput (benchmark)

Scope

Configuration precedence

Full environment-variable reference

Everyday & scale

Distributed sub-knobs

Perf opt-ins & diagnostics

Auto-config & blocking (advanced)

Probabilistic Fellegi-Sunter (advanced)

Servers, security, storage, LLM/embedding

Distributed routing

Single box

Driver-RAM projection

Overrides