Tuning & opt-ins - Golden Suite

This is the one place that lists every runtime knob GoldenMatch reads, native acceleration, backend selection, the distributed pipeline, the prep/scoring perf opt-ins, and the full GOLDENMATCH_* environment-variable table. For each, you get what it does, its default, and when to set it (or leave it alone). If you are tuning a large run and just want the answer, jump to Recommended setups. If something ran far slower than you expected, read Gotchas that cost a day first. Every item there is a real footgun.

Most GoldenMatch knobs are environment variables, not config fields, because they switch a code path (native vs Python, in-memory vs distributed) rather than describe your data. Data-shape settings, matchkeys, blocking keys, golden rules, live in Configuration. This page is the operational layer on top of that.

Recommended setups

You almost never need to set anything for correctness: auto-config and the planner choose sane defaults. These blocks are about speed at scale and reproducibility, not behavior.

Local / interactive (up to ~500K rows)

Set nothing. Zero-config is the supported path. If you want the compiled kernels for a measurable speed-up on the scoring/clustering hot loops, install the optional native wheel:

pip install goldenmatch[native]

With the wheel present, the default GOLDENMATCH_NATIVE=auto already runs every component that has a kernel natively (native is the reference; pure-Python is the lossy fallback). You do not need to set GOLDENMATCH_NATIVE=1.

Single box at scale (1M–200M rows, 16-core / 64 GB class)

# Make a missing/broken native wheel a LOUD error instead of a silent slow fallback.
export GOLDENMATCH_NATIVE=1
# Turn on the Polars-native address-normalize fast path (skips the per-row Python plugin).
export GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE=1
# Reproducible runs: don't read cross-run auto-config memory.
export GOLDENMATCH_AUTOCONFIG_MEMORY=0
# These two are already ON by default; set them explicitly in a bench harness for the record.
export GOLDENMATCH_BUCKET_SLIM_PROJECTION=1
export GOLDENMATCH_GOLDEN_SLIM_MULTIDF=1
# Windows only: skip the Polars CPU probe that can hang at import.
# export POLARS_SKIP_CPU_CHECK=1

The planner picks backend=bucket for this lane automatically when the native kernel is enabled. This is the exact env used by the quality-invariant scale ladder (1K → 100M), so it is the validated single-box recipe.

Setting backend="ray" does NOT distribute the run. On its own it falls back to an in-memory path that materializes the whole frame on the driver. Real distribution needs GOLDENMATCH_DISTRIBUTED_PIPELINE=2 plus a Ray Dataset input, see Distributed pipeline and Gotchas. At single-box scale, prefer bucket.

Distributed (Ray cluster, 50M+ rows)

pip install goldenmatch[ray]
export GOLDENMATCH_DISTRIBUTED_PIPELINE=2     # Phase 5 streaming: the real distributed path
export GOLDENMATCH_ENABLE_DISTRIBUTED_RAY=1   # let the planner pick ray for backend selection
export GOLDENMATCH_NATIVE=1
# Multi-node only: a SHARED scratch the WCC checkpoints round-by-round (node-local breaks cross-node reads).
export GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH=gs://your-bucket/rc_scratch

Phase 5 has a hard contract: a Ray Dataset input with a global __row_id__ column, and it writes golden records to output_path rather than returning an in-memory clusters dict. The recall-complete path (block-shuffle scoring + randomized-contraction WCC) is default-on and was validated end-to-end at 100M rows on a real 5-node GCP cluster (9.2 min, 20M clusters recovered exactly, 0.36 GB driver RSS). See Distributed pipeline for the full contract. At single-box scale, prefer bucket. Only reach for a Ray cluster when the data genuinely won’t fit one box.

Native acceleration

Rust is the reference implementation (as of the 2026-07 reference-mode change). The Rust/PyO3 kernel is the source of truth for GoldenMatch’s hot-path primitives; pure-Python is the lossy fallback used only when a native wheel isn’t installed (the wheel ships as pip install goldenmatch[native]). A central gate (core/_native_loader.py) selects per component. GOLDENMATCH_NATIVE (core/_native_loader.py):

Value	Behavior
`auto` (default, or unset)	Run native wherever the component’s kernel symbol exists on the wheel: native is the reference. Pure-Python is the fallback for a missing wheel/symbol.
`1`	Require native. Raises `RuntimeError` at the first kernel call if the wheel isn’t importable. Use this on scale/bench runs so a missing wheel fails loudly instead of running 100x+ slower in the Python fallback.
`0`	Force the pure-Python fallback everywhere. Use for parity debugging or if you suspect a kernel bug.

Under auto, native runs for every component that has a kernel symbol. clustering, block_scoring, pairs, featurize, hashing, autoconfig, sketch, pprl_bloom, and the multimodal perceptual hashes. Each cleared bit-/ULP-level parity against the Python reference, so the fallback is output-identical (golden-vector verified). (The former per-component allowlist, _GATED_ON, is retained in the loader as sign-off documentation but no longer governs auto.) The sketch component covers both MinHash/LSH (#1081) and SimHash-over-embeddings (#1090) band hashing (~29x faster; the single source of truth across Python/Rust/TS, so semantic blocking runs on the Rust sketch-core kernel by default). Fellegi-Sunter block scoring is native-authoritative by default as of goldenmatch 2.6.0 (GOLDENMATCH_FS_NATIVE=0 opts out to the reproducible numpy path); it is scoped to the probabilistic path only. The autoconfig component is the shared, pyo3-free decision core (goldenmatch-autoconfig-core) behind auto-config, the planner, column classifier (with the adaptive 1 − 1/√n identifier-cardinality floor), the sample→full pair-count extrapolation (the corrected ratio² + Chao1 estimate), and the adaptive decision thresholds (the sparse-match floor and the per-type exact-matchkey floor). It runs native under auto and is byte-identical to the pure-Python oracle (golden-vector parity across Python / native / wasm-TS). The same crate compiles to WebAssembly so the TypeScript surface consumes the identical decision logic, one source of truth, zero cross-surface drift. When to set =1: any large or benchmarked run where a silent Python fallback would be a disaster. The cost of a missing wheel is enormous (the scoring kernel is the dominant stage), and auto hides it. When to leave it auto: normal local use. If the wheel is installed you already get native; if it isn’t, you get working Python.

`GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE`

Default off. When set to 1, address-normalize transforms run as a Polars-native expression instead of the per-row Python refdata plugin. On address-heavy data at scale the per-row plugin is a real bottleneck in the prep stage; the native path removes it.

Set it (=1) when you have address columns and are running at scale. Output is parity-tested on representative inputs.
Leave it off for small runs, or if you hit an address-canonicalization edge case (it is opt-in precisely because one integration-level cluster-count drift hasn’t been fully root-caused).

`GOLDENMATCH_NATIVE_RAYON_MIN_PAIRS`

Default 20000000. Candidate-pair count per kernel call below which the native block-scorer runs in the calling thread instead of fanning out to rayon. This exists to avoid a futex-park stall observed on some 8-core Linux CPUs (issue #688). The Python caller already parallelizes across buckets, so the sequential path loses nothing on small calls. Leave it at the default unless you are reproducing #688; 0 forces rayon always, a huge value forces sequential always.

`GOLDENMATCH_NATIVE_SKETCH_RAYON_MIN_ROWS`

Default 10000. Row count below which the native MinHash/LSH batch sketcher (#1081) runs in the calling thread instead of fanning out to rayon, the same #688 calling-thread-vs-fanout guard, applied to the sketch kernel. The SimHash batch sketcher (#1090) shares this threshold. Relevant whenever the sketch kernel runs native, which is the default under auto since #1090 (set GOLDENMATCH_NATIVE=0 to force the Python path), or under GOLDENMATCH_NATIVE=1. 0 forces rayon always, a huge value forces sequential always.

`GOLDENMATCH_ER_MODEL_DIR`

Default ~/.cache/goldenmatch/er-matcher. Directory where the local ER-matcher LLM boost caches its downloaded GGUF weights. Set it to relocate the cache, e.g. to a shared volume, a larger disk, or a read-only mount that ships the weights pre-seeded. The pinned model is sha256-verified on resolve regardless of location, so a relocated (or externally populated) cache is checked the same way as the default one.

Backend selection

Same pipeline, different execution engine. Auto-config picks one from your row count; you can override.

Backend	Range	How it works
`polars` (default)	< 500K rows	In-memory Polars. Fastest per record.
`bucket`	1M–200M, single box	Per-bucket native scoring. The validated single-box path on 16c/64GB. Planner picks it when native is enabled.
`chunked`	5M+, memory-constrained box	Streaming reader + cross-chunk join + block-keyed index. The 4c/16GB fallback.
`duckdb`	500K–50M	Out-of-core pair store, spills to disk so RAM isn’t the ceiling.
`ray`	≥ 50M, distributed	Per-block remote tasks. See the distribution caveat below: `backend="ray"` alone does not stream.

Force one in config or on the CLI:

config = gm.GoldenMatchConfig(backend="duckdb")

goldenmatch dedupe big.csv --backend ray
goldenmatch dedupe big.csv --chunked

Auto-selection knobs:

Variable	Default	Effect
`GOLDENMATCH_PLANNER_BUCKET`	`1`	The v3 planner may pick `bucket`. Set `0` to opt the bucket backend out.
`GOLDENMATCH_DUCKDB_SCORE_DB`	unset (in-memory)	Path for the DuckDB pair store to spill to disk. Point it at scratch with room when running DuckDB at 50M+.

The old single-threshold GOLDENMATCH_AUTOCONFIG_BACKEND / GOLDENMATCH_AUTOCONFIG_BACKEND_THRESHOLD env overrides were removed in v2.0. Backend selection is now the v3 planner’s job, it decides from (ComplexityProfile, RuntimeProfile). To force a backend, set it explicitly on GoldenMatchConfig(backend=...) or the CLI --backend.

ANN blocker backend (embedding/semantic blocking)

The embedding ANN blocker (ANNBlocker) resolves one of three inner-product backends per index build: native HNSW → FAISS → numpy. Native HNSW is the optional goldenmatch-hnsw wheel, a pure-Rust IndexHNSWFlat with zero C dependencies (no FAISS); FAISS (goldenmatch[embeddings]) is exact IndexFlatIP; numpy is the always-available all-pairs fallback. Scores are the raw inner product on the HNSW/FAISS paths and cosine on numpy, identical when vectors are L2-normalized (the default embedder output). In auto mode HNSW is chosen only where it wins: a large corpus and a small neighbor count per probe. Below that gate, brute force is both faster and exact, so small indexes (and the retrieve-all pattern behind the persistent VectorIndex) stay on the exact path unchanged.

Variable	Default	Effect
`GOLDENMATCH_ANN_BACKEND`	`auto`	Force a backend: `auto` (size-gated HNSW→FAISS→numpy), `hnsw` (native, ignores the size gate), `faiss`, or `numpy`. Degrades gracefully when the named library is absent.
`GOLDENMATCH_ANN_HNSW_MIN`	`4096`	In `auto`, the minimum corpus size for HNSW. Below it, the exact backend is used (HNSW’s win is asymptotic).
`GOLDENMATCH_ANN_HNSW_MAX_K`	`512`	In `auto`, the maximum `top_k` for HNSW. Above it (retrieve-nearly-all), the exact backend is used instead.
`GOLDENMATCH_ANN_HNSW_M`	`16`	HNSW graph degree (`M`): max neighbors per node on upper layers (`2M` on layer 0). Higher = better recall, more memory.
`GOLDENMATCH_ANN_HNSW_EF_CONSTRUCTION`	`200`	Candidate-list size during index build. Higher = better graph quality, slower build.
`GOLDENMATCH_ANN_HNSW_EF_SEARCH`	`64`	Candidate-list size during search (raised to at least `top_k`, capped at the corpus size). Higher = better recall, slower query.

goldenmatch-hnsw is an optional wheel, installed separately (pip install goldenmatch-hnsw) exactly like FAISS. Without it, the ANN blocker uses FAISS if present, else the numpy fallback. No behavior change.

Distributed pipeline

This is the knob most people get wrong, so read the contract before you run a 100M job. GOLDENMATCH_DISTRIBUTED_PIPELINE selects how run_dedupe_pipeline_distributed (the Ray Dataset entry point) executes:

Value	Phase	What actually happens
unset (default)	Phase 2 “cheat-line”	Calls `take_all()` on the input and runs `dedupe_df()` in memory on the driver. Back-compat default. Does not distribute scoring.
`1`	Phase 4 scaffold	Clustering/golden stages are Ray-ready, but auto-config and scoring still materialize on the driver. Kept for compatibility.
`2`	Phase 5 streaming	The real distributed path: scoring → clustering → golden → write all stay Ray Datasets. No `take_all()` on the input.

Phase 5 (=2) contract: different from in-memory dedupe:

Input must carry a global __row_id__ column. Without it, each partition synthesizes local IDs that collide and the weakly-connected-components step merges unrelated clusters.
Output is written to output_path as parquet golden records. It does not return an in-memory clusters dict the way dedupe_df does.
It still materializes two things on the driver: the ~5K-row auto-config sample, and cluster aggregates for golden-field synthesis. These are the last “cheat-lines” being retired.

GOLDENMATCH_ENABLE_DISTRIBUTED_RAY (default 0) is separate: it only lets the v3 planner pick ray during backend selection. It does not turn on the streaming pipeline. Explicit backend="ray" always works without it. The planner stopped auto-picking ray after the Distributed Plan v1 stack failed the 5M kill criterion against the bucket backend, so ray is opt-in. Recall-complete clustering at scale (#844, shipped 2026-06-11). Phase 5’s default path now block-shuffles scoring so true duplicates co-locate, then runs a relational randomized-contraction weakly-connected-components pass that clusters correctly across input-partition boundaries. This replaced two earlier WCC implementations that died at 100M (a driver head-wedge and a Ray streaming-executor deadlock). It is default-on and validated end-to-end at 100M (9.2 min, 20M clusters exact). On a multi-node cluster the WCC checkpoints each round to GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH, which must be a shared path (e.g. gs://...). A node-local scratch breaks the cross-node parquet reads and now raises rather than silently under-merging.

Only scoring needs distribution at this scale; clustering usually doesn’t. The scored pair set is small (~(id_a, id_b, score) per match), so at 100M rows it’s ~1.8 GB, it fits one node. Clustering therefore routes to the in-memory scipy connected-components pass (seconds) automatically whenever the pairs fit driver RAM, and only falls back to the distributed WCC (a multi-hour tail) when they genuinely don’t. You don’t need to set GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD for this, it’s the default.

Correlated survivorship (field_groups and conditional field_rules) does not run on the distributed or Sail paths. The golden builder that resolves lock-step groups and conditional predicates runs on the driver. If your config includes field_groups or list-form field_rules, the Ray Phase-5 streaming pipeline (GOLDENMATCH_DISTRIBUTED_PIPELINE=2) and the Sail backend will refuse with a clear error rather than silently mis-merging. Run those configs on the in-memory path.

Distributed sub-knobs (only relevant on a real Ray cluster):

Variable	Default	Effect
`GOLDENMATCH_DISTRIBUTED_BLOCK_SHUFFLE`	`1` (on)	Block-shuffle scoring so duplicates co-locate before clustering. The recall-complete path. `0` restores the legacy per-partition `local_cc` (under-merges across partitions at scale).
`GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH`	unset	Scratch directory the randomized-contraction WCC checkpoints to. Must be shared (e.g. `gs://...`) on a multi-node cluster or the run raises.
`GOLDENMATCH_DISTRIBUTED_WCC`	`two_phase`	WCC algorithm for `build_clusters_distributed`’s own callers (the at-scale Phase-5 path forces `randomized_contraction` and ignores this). `two_phase` is chain-safe; avoid `pointer_jump` (can deadlock).
`GOLDENMATCH_DISTRIBUTED_WCC_SEED`	unset	Seed for the randomized-contraction affine hash (reproducible cluster IDs across runs).
`GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD`	unset (memory-aware)	Pair count above which clustering goes distributed. Unset (default): memory-aware: clustering stays on the fast driver-side scipy.csgraph path whenever the scored pair set fits available driver RAM, and only distributes when it genuinely won’t. Set an explicit integer to force a pair-count boundary either way.
`GOLDENMATCH_DISTRIBUTED_GOLDEN_THRESHOLD`	`5000000`	Multi-member row count above which golden-record building goes distributed.
`GOLDENMATCH_DISTRIBUTED_ON_ERROR`	`raise`	Worker failure policy for the distributed scoring path. `raise` (default) fails the run when a partition cannot score, its standardization fails, or a blocking-key build fails — all of which otherwise contribute FEWER matches while the run reports success, and none of which Ray can retry once the exception is swallowed. `skip` restores the old lossy behaviour and logs at ERROR; the dropped count is not aggregated to the driver, so treat it as an escape hatch rather than a mode.
`GOLDENMATCH_DISTRIBUTED_SCORE_NUM_CPUS`	`2`	CPUs requested per Ray scoring task.
`GOLDENMATCH_DISTRIBUTED_SCORE_PROJECT`	`1` (on)	Project to scoring-relevant columns before the block-shuffle so it moves narrow blocks, not full records (#957). Output-invariant. `0` disables.
`GOLDENMATCH_DISTRIBUTED_SHUFFLE_PARTS`	`min(256, cpu×4)`	Block-shuffle partition count. Raise it to shrink per-partition blocks so more `_score` tasks fit the object-store budget (relieves `_score` `ResourceBudget` backpressure). Co-location stays correct at any value.
`GOLDENMATCH_DISTRIBUTED_OP_RESERVATION`	unset (Ray default ~0.5)	Ray Data per-op object-store reservation ratio. Lower it (e.g. `0.2`) to hand the running `_score` op more object-store budget when it’s `ResourceBudget`-backpressured below CPU capacity (idle workers, object store not full). Tune at scale.
`GOLDENMATCH_DISTRIBUTED_SCORE_CONCURRENCY`	unset	Explicit `concurrency=` on the `_score` map_batches. Pins the task-pool size once blocks are narrow; the object-store reservation/shuffle-parts knobs above govern the actual `ResourceBudget` cap.

On a Ray cluster, exporting these four in your shell does not reach the run. ray submit hands the driver a fresh shell, so a value set around the submit never arrives. Pass them as flags on the driver script instead — --score-num-cpus, --score-concurrency, --score-project / --no-score-project, --op-reservation, --shuffle-parts — which travel in the argv the submit already ships. The engine reads all four at call time, so a driver that sets them after importing goldenmatch.distributed is fine. scripts/quality_invariant_scale.py carries these flags and records the effective values in its output JSON under score_knobs.

If the distributed score stage leaves workers idle (MapBatches(_score): … [backpressured:tasks(ResourceBudget)] with the object store well under capacity), _score is reservation-capped, not CPU-bound. Lower GOLDENMATCH_DISTRIBUTED_OP_RESERVATION (→ 0.2) and/or raise GOLDENMATCH_DISTRIBUTED_SHUFFLE_PARTS, then re-measure. The optimal pair is cluster-shaped. (Shrinking Ray Data’s target_max_block_size is not a safe lever: _score keeps each co-located partition whole via batch_size=None, so a split block would split a group and under-score.)

You bring the Ray cluster, GoldenMatch ships the pipeline, not the bootstrap. The GCP recipe is in docs/distributed-ray-cluster-setup.md. At single-box scale the bucket backend is validated through 200M and is simpler, use it unless the data won’t fit one box.

Fast path at scale (distribute only what doesn’t fit)

The single most expensive distributed-tuning mistake is distributing a stage that fits in memory. “The job is big” does not mean “every stage must be distributed.” Ask it per stage:

Stage	Fits one node at 100M?	Fast path	Slow trap
Scoring	No: 100M rows	Distributed across workers, native kernel on (`GOLDENMATCH_NATIVE=1` + `goldenmatch-native` installed on every node)	pure-Python (cluster installs plain `goldenmatch`); ~6 of 80 CPU if the shuffle blocks are oversized
Clustering (WCC)	Usually: far smaller than the records, but not small — 609.4M edges at 100M, ~81 GB driver peak measured	In-memory driver-side `scipy.csgraph` connected-components (~60s)	the distributed randomized-contraction WCC: its O(log N) `gs://`-checkpoint rounds are a multi-hour tail when forced on a pair set that fits
Golden	depends	distributed groupby, or skip it if you only need cluster membership	building golden you then discard

Concrete: don’t force the distributed WCC on a pair set that fits. By default GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD is unset and the routing is memory-aware, clustering stays on the fast in-memory scipy.csgraph path whenever the scored pair set fits available driver RAM, and only distributes when it genuinely won’t. At 100M the scored pair set measures 609.4M pairs and the in-memory route’s driver peak measures ~81 GB, so the default keeps clustering in-memory on a 128-256 GB head and distributes on a 64 GB one. Set an explicit integer only if you want to force a hard pair-count boundary either way:

export GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD=2000000000  # force in-memory WCC below 2B pairs

Setting it to 0 does the opposite: it forces the slow distributed WCC on everything. Recover a result without re-running. The Phase-5 connected-components clustering is independent of the WCC algorithm, so a completed run’s cluster assignments (written to gs:// via the assignments_output_path hook) are the result. If a run finishes but the output is lost, or you want to re-score, download the assignments and score them against your ground truth on a 64 GB box, no cluster, no re-run. The 100M group_by for that scoring OOMs a 16 GB box, so run it on a large runner, not your laptop.

Prep & scoring perf opt-ins

These trade nothing for speed/memory at scale and are safe to leave at their defaults. Two are already on.

Variable	Default	Effect	When to change
`GOLDENMATCH_BUCKET_SLIM_PROJECTION`	`1` (on)	After bucket scoring, drop columns downstream stages don’t need (smaller per-bucket frames). Measured −3.8 GB peak RSS at 10M, output invariant.	Set `0` only if a custom downstream consumer needs a column it drops.
`GOLDENMATCH_GOLDEN_SLIM_MULTIDF`	`1` (on)	Slim internal columns before the multi-DataFrame golden build.	Rarely; `0` restores the wider frame.
`GOLDENMATCH_IDENTITY_BATCH_FINGERPRINT`	`1` (on)	Compute no-PK identity hashes in one Arrow batch instead of per row (2.6x on the fingerprint step).	`0` restores per-row, for parity debugging.
`GOLDENMATCH_IDENTITY_WRITE_PIPELINE`	`1` (on)	psycopg pipeline mode for the per-record identity write path (`resolve_clusters` absorb/merge). Streams `upsert`/`emit_event`/`add_edge` statements without a per-statement network round-trip: the dominant cost of a re-resolve against a remote Postgres (Cloud SQL), where a single transaction removed the fsync but not the round-trips. Postgres-only; no-op on SQLite.	`0` restores per-statement writes, for parity debugging.
`GOLDENMATCH_IDENTITY_SQLITE_BATCH`	`1` (on)	Wrap the SQLite identity resolve writes in explicit transactions instead of per-statement autocommit. The SQLite connection is opened `isolation_level=None`, so before this every INSERT paid a WAL sync on its own (~6 statements per cluster). Measured on a 200k-row store: ~750 µs/statement autocommit vs ~30–90 µs inside a transaction (8–25x). SQLite-only; no-op on Postgres/Mongo.	`0` restores the per-statement autocommit path, for parity debugging.
`GOLDENMATCH_IDENTITY_SQLITE_BATCH_SIZE`	`10000`	Statements per SQLite transaction inside `bulk_writes` (when batching is on). The batch commits and re-opens every N writes so the WAL cannot grow without bound on a multi-million-row resolve: the memory-bounded half of the fix. Non-positive or unparseable values fall back to `10000`.	Lower to cap WAL growth harder; raise to commit less often.
`GOLDENMATCH_IDENTITY_BULK`	`1` (on)	Route brand-new clusters through the bulk write fast-path in `resolve_clusters` (staging table + `executemany` + `INSERT..SELECT..ON CONFLICT`) instead of the per-cluster row loop, on both SQL backends (Postgres COPY / SQLite staging). Cut a 40k-singleton SQLite `emit_singletons=True` resolve ~47× (185.9s → 3.95s). Store contents are byte-identical to the per-row path.	`0` restores the per-row `upsert_*` loop, for parity debugging.
`GOLDENMATCH_IDENTITY_BULK_FLUSH_ROWS`	`250000`	Source-record rows accumulated before the bulk fast-path flushes a batch. Caps the accumulator (and per-flush frame) at O(batch) so the write side never stacks a second O(N) frame-residency term on the prep. Small/medium resolves stay a single flush (unchanged behaviour). Non-positive or unparseable values fall back to `250000`.	Lower to bound write-side memory harder on very large resolves; raise to flush less often.
`GOLDENMATCH_IDENTITY_INITIAL_LOAD`	`0` (off)	Opt-in initial-load fast path for a from-empty Postgres build (`IdentityStore.initial_load_writes`). Drops the secondary indexes, routes the bulk writes to a direct COPY into the real tables (no temp staging, no `INSERT..SELECT`, no `ON CONFLICT`), then rebuilds the indexes and `ANALYZE`s. Sound only from empty tables of brand-new clusters, so the store gates on emptiness and it is a no-op on a non-empty store. Postgres-only.	`1` enables it; leave `0` for incremental re-resolves.
`GOLDENMATCH_IDENTITY_INITIAL_LOAD_UNLOGGED`	`0` (off)	Under the initial-load fast path, also load the tables `UNLOGGED` (skip WAL) and `SET LOGGED` after. Only safe when the build is re-runnable from source, since an `UNLOGGED` table is truncated on crash and `SET LOGGED` rewrites it. Ignored unless `GOLDENMATCH_IDENTITY_INITIAL_LOAD` is on and engages.	`1` enables the WAL skip; leave `0` for a durable load.
`GOLDENMATCH_ENSEMBLE_KERNEL`	`1` (on)	The `ensemble` scorer rides the bucket fast path (per-pair + the float64 `_score_block_vec` vec lane + the native `score_one` id 12 arrow kernel), measured 1.47× faster end-to-end on Febrl3 with byte-identical recall. `0` (`off`/`false`) is the kill-switch → the historical float32 `find_fuzzy_matches` decline path. The earlier default-on attempt regressed on name-scorer matchkeys (`name_freq_weighted_jw` / `given_name_aliased_jw`: neither vec nor native, so they ran the O(N²) per-pair Python loop, 19× slower than `find_fuzzy_matches`); that is now fixed at the source by a perf guard in `_resolve_fast_path` that declines the fast path whenever a field would force per-pair Python.	Leave on. Set `0` only to reproduce pre-2026-07-21 output byte-for-byte. See `docs/superpowers/specs/2026-07-21-ensemble-kernel-measurement.md`.

Diagnostic-only: these can SLOW the run:

Variable	Default	Effect
`GOLDENMATCH_BUCKET_DEBUG`	`0`	Print per-bucket prep / kernel / post-filter timing for `backend=bucket`. Output-invariant, near-zero cost, but it’s a debug aid, not a speed-up.
`GOLDENMATCH_CLUSTER_DEBUG`	`0`	Print a per-step timing split for the DRIVER-SIDE clustering stage: pull pairs to driver / derive id universe / map ids to dense index / connected components / build output table. Output-invariant; one env read plus a `perf_counter` per step. Use it when that stage dominates a distributed run — at 100M rows it measured 707s, of which connected components is 315.9s (44.7%) and pulling pairs to the driver 245.5s (34.7%) — it is not the pure-transport stage an earlier ~70s estimate for the algorithm implied. On a Ray cluster set it in the cluster yaml’s `docker run_options`, not in a shell around `ray submit`: the driver runs through a fresh `docker exec` and would never see the export.
`GOLDENMATCH_CANDIDATE_COUNT_MAX_BLOCKS`	`10000`	Max blocks for which scoring counts candidate pairs. Above it the loop is skipped — counting materialises each block — and `ScoringProfile.candidates_compared` stays 0 with `candidates_counted=False`. Raise it (e.g. `1000000000`) only to make an auto-config diagnostic answerable: it buys a serial pass over every block. Garbage or negative values fall back to the default.
`GOLDENMATCH_TRANSFORM_DEBUG`	`0`	Print the prep-transform arrow-lane split (`pl.from_arrow` / `_do_transform` / `to_arrow`) to size the bridge vs the transform compute. Output-invariant, near-zero cost; a debug aid, not a speed-up.
`GOLDENMATCH_PREP_STAGED_COLLECT`	`0`	Collect prepared records in stages (memory-management experiment). Can be slower; leave off unless diagnosing prep RSS.
`GOLDENMATCH_STANDARDIZE_STAGED`	`0`	Staged standardization pipeline (experimental). Same caveat.

Fused kernel auto-routing

The pipeline can route covered slow-path runs to the fused Arrow-native kernels, byte-identically, with no config change. There are two seams with very different maturity: golden routing is a broad default-on win; match routing is a narrow capacity-survival path that fires only under memory pressure.

Variable	Default	Effect	When to change
`GOLDENMATCH_GOLDEN_FUSED`	on	Golden seam auto-routing (default-on, byte-identical). Tries the fused Arrow-native golden kernel on every covered slow-path config, reaching golden output at ~2x lower peak RSS. Declines to the classic builder (unchanged output) when the config is uncovered, the native kernel is absent, full `ClusterProvenance` lineage is requested, or the run is fast-path-eligible. Surfaced on `DedupeResult.golden_fused_used`.	`0` / `false` / `off` is the kill-switch (forces the classic golden builder everywhere).
`GOLDENMATCH_MATCH_FUSED`	on	Match seam auto-routing: a narrow capacity-survival path. A controller post-step short-circuits the whole match stage to the fused match kernel under a memory-pressure gate AND a narrow config profile: `auto_split=False`, no identity/adaptive/memory/llm_boost/confidence_majority/full-provenance, not across-files-only, and a covered matchkey: weighted (single-key or multi-pass) or Fellegi-Sunter (single-key or multi-pass; the FS path fits EM inside the short-circuit and was wired in #1892). Zero-config auto-config commits `auto_split=True` and explicit configs bypass the controller, so it stays dormant unless you configure that artifact-free profile and hit the RSS gate. When it fires it is a capacity mode that sheds `scored_pairs`/`review_pairs` + cluster-confidence/lineage to survive; cluster membership + golden stay byte-identical. Marked `DedupeResult.match_fused_capacity_mode` + telemetry `rule_name` `+fused_match_post_step`. Measured 2–19× leaner peak RSS on covered FS shapes.	`0` / `false` / `off` is the kill-switch.
`GOLDENMATCH_FUSED_PRESSURE_FRACTION`	`0.65`	Fraction of available RAM the estimated classic-path peak RSS must exceed before match routing’s pressure gate fires. Only relevant to the pressure-gated match seam.	Lower to make the capacity path trip earlier; raise toward `1.0` to make it trip only under extreme pressure.
`GOLDENMATCH_FUSED_RSS_SCALE` / `_BYTES_PER_PAIR` / `_BYTES_PER_CELL` / `_BLOCK_CONCURRENCY`	`0.763` / `64` / `40` / `4`	Calibration coefficients for the est-peak-RSS model that drives the match pressure gate. Defaults are fit against measured runs.	Only when re-calibrating the est-RSS model against a new box/workload; leave alone otherwise.

Auto-config & planning

Variable	Default	Effect	When to set
`GOLDENMATCH_AUTOCONFIG_MEMORY`	`1` (on)	Read/write cross-run auto-config memory at `~/.goldenmatch/autoconfig_memory.db`.	Set `0` in CI or any run that must be reproducible and not influenced by prior runs.
`GOLDENMATCH_AUTOCONFIG_DETERMINISTIC`	`0` (off)	Removes every wall-clock cut from the auto-config search: the controller’s `max_seconds` stop and the per-indicator time budgets, leaving the iteration caps as the only stopping rule. With it off, how far the search gets (and so which config is committed) depends on how fast and how loaded the host is, which makes any pin or gate over that config unstable: red on a slow runner with no code change, green on a fast one while masking a real change. Runtime becomes unbounded in exchange.	Set `1` alongside `GOLDENMATCH_AUTOCONFIG_MEMORY=0` for anything that pins, blesses or gates auto-config output. Leave off for interactive runs, which want a bounded wait more than a bit-reproducible search.
`GOLDENMATCH_PLANNING_EFFORT`	`normal`	Iteration budget for auto-config: `fast` / `normal` / `thinking` / `einstein`. Higher = more thorough, slower; `thinking`+ measure real full-frame pair counts.	`fast` to speed up config on huge inputs; `thinking` when auto-config picks a poor blocking key. `normal` is byte-identical to the historical default.
`GOLDENMATCH_CONFIG_LINT`	`warn`	Pre-flight config linter on the resolved config (zero-config or explicit). `warn` logs data-shape findings and runs; `strict` raises `ConfigLintError` on an error finding before the run; `off` skips. Findings attach to `DedupeResult.lint_findings`.	`strict` in CI/production to fail fast on a degenerate config (e.g. a blocking key that will OOM). `off` to silence.
`GOLDENMATCH_AUTOCONFIG_LLM`	`0`	Let an LLM propose a config when heuristics are exhausted on a RED profile. Needs `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`.	Opt in only when heuristic auto-config can’t find a healthy config and you accept the API cost.
`GOLDENMATCH_AUTOCONFIG_INDICATOR_BUDGET`	unset	Set `fast` to skip the expensive complexity indicators.	Speeding up config on very large inputs.
`GOLDENMATCH_AUTOCONFIG_FORCE_INCLUDE` / `_FORCE_EXCLUDE`	unset	Comma-separated columns to force into / out of auto-config analysis.	Rescue a column auto-config drops, or exclude a noisy one, without writing a full explicit config.
`GOLDENMATCH_FIELD_GROUP_SURVIVORSHIP`	`0` (off)	`1` enables auto-detection of correlated field groups for survivorship. When on, auto-config proposes lock-step field groups (address, person name, contact) so related columns are always promoted from the same source record. Explicitly declared `field_groups` in YAML are always honored regardless of this flag. Off by default — output is byte-identical to prior behavior when unset. Group promotions are visible in the lineage `golden_records` section, `explain --cluster`, and the MCP `lineage` tool (see Surfacing the group decision).	Set when you want auto-config to detect field groups without writing explicit `field_groups` YAML. Equivalent to `field_group_detection: true` in the `golden_rules` config block.
`GOLDENMATCH_PERCEPTUAL_AUTOCONFIG`	`0` (off)	`1` enables multimodal-ER media-as-evidence wiring (ADR 0022). When on, auto-config detects columns holding fixed-width hex perceptual hashes (a 16-char image pHash from `core.perceptual.phash_hex`, a 96-char rotation/crop-aware radial-variance profile from `radial_hex`, or a multi-word audio fingerprint from `audio_fp_hex`) and appends a `phash` / `radial` / `audio_fp` matchkey, plus `perceptual` banded-hamming-LSH blocking for an image column when the committed config has nothing else to block on (the rotation-aligned `radial` feature gets no LSH blocking). Additive + idempotent + fail-open; output is byte-identical to prior behavior when unset.	Set when a dataset carries a precomputed perceptual-hash column you want treated as a match feature without writing explicit matchkey YAML.
`GOLDENMATCH_AUTOCONFIG_ROUTE_PROBABILISTIC`	`0` (off)	`1` lets zero-config auto-config route a probabilistic-shaped dataset to the Fellegi-Sunter path instead of the default exact+weighted matchkeys. The trigger fires when the emitted config has no surviving exact matchkey backed by a strong-identity column (`identifier` / `email` / `phone`) and ≥2 fuzzy fields: i.e. there’s no clean key carrying the dedup, so EM-weighted comparison wins. It delegates to `auto_configure_probabilistic_df` (which also diversifies blocking onto orthogonal stable keys). A behavior change, so default-off pending a broader regression sweep; measured lift on biographical/PII data (historical_50k F1 0.466 → 0.83, recall 0.39 → 0.75) with no regression on datasets that retain a strong key.	Opt in on no-strong-identifier, error-heavy datasets (historical/biographical records, dirty PII) where exact matching under-recalls. Leave off when your data has a reliable id/email/phone.

If auto-config raises ControllerNotConfidentError, that’s by design at ≥100K rows on a RED profile. It’s refusing to run a noise-result dedupe. See Passing an explicit config for recovery (the in-repo docs/explicit-config.md has the failing-sub-profile table). There are also a number of experimental blocking and probabilistic knobs (GOLDENMATCH_BLOCKING_*, GOLDENMATCH_FS_*, GOLDENMATCH_NOISE_AWARE_*, GOLDENMATCH_QUALITY_AWARE_BLOCKING, GOLDENMATCH_FD_NEGATIVE_EVIDENCE). These are research/tuning levers, default-safe, and listed in the full table below. Most users never touch them.

Config-suggestion (healer) knobs

These tune the healer: the review_config config-suggestion loop, which is now wired into the default dedupe_df pipeline as an advisory surface (a free controller signal gates a cheap attach; see GOLDENMATCH_SUGGEST_ON_DEDUPE). The healer needs goldenmatch[native] and degrades gracefully without it. You almost never need to change these, the defaults are the values the suggester gym selected.

Variable	Default	Effect	When to set
`GOLDENMATCH_SUGGEST_ON_DEDUPE`	on	Master switch for the default-pipeline healer surface (Python `dedupe_df` / CLI / servers and the TS/WASM `dedupe(rows, { ... })`): attaches advisory `suggestions` when a free trigger fires. Set `0` to disable the trigger + attach entirely (the explicit `suggest=`/`heal=` opt-ins and `review_config` still work).	Set `0` to opt out of the advisory default-run surface.
`GOLDENMATCH_HEAL_STEP_CAP`	`5`	Max healer iterations — the coarsest cost lever (each step is a full dedupe plus up to `SUGGEST_MAX_VERIFY` verification re-runs). Clamped to >= 1; invalid input falls back to the default.	Lower to cap healer cost on large inputs; raise to allow more repair steps.
`GOLDENMATCH_HEAL_MIN_HEALTH_GAIN`	unset (disabled)	Marginal-gain early stop (#1404): stop the heal loop once an iteration’s cluster-health gain falls below this float. Unset = run up to `HEAL_STEP_CAP`; `0.0` stops on the first non-improving step.	Set a small positive value to stop early once suggestions stop paying off.
`GOLDENMATCH_SUGGEST_MAX_VERIFY`	`8`	Max suggestions verified by a full pipeline re-run before the remaining tail passes through unverified. Each verified candidate is one extra re-run. Values `< 1` are clamped to `1`.	Lower to cut healer cost; raise to verify more candidates end-to-end.
`GOLDENMATCH_SUGGEST_HEALTH`	`cohesion`	Self-verify health proxy the gate uses to keep only non-worsening suggestions. `cohesion` (precision-sensitive `min_edge` × saturating coverage) is the default since 2026-06; `legacy` restores the older `matched_rate × avg_conf − HHI` proxy.	`legacy` only to reproduce pre-2026-06 suggestion behavior. The bake-off showed `cohesion` lifts live recovery 0.15 → 0.54 with no real-pair net-negatives.
`GOLDENMATCH_SUGGEST_COHESION`	`min_edge`	Cohesion statistic: `min_edge`, `mean_bottomk_edge`, or `edge_below_cutoff_fraction`.	Leave alone; `min_edge` won the bake-off.
`GOLDENMATCH_SUGGEST_COVERAGE_CAP`	`0.50`	Coverage-saturation cap for the cohesion proxy (matched-rate at which coverage saturates to 1.0).	Leave alone; `0.50` is the safe operating point: tighter caps let a real-pair net-negative through.
`GOLDENMATCH_SUGGEST_VERIFY`	on	The self-verify gate. Set `0` to return raw kernel suggestions unfiltered (no no-net-negative guarantee).	Diagnostics only: to see what the kernel would propose before the gate.
`GOLDENMATCH_SUGGEST_FULL_DIST`	`0` (off)	Source the kernel’s score distribution from a threshold-0 diagnostic run so the dip (`lower_threshold`) rule sees the full pre-threshold tail.	Set `1` when you want the dip rule to fire on configs whose threshold is well above the score valley.
`GOLDENMATCH_SUGGEST_RECALL_MIN_SHED`	`0.02`	Min recall the tighten-key rule must shed before it will even consider vetoing a matchkey.	Leave alone; raising it makes the recall-shed guard more conservative.
`GOLDENMATCH_SUGGEST_RECALL_COH_ABS`	`0.15`	Max cohesion gain that still counts as “small” when weighing a recall shed.	Leave alone; the gym-selected operating point.
`GOLDENMATCH_SUGGEST_RECALL_RATIO`	`2.0`	Min cohesion-gain-per-recall-shed ratio required to justify shedding recall.	Leave alone; higher = demands more cohesion payoff per unit recall lost.
`GOLDENMATCH_TOKEN_BLOCKING`	`1` (on)	Let auto-suggest propose the `token` blocking strategy (DF-pruned inverted-index token blocking) for free-text columns. The strategy itself is always available from an explicit config; this flag only controls whether blocking auto-suggest may choose it.	Leave on. Only columns that measure as free text are eligible, so names and addresses keep their prefix/soundex keys; on Amazon-Google the token plan reaches 0.982 blocking recall against the previously committed key’s 0.041. It shipped default-OFF under #2488 on the reading that committing it took that benchmark’s F1 to 0.0000 — that was wrong: nothing ever set `strategy="token"`, so `build_token_blocks` never ran and the 0.0000 came from whatever config was committed instead (#2717). Set to `0` to force auto-suggest back off.

Servers, security & storage

When you expose any server (MCP, REST, A2A, web UI) on a non-loopback host, it is fail-closed: it refuses to start without the matching bearer token.

Variable	Used by	Notes
`GOLDENMATCH_MCP_TOKEN`	MCP server	Required on non-loopback binds. `/.well-known/` card stays public.
`GOLDENMATCH_MCP_ALLOW_PUBLIC`	MCP server	Opt-out of the fail-closed default: set `1`/`true` to intentionally run an open, unauthenticated public MCP server (e.g. a showcase deployment) without `GOLDENMATCH_MCP_TOKEN`. Unset/`0` = fail closed (a non-loopback bind without a token is refused). A set `GOLDENMATCH_MCP_TOKEN` still takes precedence.
`GOLDENMATCH_API_TOKEN`	REST API	Required on non-loopback binds (all except `/health`).
`GOLDENMATCH_AGENT_TOKEN`	A2A agent	Required on non-loopback binds (except `/health` + agent card).
`GOLDENMATCH_WEB_TOKEN`	Web UI	Required on non-loopback binds (except `/api/v1/healthz`).
`GOLDENMATCH_API_CORS_ORIGINS`	REST API	Comma-separated allow-list; replaces wildcard CORS.
`GOLDENMATCH_ALLOWED_ROOT`	All file-taking surfaces	Opt-in path sandbox. When set, every user-supplied path must resolve under this root or the call is rejected. Recommended for deployed/shared instances (e.g. the Railway MCP server sets it to `/data`). Unset = no containment (correct local-first default).
`GOLDENMATCH_MCP_MAX_UPLOAD_BYTES`	MCP server (inline upload)	Max decoded size for `file_content` / `upload_dataset` inline uploads. Default `67108864` (64 MB). Over the cap the call is rejected: pass a public `http(s)://` URL as `file_path` for larger datasets.
`GOLDENMATCH_MCP_UPLOAD_TTL`	MCP server (inline upload)	Age in seconds after which an inline-uploaded temp file is reaped. Default `86400` (24 h). Uploaded paths are ephemeral scratch; re-upload if you need one older than this.
`GOLDENMATCH_MCP_SESSION_MAX`	MCP server (session state)	Max number of per-MCP-session dedupe/match runs kept in memory so the stateful tools (`list_clusters`, `get_cluster`, `get_golden_record`, `explain_match`, `evaluate`, `export_results`, `match_record`, `find_duplicates`) can read a run made earlier in the same session. Default `64`; least-recently-used sessions evict past the cap.
`GOLDENMATCH_MCP_SESSION_TTL`	MCP server (session state)	Seconds a stored session run stays readable (anchored to when the run was made, not last access). Default `3600` (1 h); past it the stateful tools return a clean “no run loaded” error until you re-run `agent_deduplicate`/`match_sources`.

Storage / persistence:

Variable	Default	Effect
`GOLDENMATCH_DATABASE_URL` / `GOLDENMATCH_IDENTITY_DSN`	unset	Postgres/SQLite connection for the identity graph + review queue / Alembic migrations.
`GOLDENMATCH_PREPARED_RECORD_STORE_DIR`	unset	Directory for the disk-backed prepared-record store.
`GOLDENMATCH_PREPARED_RECORD_STORE_PERSIST`	`0`	`1` keeps the prepared-record store across calls instead of cleaning it up.
`GOLDENMATCH_VECTOR_INDEX_DIR`	`.goldenmatch_vectors`	Default on-disk directory for the persistent semantic `VectorIndex` (embeddings + records + manifest) when a path isn’t passed explicitly. The index survives across runs/processes so embeddings aren’t rebuilt every run.

Product analytics (opt-in): Anonymous usage analytics: off by default. Emits only when GOLDENMATCH_ANALYTICS is on and a POSTHOG_API_KEY is set, so a user’s machine stays silent unless they opt in (the project’s own hosted services set the flag server-side). PII-free by construction: only coarse scalars (scale bands, backend, version) ever leave, never record values, column names, or paths. Full collection list + guarantees in packages/python/goldenmatch/ANALYTICS.md.

Variable	Default	Effect
`GOLDENMATCH_ANALYTICS`	`0` (off)	`1`/`true`/`on` opts in to anonymous, PII-free usage events. Off = nothing is collected or sent.
`POSTHOG_API_KEY`	unset	Destination PostHog project key. Required for any event to send; absent = silent no-op.
`POSTHOG_HOST`	`https://us.i.posthog.com`	PostHog ingestion host override.

Gotchas that cost a day

Read these before a big run. Each is a real trap that produces a slow or wrong result with no error.

backend='ray' ran in-memory and never used the cluster

backend="ray" (or dedupe_df(backend="ray")) alone runs the Phase 2 cheat-line: it collects the whole frame to the driver and dedupes in memory. The cluster sits idle. To actually stream across workers you need GOLDENMATCH_DISTRIBUTED_PIPELINE=2, a Ray Dataset input with a global __row_id__, and you read golden output from output_path. Not the return value. On a multi-node cluster also set a shared GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH (e.g. gs://...). For single-box scale, use bucket, not ray.

It ran 100x slower than the docs because native silently fell back to Python

Under the default GOLDENMATCH_NATIVE=auto, if the native wheel isn’t importable (not installed, ABI mismatch, stale build) the scoring kernel silently runs in Python. Orders of magnitude slower. On any run where speed matters, set GOLDENMATCH_NATIVE=1 so a missing wheel raises instead of crawling. Confirm with python -c "import goldenmatch_native" or check goldenmatch.core._native_loader.native_available().

The prep stage dominated wall on address-heavy data

The per-row Python address-normalize plugin is a real bottleneck at scale. Set GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE=1 to use the Polars-native expression instead. It’s off by default, so you have to opt in.

A distributed 100M run took hours instead of minutes

The scored pair set fits one node on a large head (609.4M pairs at 100M, ~81 GB driver peak measured — fits 128-256 GB, not 64 GB). By default GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD is unset and routing is memory-aware, clustering stays in-memory (scipy.csgraph, ~60s) whenever the pair set fits driver RAM, so a 100M run should already keep clustering on the fast path. The multi-hour tail happens if something forces the distributed WCC (whose gs://-checkpoint rounds are slow): check you haven’t set the threshold to 0 (forces the slow path on everything) or to a value below your pair count. See Fast path at scale. Distribute the scoring (the rows don’t fit), not the clustering (the pairs do).

Results changed between runs for no reason

Cross-run auto-config memory (GOLDENMATCH_AUTOCONFIG_MEMORY=1, the default) lets a prior run influence the next. For reproducible benchmarks set GOLDENMATCH_AUTOCONFIG_MEMORY=0.

Python hung at import on Windows

Polars runs a WMI CPU probe at import that can hang on some Windows boxes. Set POLARS_SKIP_CPU_CHECK=1. (Add PYTHONIOENCODING=utf-8 if non-ASCII output mangles the console.)

Throughput tier (sketch-then-verify)

New in #1083, builds on #1080. An opt-in high-throughput path for large text corpora where you care about near-duplicate recall but can trade precision for speed. Instead of the full fuzzy/Fellegi-Sunter pipeline, it blocks with MinHash/LSH (lsh) or SimHash (simhash) and confirms candidate pairs by cheap sketch distance before any field-level scoring. Think: training data dedup, web crawl dedup, paper-corpus cleaning.

Enabling it

Pass throughput= to dedupe_df:

import goldenmatch as gm

# Float: sets recall_target (0 to 1). Adjusts LSH banding.
result = gm.dedupe_df(df, throughput=0.95)

# True: uses the default recall_target (0.95).
result = gm.dedupe_df(df, throughput=True)

# None (default): the throughput tier is off. Byte-identical to today.
result = gm.dedupe_df(df)

# ThroughputConfig: full control.
from goldenmatch import ThroughputConfig
result = gm.dedupe_df(df, throughput=ThroughputConfig(
    recall_target=0.90,
    similarity_threshold=0.75,
))

Or via environment variables:

Variable	Default	Effect
`GOLDENMATCH_THROUGHPUT`	`0` (off)	`1` enables the throughput tier with default settings.
`GOLDENMATCH_THROUGHPUT_RECALL`	`0.95`	LSH recall target (tunes banding).
`GOLDENMATCH_THROUGHPUT_SIMILARITY`	unset	Override the near-dup similarity threshold (Jaccard default `0.8`; cosine default `0.85`).

What it does

Auto-selects blocking strategy. Picks simhash (cosine/semantic) when an embedder is reachable, otherwise falls back to lsh (MinHash/Jaccard on shingles). Uses the longest text column in the frame.
Blocks with sketch buckets. Records that share at least one LSH band become candidates. The recall_target controls the banding split via the LSH probability curve.
Verifies by sketch distance only. Pairs that clear similarity_threshold are kept; the field-level fuzzy/FS scorer does not run.
Raises ThroughputNotApplicableError when the frame has no text column. Fix: pass an explicit BlockingConfig to direct it to a column.

Posture and honest reporting

The throughput tier does not measure F1 or precision. DedupeResult.throughput_posture (and the controller telemetry throughput block) carry the metrics the tier can honestly report:

Key	Meaning
`metric`	`"jaccard"` (LSH) or `"cosine"` (SimHash)
`recall_target`	The recall target you passed (tunes banding)
`similarity_threshold`	The sketch-distance threshold applied at verify
`bands`	LSH band count selected for the recall target
`rows_per_band`	Rows per band
`expected_recall`	LSH-theoretic recall at the banding: `1 - (1 - t^r)^b` where `t = similarity_threshold`, `r = rows_per_band`, `b = bands`. This is an analytic bound, not measured on your data.
`reduction_ratio`	`1 - candidate_pairs / possible_pairs`. Observed.
`candidate_pairs`	Pairs that passed blocking.
`verified_pairs`	Pairs that passed sketch-distance verification.
`notes`	Human-readable posture string.

expected_recall is the LSH-theoretic probability that a pair at exactly similarity_threshold Jaccard/cosine collides in at least one band. It is not a measured F1 or recall against ground truth. For pairs above threshold the curve is tighter; for pairs below it, collision probability drops rapidly. This is an honest bound, not a precision measurement.

Measured throughput (benchmark)

The bench-corpus-dedup harness (scripts/bench_corpus_dedup/, .github/workflows/bench-corpus-dedup.yml) measures the tier end-to-end against a real public corpus, with injected ground-truth near-dups so recall is measurable. On a slice of FineWeb (web crawl), the zero-config dedupe_df(df, throughput=…) path measured:

docs	docs/sec	MB/sec	measured recall
14,000	275	0.84	0.44
70,000	1,192	3.64	0.43

docs/sec rises with scale because the wall is dominated by a fixed ~50s of zero-config auto-config at these sizes; the sketch dedup itself is ~9s at 70k (≈7,800 docs/sec raw). Measured recall ≈ 0.43 is the honest LSH recall over a mix of exact / partial-overlap / paraphrase near-dups, the paraphrase and partial dups often do not collide in the LSH bands. (Contrast expected_recall above: the per-pair collision probability at the similarity threshold, not a recall over a real corpus, which is exactly why the bench measures recall directly.) A deterministic per-PR CI perf gate (throughput-gate job, via scripts/bench_corpus_dedup/throughput_perf_gate.py) guards regression on the machine-independent cost, candidate pairs, reduction ratio, and measured recall on a vendored offline corpus, against a committed baseline, so a change can’t quietly blow up the candidate count or drop recall.

Scale note: the published numbers are the validated 14k–70k range. 100k+ corpora currently bottleneck on the survivorship/golden stage; lifting that ceiling (and a datatrove head-to-head) are tracked follow-ups.

Scope

The throughput tier is single-node only. Distributed throughput (#1084) and the product surface (#1085) are follow-on issues. Default-off (throughput=None) is byte-identical to today in every metric.

Configuration precedence

When the same setting is reachable multiple ways, the more specific wins:

Explicit API / CLI argument (dedupe_df(..., backend=...), --backend): highest.
Config field (GoldenMatchConfig(backend=...), YAML).
Environment variable (GOLDENMATCH_*).
Built-in default (auto-config / planner choice): lowest.

So an env var sets the default behavior for a process, a config field overrides it for a run, and an explicit call argument overrides it for one call.

Full environment-variable reference

Every GOLDENMATCH_* variable the package reads (plus the one GOLDEN_* exception, GOLDEN_DIAGNOSTICS), grouped by who should care. Defaults verified against source. Most users only ever touch the recommended setups.

Everyday & scale

Variable	Default	Values	Effect
`GOLDENMATCH_NATIVE`	`auto`	`auto` / `1` / `0`	Native kernel gate. `1` requires native (raises if missing); `0` forces Python.
`GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE`	off	`1`	Polars-native address-normalize fast path.
`GOLDENMATCH_FRAME`	`arrow`	`arrow`/`polars`	Frame backend. `arrow` is the DEFAULT since v3.0.0 (measured ~36% faster end-to-end on the 100K zero-config A/B): file ingest via pyarrow, the whole engine runs on the Frame seam, and result frames are `pyarrow.Table`. Since v3.1.0 polars is an OPTIONAL dependency and the polars-free install is the FAST configuration (measured head-to-head at 500K: 7.11s polars-free vs 7.55s polars-present, identical outputs — the Rust fused kernels carry the hot paths). `polars` here requires `pip install 'goldenmatch[polars]'` (a clear error names the extra otherwise); the extra is a COMPATIBILITY surface (this classic lane, the kernel-absent golden fast replay, goldencheck cell-quality weighting), not an accelerator. Output-equivalence enforced by a differential harness + frozen fixtures.
`GOLDENMATCH_FRAME_LANE`	`1` (on)	`0`/`1`	Arrow “frame lane” for the output-writing pipeline (keeps the spine arrow through the final collect). `0` restores the classic polars shim. Ignored when `GOLDENMATCH_COLUMNAR_PIPELINE=1`.
`GOLDENMATCH_AUTOCONFIG_MEMORY`	`1`	`0`/`1`	Cross-run auto-config memory. `0` for reproducible/CI runs.
`GOLDENMATCH_AUTOCONFIG_DETERMINISTIC`	`0`	`0`/`1`	Drop the search’s wall-clock budgets so the committed config is host-independent. `1` for anything that pins or gates auto-config output.
`GOLDENMATCH_PLANNING_EFFORT`	`normal`	`fast`/`normal`/`thinking`/`einstein`	Auto-config iteration budget.
`GOLDENMATCH_DISTRIBUTED_PIPELINE`	unset	`1`/`2`	Distributed execution phase. `2` = real streaming (needs `__row_id__`, writes to `output_path`).
`GOLDENMATCH_ENABLE_DISTRIBUTED_RAY`	`0`	`0`/`1`	Let the planner pick `ray`. Does not enable streaming on its own.
`GOLDENMATCH_PLANNER_BUCKET`	`1`	`0`/`1`	Allow the planner to pick `bucket`.
`GOLDENMATCH_BUCKET_DEFAULT`	`1` (on)	`0`/`off`/`false`/`no`	Default in-memory bucket FS route on runs up to 750K rows (the #1798-safe path). Opt out to force the legacy per-block path. Distinct from `GOLDENMATCH_FS_DEFAULT_BUCKET`, which routes the FS bucket path itself.
`GOLDENMATCH_ATTRIBUTE_DEMOTION`	`1` (on)	`0`/`1`	Demote an EXACT matchkey on a shared group/list/facility value (shared clinic phone, campaign id, facility NPI) when its large shared-value groups do not co-agree on the person name. `0` restores the pre-2.7.0 behavior.
`GOLDENMATCH_DISCRIMINATIVE_VETO`	`1` (on)	`0`/`1`	The #1351 exact-key discriminative-power veto (demote a shared-locality exact key). `0` disables it.
`GOLDENMATCH_DISCRIMINATIVE_TAU`	`0.5`	float [0,1]	Co-agreement floor for the discriminative veto: an exact key whose large shared-value groups co-agree on the person name below this fraction is vetoed. Out-of-range values fall back to the default.
`GOLDENMATCH_TF_NAME_WEIGHTING`	`1` (on)	`0`/`1`	Data-driven per-dataset name-frequency downweight on `name_freq_weighted_jw` fields. `0` restores the static-census behavior.
`GOLDENMATCH_BASE_STORE`	`auto`	`auto`/`memory`/`lance`	Candidate base store for the ANN `match_one` path. `auto` picks Lance only above ~2M rows when installed.
`GOLDENMATCH_DUCKDB_SCORE_DB`	unset	path	DuckDB pair-store spill path.
`GOLDENMATCH_NATIVE_RAYON_MIN_PAIRS`	`20000000`	int	Native kernel sequential-vs-rayon threshold.
`GOLDENMATCH_NATIVE_RAYON_MIN_BLOOM_ROWS`	`10000`	int	Native PPRL bloom-filter sequential-vs-rayon threshold (row count). `0` = always parallel; a very large value = always sequential. Sibling of `_RAYON_MIN_PAIRS` (the #688-style futex-park guard).
`GOLDENMATCH_KEY_INTEGRITY_NATIVE`	off	`0`/`1`	Route the structural key-integrity certifier (`certify_key_integrity`) through the shared `key-integrity-core` Rust kernel instead of the pyarrow group_by. Opt-in and fail-open (a missing native symbol / non-JSON-safe group column falls back to pyarrow). The pyarrow path stays the default + reference: the kernel path exists for the single-Rust-owner guarantee (shared with the TS/WASM surface), not speed (it is JSON-marshaled).
`GOLDENMATCH_CONFIG_LINT`	`warn`	`warn`/`strict`/`off`	Pre-flight config linter: warn-and-run, refuse on error, or skip.
`GOLDENMATCH_THROUGHPUT`	`0` (off)	`0`/`1`	Enable the sketch-then-verify throughput tier. Equivalent to `dedupe_df(throughput=True)`.
`GOLDENMATCH_THROUGHPUT_RECALL`	`0.95`	float	LSH recall target; tunes the band/row split.
`GOLDENMATCH_THROUGHPUT_SIMILARITY`	unset	float	Override the near-dup similarity threshold (Jaccard `0.8` / cosine `0.85` when unset).
`GOLDENMATCH_AUTO_SEMANTIC_BLOCKING`	on	`0`/`1`	Auto-enable SimHash-over-embeddings blocking for text-heavy data (#1090). Default-on; a no-op when no embedder is reachable. `0` disables.
`GOLDENMATCH_SEMANTIC_BLOCKING_THRESHOLD`	`0.6`	float (0,1)	Recall threshold for auto semantic blocking; lower = more bands = higher recall. Drives the SimHash band/row split.

Distributed sub-knobs

Variable	Default	Effect
`GOLDENMATCH_DISTRIBUTED_BLOCK_SHUFFLE`	`1` (on)	Block-shuffle scoring (recall-complete path). `0` = legacy `local_cc`.
`GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH`	unset	Randomized-contraction WCC scratch. Must be shared (`gs://...`) on multi-node.
`GOLDENMATCH_DISTRIBUTED_WCC`	`two_phase`	WCC for `build_clusters_distributed`’s own callers (Phase-5-at-scale forces `randomized_contraction`). Avoid `pointer_jump`.
`GOLDENMATCH_DISTRIBUTED_WCC_SEED`	unset	Randomized-contraction hash seed (reproducible cluster IDs).
`GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD`	unset (memory-aware)	Pair count → distributed clustering. Unset default routes by driver-RAM fit; set an integer to force a hard boundary.
`GOLDENMATCH_DISTRIBUTED_GOLDEN_THRESHOLD`	`5000000`	Row count → distributed golden.
`GOLDENMATCH_DISTRIBUTED_ON_ERROR`	`raise`	Worker failure policy for the distributed scoring path. `raise` (default) fails the run when a partition cannot score, its standardization fails, or a blocking-key build fails — all of which otherwise contribute FEWER matches while the run reports success, and none of which Ray can retry once the exception is swallowed. `skip` restores the old lossy behaviour and logs at ERROR; the dropped count is not aggregated to the driver, so treat it as an escape hatch rather than a mode.
`GOLDENMATCH_DISTRIBUTED_SCORE_NUM_CPUS`	`2`	CPUs per Ray scoring worker.

Perf opt-ins & diagnostics

Variable	Default	Effect
`GOLDENMATCH_BUCKET_SLIM_PROJECTION`	`1` (on)	Drop unneeded columns after bucket scoring.
`GOLDENMATCH_GOLDEN_SLIM_MULTIDF`	`1` (on)	Slim internal columns before golden build.
`GOLDENMATCH_GOLDEN_FUSED`	on	Golden-seam fused auto-routing (default-on, byte-identical, ~2x lower peak RSS). `0` kills it. See Fused kernel auto-routing.
`GOLDENMATCH_MATCH_FUSED`	on	Match-seam fused auto-routing (weighted + Fellegi-Sunter) - a narrow capacity-survival path that fires only under memory pressure on an artifact-free config. `0` kills it. See Fused kernel auto-routing.
`GOLDENMATCH_FUSED_PRESSURE_FRACTION`	`0.65`	RAM-fraction pressure gate for the match-seam routing.
`GOLDENMATCH_FUSED_RSS_SCALE` / `_BYTES_PER_PAIR` / `_BYTES_PER_CELL` / `_BLOCK_CONCURRENCY`	`0.763` / `64` / `40` / `4`	est-peak-RSS calibration coefficients for the match pressure gate.
`GOLDENMATCH_IDENTITY_BATCH_FINGERPRINT`	`1` (on)	Batch no-PK identity hashing.
`GOLDENMATCH_IDENTITY_WRITE_PIPELINE`	`1` (on)	Pipeline the per-record identity writes on remote Postgres (#1912).
`GOLDENMATCH_IDENTITY_SQLITE_BATCH`	`1` (on)	Batch the SQLite identity resolve writes into explicit transactions (#2105). `0` = per-statement autocommit.
`GOLDENMATCH_IDENTITY_SQLITE_BATCH_SIZE`	`10000`	Statements per SQLite transaction inside `bulk_writes`; bounds WAL growth on large resolves.
`GOLDENMATCH_IDENTITY_BULK`	`1` (on)	Bulk write fast-path for brand-new clusters in `resolve_clusters` (Postgres COPY / SQLite staging). `0` = per-row loop.
`GOLDENMATCH_IDENTITY_BULK_FLUSH_ROWS`	`250000`	Records accumulated before the bulk fast-path flushes; bounds write-side memory to O(batch).
`GOLDENMATCH_IDENTITY_INITIAL_LOAD`	`0` (off)	Opt-in from-empty Postgres initial-load fast path (direct COPY, indexes rebuilt after). `1` = on; gated to empty tables.
`GOLDENMATCH_IDENTITY_INITIAL_LOAD_UNLOGGED`	`0` (off)	Also load `UNLOGGED` (skip WAL) under the initial-load path; only for re-runnable builds.
`GOLDENMATCH_BUCKET_DEBUG`	`0`	Per-bucket timing breakdown (diagnostic).
`GOLDENMATCH_CLUSTER_DEBUG`	`0`	Driver-side clustering stage timing breakdown (diagnostic).
`GOLDENMATCH_CANDIDATE_COUNT_MAX_BLOCKS`	`10000`	Block-count gate for the candidate-count loop; raise only for diagnostics (costs a serial pass).
`GOLDENMATCH_TRANSFORM_DEBUG`	`0`	Prep-transform arrow-lane bridge split (diagnostic).
`GOLDEN_DIAGNOSTICS`	`1` (on)	Anomaly diagnostics (note: no `MATCH` in the name). On states that are probably a GoldenMatch bug — a native wheel-skew slow path, an unexpected `dedupe_df`/`match_df` crash, the config linter itself crashing, or a broken native install — emit a warning with a prefilled GitHub issue URL plus a PII-safe environment report. Never fires on expected fallbacks or user errors, and sends nothing anywhere (a better error message, not telemetry). `0` disables.
`GOLDENMATCH_PREP_STAGED_COLLECT`	`0`	Staged prep collect (experimental, may slow).
`GOLDENMATCH_STANDARDIZE_STAGED`	`0`	Staged standardization (experimental, may slow).
`GOLDENMATCH_BUCKET_VEC_MIN` / `_MAX`	`32` / `2000`	Block-size band for the vectorized float64 NxN cdist path in bucket scoring.
Fellegi-Sunter routing & scoring knobs (`GOLDENMATCH_FS_*`)	:	Have their own home: see Probabilistic / Fellegi-Sunter for the path-routing map + the complete `FS_*` table.
`GOLDENMATCH_BENCH_DUMP_PAIRS`	unset	Dump per-stage pairs for benchmarking.
`GOLDENMATCH_SKIP_MATCH_LOG` / `GOLDENMATCH_MATCH_LOG_FLUSH_PAIRS`	:	Streaming match-log controls.

Auto-config & blocking (advanced)

Variable	Default	Effect
`GOLDENMATCH_AUTOCONFIG_LLM`	`0`	LLM config fallback (needs API key).
`GOLDENMATCH_AUTOCONFIG_INDICATOR_BUDGET`	unset	`fast` skips expensive indicators.
`GOLDENMATCH_AUTOCONFIG_ARROW_NATIVE`	`1` (on)	Keep auto-config on the arrow-native path instead of coercing the arrow input to polars. Auto-forced off (warn-once per feature) for the sub-features that still assume a polars target (throughput tier, cross-source match). `0` = always coerce arrow→polars.
`GOLDENMATCH_AUTOCONFIG_FORCE_INCLUDE` / `_FORCE_EXCLUDE`	unset	Force columns into/out of analysis.
`GOLDENMATCH_FIELD_GROUP_SURVIVORSHIP`	`0` (off)	Auto-detect correlated field groups for survivorship. Equivalent to `field_group_detection: true` in `golden_rules`. Off = byte-identical to prior behavior. Group promotions are visible in the lineage `golden_records` section, `explain --cluster`, and the MCP `lineage` tool.
`GOLDENMATCH_AUTOCONFIG_SAMPLE_STRATEGY`	stratified	`random` forces random sampling.
`GOLDENMATCH_AUTOCONFIG_ZERO_LABEL_COMMIT`	`1`	Zero-label commit tiebreaker. `0` = legacy.
`GOLDENMATCH_AUTOCONFIG_ZERO_LABEL_STABILITY`	`0`	Perturbation re-runs (slow, experimental).
`GOLDENMATCH_MULTISOURCE_AUTOCONFIG`	`1` (on)	Multi-source-aware auto-config. `0`/`false`/`disabled` = legacy single-source.
`GOLDENMATCH_BLOCKING_PAIRS_PER_ROW`	`50`	Candidate-pairs-per-row budget the blocking selector targets.
`GOLDENMATCH_BLOCKING_MAX_RATIO`	`0.95`	Skip blocking keys above this cardinality ratio.
`GOLDENMATCH_BLOCKING_DIVERSIFY_PAIR_BUDGET`	`1` (on)	Hold the orthogonal-anchor passes that `_diversify_probabilistic_blocking` adds (date YEAR, postcode/zip/identifier) to the same `GOLDENMATCH_BLOCKING_PAIRS_PER_ROW` budget the primary-key selector uses. Previously those passes were bounded only by the FS scorer’s per-block ROW cap, which limits per-block memory and not the WORK a pass creates: measured on person @ 100K, the birth-YEAR anchor sits at 774 pairs/row against a budget of 50 and accounts for 71,337,394 of 121,391,850 comparisons (59%) while costing zero F1 to remove, whereas the postcode anchor is 4 pairs/row and carries all the recall (-0.0269 F1 without it). Both clear the row cap, so it cannot tell them apart. Enabling takes the plan to 50,054,456 comparisons at an unchanged pairwise F1 0.9970. `0` restores the row-cap-only behaviour — the measurement is person-shaped, and the lever exists for error-heavy PII where a cheap-looking anchor may be the only thing co-blocking corrupted names.
`GOLDENMATCH_BLOCKING_MIN_RATIO`	`0.01`	Minimum cardinality ratio for a blocking key.
`GOLDENMATCH_BLOCKING_CARDINALITY_SCALER`	Chao1	`observed` uses sample-observed cardinality.
`GOLDENMATCH_BLOCKING_DEGENERATE_THRESHOLD` / `_MAX_AVG_BLOCK_SIZE`	`0.95` / `10000`	Degenerate-blocking guard.
`GOLDENMATCH_BLOCKING_PRUNE_PASSES` / `_PASS_MIN_WEAKPOS`	`0` / `1`	Weak-positive multi-pass pruning.
`GOLDENMATCH_BLOCKING_COST_AWARE`	`0`	Demote a small-fixed-domain year/date column from the sole primary blocking key when a bounded name/compound fallback exists (it stays a recall pass), so identifier-poor data doesn’t explode candidate pairs at scale. Domain-routed: skipped on bibliographic data, where a publication-year key is a legitimately strong same-year blocking signal. Stays opt-in: the 2026-08-07 OFF-vs-ON bench measured it as a byte-identical no-op on both bench shapes (the bench person fixture has a `postcode`, so a year key never becomes the sole primary there), so there is no measured evidence for a default flip: enable it only on genuinely identifier-poor data.
`GOLDENMATCH_QUALITY_AWARE_BLOCKING`	`0`	Add fuzzy passes for noisy columns (GoldenCheck door #1).
`GOLDENMATCH_FD_NEGATIVE_EVIDENCE`	`0`	FD-driven negative evidence (GoldenCheck door #3).
`GOLDENMATCH_FACILITY_NAME_NE`	`0`	On company/location-mode data (shared phone/address/company demoted because the sharers don’t co-agree on the person name), add the person full name (first+last) as `token_sort` negative evidence so distinct colleagues at one facility don’t fuse. Opt-in; only fires when facility mode is detected, so person datasets (e.g. Febrl) are unaffected even when set.
`GOLDENMATCH_QUALITY_GATED_REVIEW`	`0`	Downgrade auto-merge on low-quality records.

Probabilistic Fellegi-Sunter (advanced)

A type: probabilistic matchkey trains EM once, then its candidate pairs are scored by one of four paths, chosen automatically. You rarely set these by hand, but this is the map when you need to:

bucket (default): memory-bounded field-hash buckets; the #1798-safe route for static/multi_pass blocking on the default backend.
external-blocks. Memory-bounded scorer for strategy-generated candidates (lsh/ann/learned/canopy/sorted_neighborhood) that buckets can’t re-derive from field hashes.
batched. The legacy per-block path. Reached by GOLDENMATCH_FS_DEFAULT_BUCKET=0 or an explicit non-default backend. Eager build_blocks, so memory grows with dataset size.
fused. A single Arrow-native FFI call (block + score + dedup + cluster) under est-RSS pressure on an artifact-free config; 2–19× leaner peak RSS. See Fused kernel auto-routing.

Within whichever path is chosen, the per-block math runs on the native Rust FS kernel by default (GOLDENMATCH_FS_NATIVE), falling back to the vectorized numpy scorer (GOLDENMATCH_FS_VECTORIZED) and then the scalar loop. Every knob below is an escape hatch or an advanced tuning lever, the defaults are the recommended path. Path routing

Variable	Default	Effect
`GOLDENMATCH_FS_DEFAULT_BUCKET`	`1` (on)	Route `static`/`multi_pass` FS on the default backend to the memory-bounded bucket scorer (the #1798 OOM fix). `0` = legacy batched path (eager `build_blocks`, memory grows with N): the literal “legacy” hatch; also disables the external-blocks route.
`GOLDENMATCH_MATCH_FUSED`	on	Whole-stage fused match kernel (weighted and FS). A covered FS run routes here under est-RSS pressure on an artifact-free config. `0`/`false`/`off` kill-switch. See Fused kernel auto-routing.
`GOLDENMATCH_AUTOCONFIG_ROUTE_PROBABILISTIC`	`0`	Route zero-config auto-config to the probabilistic (FS) matchkey builder instead of the weighted one.
`GOLDENMATCH_FS_ROUTE_MIN_ROWS`	`500`	Small-N floor on FS routing: a probabilistic-shaped dataset with fewer rows stays on the robust weighted path (FS EM is data-starved at small N: it commits a RED config and its discrete-level comparison under-merges fuzzy-close variants). `0` disables the floor (route purely by shape). Every dataset that validated the FS-default win sits far above the floor, so routing there is unchanged; only tiny corpora (KG entity sets, small demos) fall back.

Scoring engine (native vs numpy vs scalar)

Variable	Default	Effect
`GOLDENMATCH_FS_NATIVE`	on (`auto`)	Native Rust FS block scorer: the reference/answer under `auto`. NE-bearing matchkeys score natively when the wheel advertises `FS_SUPPORTS_NE` (`goldenmatch-native` ≥ 0.1.15). `0`/`false`/`off` = the reproducible numpy fallback.
`GOLDENMATCH_FS_BUCKET_NATIVE`	`1` (on)	Use the batched native scorer inside the bucket route. `0` = the per-block native loop (byte-identical parity hatch). Only gates the batched call; per-block nativeness still follows `GOLDENMATCH_FS_NATIVE`.
`GOLDENMATCH_FS_VECTORIZED`	on	Vectorized (`rapidfuzz.cdist` N×N) block scoring (~9×). `0` = scalar per-pair loop. String scorers only: `embedding`/`record_embedding` are matrix-only and always run vectorized.
`GOLDENMATCH_FS_BATCH_ROWS`	`256`	Small-block coalescing cap for the batched vectorized scorer (amortizes per-call FFI/marshal overhead). Sweep knee; raising past it stops helping.
`GOLDENMATCH_FS_BUCKET_BATCH`	`1` (on)	Coalesce a bucket’s small blocks into row-capped (`GOLDENMATCH_FS_BATCH_ROWS`) batched vectorized-scorer calls on the non-native bucket path (when the native kernel is declined, e.g. `missing="disagree"`), instead of one `prob_scorer` call per block. Byte-identical clusters/partitions. `0` = per-block loop (parity escape hatch). Only affects the vectorized-numpy scorer; scalar/ensemble/embedding fallbacks stay per-block.
`GOLDENMATCH_FS_VEC_MAX_ELEMS`	`50000000`	Max N×N matrix elements before a block is split: the vectorized-scorer memory guard against a single mega-block.
`GOLDENMATCH_FS_OVERSIZED_CHUNK_PAIRS`	`4000000`	Candidate-pair budget for ONE scoring call on an un-splittable oversized block (`skip_oversized=false` and auto-split found no useful split: the #1826 hub shape). Above it the block is scored in bounded tiles instead of whole, so a single call never carries the block’s full `C(n,2)`: the vectorized lane’s dense matrix stays ~`16 × budget` bytes per field (~64 MB at the default) instead of `O(n²)`, and the native lane stops double-buffering the whole block’s surviving pairs. Same pair set and scores, ~2× the pair comparisons on that block only (cross tiles recompute the intra-group pairs they discard); emission order within the block becomes tile order. `0` = score whole (pre-#2417 parity hatch). See #1826/#2417.
`GOLDENMATCH_FS_WORKERS`	`min(16, cpu)`	Thread-pool size for the batched FS scorer (both kernels release the GIL). `1` forces the deterministic sequential reference loop.
`GOLDENMATCH_DISTRIBUTED_FS_TRAIN_ROWS`	`200000`	EM training-sample row cap on the distributed FS path.

EM training

Variable	Default	Effect
`GOLDENMATCH_FS_EM_COUNTED`	`0` (off)	Train the FS model on collapsed comparison vectors over a ~200x larger blocked-pair budget instead of the 10,000-pair sampler. `1` = on at the default 2,000,000-pair budget; any integer sets the budget. Identical vectors are counted rather than iterated, so the EM loop’s cost tracks `prod(levels + 1)` (thousands of rows) rather than the pair count: the collapse is EXACT, not an approximation, because every M-step quantity is a sum linear in how many pairs share a vector. Collapsing happens per blocking pass: two pairs are only interchangeable if the same fields were conditioned out for both, which is constant within a pass and not across them. Declines to the sampler when a per-pair override is in play (`label_pairs`, caller `pair_weights`, negative-evidence fields), since collapsing is exactly what discards which pair a vector came from. Default off is byte-identical. The distributed twin is `spark.em.train_em_distributed`.

Model & math

Variable	Default	Effect
`GOLDENMATCH_FS_AUTOCONFIG_V2`	`1` (on)	FS auto-config v2 comparison-set + blocking curation. `0` = legacy field set.
`GOLDENMATCH_FS_DOMAIN_COMPARATORS`	`0` (off)	Admit `date` columns with the magnitude-aware `date_diff` scorer (day-distance bands: a full-year DOB gap is a weak partial) instead of `levenshtein`. Scale-neutral (just a new scorer: same blocks/pairs/EM/clustering). Default off is byte-identical. Spec: `docs/superpowers/specs/2026-07-23-fs-domain-comparators-design.md`.
`GOLDENMATCH_FS_TF_ADJUSTMENT`	`0` (off)	Winkler term-frequency adjustment on skewed categorical fields (name/string/geo/zip): an exact agreement on a rare value out-weights one on a common value (“Zelinski” > “Smith”). Targets the over-merge regime (historical_50k precision below Splink). Stays on the native FS path (scale-safe); default off is byte-identical.
`GOLDENMATCH_FS_STRIP_HONORIFICS`	`0` (off)	Append the `strip_honorifics` transform to name comparison fields, so a title/rank token leaked into a name field (“Sir”, “Baronet”, “Bt.”) stops carrying match weight (a honorific-only value becomes missing, not an empty-string agreement). Reaches the over-merge that `GOLDENMATCH_FS_TF_ADJUSTMENT` can’t on historical_50k (F1 +0.0108, precision +0.0245 in the A/B). Regnal numerals are kept. Default off is byte-identical.
`GOLDENMATCH_FS_ORTHOGONAL_BLOCKING`	`auto`	Field-agnostic orthogonal-anchor blocking. Adds an additive single-field blocking pass on every well-populated, moderate-cardinality field the name/date/zip passes don’t already cover, so an orthogonal identity anchor a dataset’s classifier missed (e.g. `birth_place` profiling as `name`) still co-locates the pairs its corrupted name keys can’t. Additive passes stay EM-trained (co-locate without demoting the discriminator). `auto` (default) fires ONLY on person-shaped data (a `name` + a `date` col_type): the regime where it generalises (historical_50k B³ 0.808→0.873, febrl4 holdout +0.006), and is a no-op on bibliographic/product data (dblp_scholar −0.24, no name+date). `1` forces everywhere; `0` disables.
`GOLDENMATCH_FS_ATOMIC_NAME_BLOCKING`	`off`	Atomic-name-soundex blocking completion. Adds an additive `surname`/`first_name` soundex pass when names are blocked via a composite soundex pass but no atomic single-name pass exists (a corrupted first name otherwise breaks the composite key). Known-negative: leave off. It raises candidate/blocking recall +6pp (historical_50k 0.886→0.946), but the canonical `bench_er_headtohead` panel measures it as a regression: historical_50k pairwise F1 −0.0148 (B³ −0.0078), because the corrupted-name candidates fall below the FS threshold and the EM shift degrades the operating point (both precision and recall drop). febrl3/febrl4 no-op. Kept gated for reference. `auto` = person-shaped only; `1`/`on` = everywhere, do NOT enable except to experiment.
`GOLDENMATCH_FS_CALIBRATED`	`linear`	`posterior` = true match-probability calibration (frontier-neutral: ranks pairs identically, so it can’t change F1; it makes the score a real probability with a meaningful 0.5 boundary).
`GOLDENMATCH_FS_MONOTONIC`	`warn`	Non-monotonic EM-weight handling. `warn` (default) = detect + warn, don’t modify (Splink posture). `enforce`/`1`/`on` = isotonic (PAV) repair. `0`/`off` = silent (no detection).
`GOLDENMATCH_FS_MISSING`	per-dataset	Missing-value semantics (#1846). `unobserved` = textbook FS (a missing value is absence of evidence); `disagree` = evidence against a match. Env overrides `mk.missing`; by default auto-config picks per-dataset from profiled null rates.
`GOLDENMATCH_FS_CALIBRATE_THRESHOLD`	`0` (off)	Pick the FS link cutoff per-dataset from the training-pair score distribution (Otsu’s between-class-variance split) instead of the fixed `0.50` default. Adapts the cutoff to each dataset’s over/under-merge: raises it where namesakes pile up just below (historical_50k F1 +0.04, precision 0.72->0.93), lowers it where the data under-merges (dblp_acm F1 +0.49, recall 0.27->0.93), and leaves well-separated data unchanged (febrl3/synthetic). Default off is byte-identical.
`GOLDENMATCH_FS_REFIT_THRESHOLD`	`0` (off)	Refit the FS link cutoff from the ACTUAL scored-pair distribution (re-cluster only, no re-scoring): the health-gated threshold-refit loop. Finds the class-separating VALLEY between a false-pair band and the true-match mass, but commits it only behind two guards: a deep-valley gate (the trough must be near-empty, not a shoulder) and a cluster-shape guard (accept only if re-clustering there reduces the max cluster size). Recovers over-merge the fixed `0.50` can’t see on over-merge-prone shapes (household hard-negatives F1 0.947→1.000, co-tenant address over-merge 0.318→0.996). Was flipped ON 2026-08-02, reverted to OFF 2026-08-07 (#2387), the cluster-shape guard has a known defect: `max_candidate < max_default` is a single-outlier statistic, so it accepts a candidate that fixes one over-merged cluster while shattering many correct ones. On `ncvr_synthetic` (all gold clusters exactly size 2) it commits `0.50→0.70`, satisfies its own objective (max cluster 5→2), and still costs −0.0966 F1, which reddened the nightly `bench-suggest-quality` gate for five nights. Enable with `=1` (or `true`/`on`/`yes`/`enabled`) if your data’s dominant error mode is over-merge; default off is byte-identical to the fixed cutoff. Re-flipping needs a global accept criterion + a panel including ncvr_synthetic. Spec: `docs/superpowers/specs/2026-08-02-fs-refit-loop-design.md`.
`GOLDENMATCH_TF_NAME_WEIGHTING`	`1` (on)	Data-driven term-frequency down-weighting for `name_freq_weighted_jw` fields (shared with the weighted path). `0` = static-census behavior.
`GOLDENMATCH_FS_EM_SAMPLE_ROWS`	`100000`	Cap on the frame rows used to build EM-training blocks on the bucket route (scoring re-partitions internally, so only EM training sees the sample). Bounds the FS memory peak at scale; byte-identical F1 when `height <= cap`. `0` restores full-frame EM.
`GOLDENMATCH_FS_EM_BLOCK_SLIM`	`1` (on)	Project the EM-training block frame to `__row_id__` + the blocking-key columns before `build_blocks` (EM reads only those). Cuts the EM `build_blocks` memory peak; byte-identical output. `0` restores full-width blocks (the parity oracle).
`GOLDENMATCH_FS_EM_AGG_BLOCKS`	`1` (on)	Build the EM-training blocks as compact `int64` row-id arrays via one `group_by().agg()` per pass (no per-block frames): supersedes the block-slim lever, killing the per-block-object floor. Byte-identical output on `static`/`multi_pass`. `0` restores the frame path.
`GOLDENMATCH_FS_OUT_OF_CORE`	`0` (off)	Opt-in single-box out-of-core FS path for datasets past the ~40M in-memory wall: the prepared frame spills to a DuckDB file and blocks stream one group at a time (bounded peak). Reached via `dedupe_to_parquet(..., out_dir=...)` on a single-probabilistic-matchkey `static`/`multi_pass` config, or the scoring branch inside `dedupe_df`. Byte-parity with the in-memory route absent oversized blocks.
`GOLDENMATCH_FS_OOC_ARROW_CLUSTER`	`1` (on)	Within the out-of-core path, keep the scored pairs as an Arrow `PAIR_STREAM` table (never `list[tuple]`), dedup them with the native `dedup_pairs_arrow` kernel, and cluster with the Rust `build_clusters_arrow_native` Union-Find: so pairs never enter Python as objects and no `dict[int,dict]` is built. `0` restores the `list[tuple]` + Python `build_clusters` path (rollback). Distinct from the in-memory `GOLDENMATCH_FS_ARROW_STREAM`.
`GOLDENMATCH_FS_OOC_DEBUG`	`0` (off)	Print a per-phase progress line for the out-of-core FS scorer (load+index, and per blocking pass: block-map build time, scan+score wall, block-row count), so a long-running ≥25M streaming leg shows live progress instead of a blank spinner. Output-invariant.
`GOLDENMATCH_FS_BLOCK_SOURCE`	`auto`	Select the FS streaming block source: `sequential` (in-RAM Arrow/Rust batches: the fused FS kernel groups + scores + clusters in one call, no spill), `spill` (DuckDB-free out-of-core: per-pass scored-edge shards on disk + an external union-find that streams them back, bounds the edge term with no DuckDB dependency), `bucketed` (DuckDB-free frame-residency out-of-core: hash-bucket the prepared frame to disk by each blocking pass’s key, then score bucket-by-bucket, never a whole-frame scoring copy, so it bounds the FRAME term the `spill`/fused paths leave resident; partition-exact with `sequential`), `duckdb` (the DuckDB file-spilled out-of-core path, the past-RAM tier), `frame`/`eager` (in-memory bucket variants), or `auto` (pick by dataset). Unifies the older `GOLDENMATCH_FS_OUT_OF_CORE=1` toggle (now an alias for `duckdb`).
`GOLDENMATCH_FS_OOC_WAVE_ROWS`	`2000000`	Max resident block-rows scored per parallel wave in the in-RAM `sequential` FS scorer (the fallback path when the fused kernel is uncovered): bounds peak to one wave of buffered blocks + their pair results, not a whole blocking pass. `0`/invalid → the 2M default. Output-invariant.
`GOLDENMATCH_FS_BUCKET_SCORE_WORKERS`	`min(cpu, 8)`	Thread concurrency for the `bucketed` block source’s shard-scoring loop. Shards are independent and `score_buckets_arrow` releases the GIL (native/rapidfuzz), so scoring them concurrently fills the cores/RSS the serial loop left idle during the wall-dominant scoring phase (measured ~−19% wall at 4M on a 4-core box). `1` = serial (byte-identical to the pre-parallel path). Output-invariant: the external union-find is edge/shard-order-invariant. More workers ⇒ more shards in flight ⇒ higher peak RSS, so raise it only with headroom.
`GOLDENMATCH_FS_SPILL_FORCE_SHARD`	`0` (off)	Within the `spill` block source, force the edge-shard mechanism (per-pass shards on disk + `external_wcc_from_shards`) by SKIPPING the fused-kernel short-circuit that otherwise clusters a fused-covered config in one call. Default off means `spill` tries the fused kernel first (already bounded, no pair list) and the shard path is a fallback for uncovered configs; `1` reaches the bounded-edge path directly (for benchmarking/validation). Byte-identical partition either way.
`GOLDENMATCH_FS_MAX_PASS_PAIRS`	`300000000`	Global total candidate-pair budget for probabilistic-blocking auto-config. Bounds the emitted passes so a low-cardinality pass can’t compound into a megablock (the OOM/over-generation gate).
`GOLDENMATCH_FS_ARROW_STREAM`	`0` (off)	Opt the eligible single-matchkey FS bucket dedupe path into incremental per-bucket Arrow pair-stream accumulation (banks the exclude-set + scoring-accumulation peak). Byte-identical clusters.
`GOLDENMATCH_FS_COLUMNAR_CLUSTER`	`1` (on)	B2c (#1811/#2006, default-on, Arrow-native): thread the `score_buckets_arrow` pair table STRAIGHT into the frames-out clustering (`build_cluster_frames` + the native arrow kernel) on the eligible single-FS-matchkey in-memory bucket dedupe: Arrow end-to-end with no polars. Neither the driver-resident `all_pairs` Python `list[tuple]` NOR `build_clusters_columnar`’s legacy per-cluster / 16M-key `pair_scores` dict is ever built (the late-stage OOM + the measured ~40s clustering cost at 5M of #1811; scoring itself was already native-fast). The review band rides the Arrow table and still attaches to `review_pairs`. Superset of `GOLDENMATCH_FS_ARROW_STREAM` (also drops `matched_pairs`). Ineligible configs (multi-matchkey, across-files, semantic blocking, bench-dump, non-bucket scale backends) fall back to the list path unchanged. Clusters are NOT byte-identical (the FS bucket pipeline is ~0.1%-nondeterministic run-to-run regardless), so the flip moves the default output within that band only. Set `0`/`false` to force the legacy list path (the byte-parity oracle). Post-cluster `scored_pairs` is deduped Arrow-native + shed above `GOLDENMATCH_FS_SCORED_PAIRS_MAX` (#2006).
`GOLDENMATCH_FS_SCORED_PAIRS_MAX`	`50000000`	B2c (#2006): cap on how many deduped pairs the FS columnar-cluster path materializes into `DedupeResult.scored_pairs`. Above the cap the driver-resident `list[tuple]` is SHED (empty list + `scored_pairs_shed=True`): the last O(pairs) accumulator #1811 left; `clusters`/`golden`/`dupes`/`unique` are unaffected, only the steward-facing raw pair list drops, and never silently (the marker). `0` disables the cap (always materialize). Scoped to `GOLDENMATCH_FS_COLUMNAR_CLUSTER`; the weighted columnar lane is unchanged.
`GOLDENMATCH_FS_REQUIRE_POSITIVE_EVIDENCE`	`1` (on)	Require strictly positive net evidence (summed match weight `W > 0`) to link a pair in linear mode: drops `W <= 0` pairs the min-max normalization would otherwise map onto a score `>= 0.5`, killing the scale-growing FS over-merge. `0` restores the legacy emit-at-neutral behavior. Linear-only (posterior calibration already folds the prior into the 0.5 Bayes cut); the wasm/DuckDB/Postgres surfaces pass `false`.
`GOLDENMATCH_FS_POST_BLOCKING_U`	`0` (off)	Opt-in u-probability bias correction for bibliographic dedup: FS estimates `u` (non-match agreement) from random pairs but only scores blocked pairs, where blocking-correlated fields agree far more often: inflating a candidate-common field’s weight and driving false merges. ON deconvolves `u` from the blocked population in the final weights (`u_blocked = (blocked_rate - p_match*m) / (1 - p_match)`), deflating candidate-common fields toward weight ~0 while keeping discriminative fields. Verified NOT a default (deflates discriminative person-data fields → controller quality gate regresses); default off is byte-identical.
`GOLDENMATCH_FS_EVIDENCE_CUT`	unset (off)	Opt-in prior- and endpoint-invariant link threshold (bits): set the cut to `sigma(c + prior_w)` on the posterior axis, exactly `W >= c` where `W = sum log2(m/u)`. Decouples the operating point from the min-max envelope so a weight change (TF/honorific/u-correction) no longer relocates every pair’s decision. Implies posterior scoring; `GOLDENMATCH_FS_CALIBRATED=linear` overrides. No single `c` fits every dataset (two unsupervised selectors were refuted), so it stays a knob, default OFF (byte-identical).

Other advanced (scorer / golden / cluster: not FS-specific)

Variable	Default	Effect
`GOLDENMATCH_NOISE_AWARE_SCORERS`	`1` (on)	Auto-upgrade `token_sort`→`jaro_winkler` on noisy text. `0` = legacy.
`GOLDENMATCH_NOISE_AWARE_TARGET`	`jaro_winkler`	Override the noise-aware target scorer.
`GOLDENMATCH_GOLDEN_STRATEGY_STRICT`	`0`	Strict golden survivorship.
`GOLDENMATCH_GOLDEN_TUNER_MIN_CORRECTIONS` / `GOLDENMATCH_NE_TUNER_MIN_CORRECTIONS`	:	Memory re-tuning floors.
`GOLDENMATCH_CLUSTER_SPLIT_EDGE_BUDGET`	∝ n_rows	Oversized-cluster auto-split work budget.
`GOLDENMATCH_TRANSITIVE_POSTFLIGHT`	`0` (off)	Transitive-consistency cluster postflight. Splits clusters held together by a single weak transitive bridge: an edge whose removal leaves two ≥2-node groups where the weakest edge sits materially below the average (a false pair chaining two entities: A-B real, B-C false, C-D real ⇒ one cluster). A cluster-level fix for a failure mode pair-level threshold calibration can’t undo; the split set is reported under `report["transitive_consistency"]`. Default off is byte-identical.
`GOLDENMATCH_TRANSITIVE_WEAK_MARGIN`	`0.15`	How far below the average edge the weakest edge must sit for a cluster to count as a weak transitive bridge (larger = more conservative, splits fewer). Only consulted when `GOLDENMATCH_TRANSITIVE_POSTFLIGHT` is on.
`GOLDENMATCH_COLUMNAR_PIPELINE`	`0`	Columnar pipeline experiment (Phase A opt-in).

Servers, security, storage, LLM/embedding

Variable	Default	Effect
`GOLDENMATCH_MCP_TOKEN` / `_API_TOKEN` / `_AGENT_TOKEN` / `_WEB_TOKEN`	unset	Bearer tokens; required on non-loopback binds.
`GOLDENMATCH_API_CORS_ORIGINS`	unset	REST CORS allow-list.
`GOLDENMATCH_ALLOWED_ROOT`	unset	Opt-in path sandbox root.
`GOLDENMATCH_DATABASE_URL` / `GOLDENMATCH_IDENTITY_DSN`	unset	DB connection / migration DSN.
`GOLDENMATCH_SAIL_IDENTITY_ID_SCHEME`	`h1`	Identity record-id scheme on the Sail (Spark Connect) tier.
`GOLDENMATCH_SPARK_JAR`	unset	Path to the JVM scorer jar shipped to Spark executors via `addArtifact`. Unset means “look for the build output”; the jar is built rather than vendored, so a source checkout has none until you build it.
`GOLDENMATCH_SPARK_VECTOR_SCORER`	`0`	Score every comparison field of a pair in ONE UDF call instead of one per field. MEASURED SLOWER — do not enable. On a 1M-row / 5.5M-pair cluster run it was 2.3x slower than the per-field path (scoring 26.11s -> 59.75s, 17.13s -> 40.38s; counts stage 45.42s -> 101.37s) with byte-identical pattern counts and model. Kept, off, as a recorded negative result: Spark arrays are `ArrayData` of `InternalRow`, so a returned similarity vector pays row-wise object churn — and `getItem(i)` per field appears not to be common-subexpression-eliminated, so the UDF is likely evaluated once per field anyway, plus the array. Batching inside Spark SQL loses to a columnar transfer.
`GOLDENMATCH_SPARK_FUSED_BLOCK_JOIN`	`1`	Score and count INSIDE the blocking self-join rather than projecting it to `(a, b)` ids and joining both records back. `0` forces the legacy three-join path. MEASURED on the plan: one counts pass goes from 7 exchanges / 3 sort-merge joins to 3 / 1, and the score path from 8 / 4 to 4 / 2 — the joins that disappear are the ones consuming a frame with O(pairs) rows. On the 100M lane this cut shuffle write 43% (174.9 -> 99.0 GB), memory spill 92% (56.4 -> 4.29 GB) and stages 119 -> 74, with identical pairs, scores and quality. It did NOT move total wall (958.31s -> 975.30s): on that workload resource use and wall are close to decoupled, and the flag exists so the A/B stays takeable in both directions.
`GOLDENMATCH_SPARK_EXACT_PROFILING`	`0`	Measure auto-config column statistics on the CLUSTER instead of on a 20,000-row driver sample. Fixes a statistic that is currently mislabelled: `profile_columns` stores its confirm pass as `full_n_distinct`, documented as “the EXACT full-frame count, never the sampled”, but on the Spark path it is handed a driver sample, so on a 500M-row table the “exact” count describes 20k rows. Two rules read it as though it described the population, and both have already cost a run: an `email` column with a true distinct-fraction of 0.28 read 0.72 -> 1.00 across 100K -> 5M and was discarded as a surrogate key (#876), and the drop-constant rule misses a value at frequency 1e-4 90% of the time from a 1,000-row sample (#2687). The verdict moves with scale rather than with the data. Cost is bounded: `approx_count_distinct` runs for every column and an exact `count_distinct` (a shuffle per column) only for boundary columns near `<= 1` or near-unique — cheap statistic nominates, exact one decides. Default OFF on purpose: several rules compare these fields to absolute cuts chosen while the inputs were sample-derived, so making them truthful can move quality scores the way rebasing `mass_above_threshold` moved `anchor_person_match` from 1.0000 to 0.7303. This flips on quality-gate evidence, not on the argument that it is more correct. The `exact_profiling` argument overrides the environment in either direction, so a caller or test can pin it without touching process state.
`GOLDENMATCH_PREPARED_RECORD_STORE_DIR` / `_PERSIST`	unset / `0`	Prepared-record store path / persistence.
`GOLDENMATCH_LLM_MODEL`	provider default	Override the LLM scorer model.
`GOLDENMATCH_EMBEDDING_PROVIDER` / `GOLDENMATCH_INHOUSE_MODEL`	unset	Embedding provider / in-house model id.
`GOLDENMATCH_GPU_MODE` / `_GPU_ENDPOINT` / `_GPU_API_KEY`	auto	GPU embedding mode / remote endpoint / key.
`GOLDENMATCH_LLAMA_GGUF`	unset	Path to a local GGUF embedding model for the `model="llama"` / `"llama:/path.gguf"` backend: offline embeddings, no cloud/torch (`pip install goldenmatch[llama]`). Benchmarks: `scripts/bench_llama_embedder.py`, `scripts/bench_llama_product_matching.py`.
`GOLDENMATCH_LLM_BASE_URL`	`https://api.openai.com/v1`	OpenAI-compatible base URL for the LLM scorer: point at a local llama.cpp / llamafile server for offline, free LLM judging (also honors `OPENAI_BASE_URL`).
`GOLDENMATCH_LOCAL_LLM`	`auto`	Gate for the in-process self-hosted LLM adapter (`provider="local"`, `pip install goldenmatch[local-llm]`). `auto` loads the pinned model when reachable, else abstains (candidates keep fuzzy scores); `1` requires it (raises if unavailable); `0` forces abstain.
`GOLDENMATCH_SEMANTIC_NAMER`	unset	Hard kill-switch for the OPTIONAL advisory LLM namer on semantic-model discovery (`discover_semantic_model(name=True)` / `discover-model --name`). The namer is opt-in per call and never authoritative (business names land only in `ProposedModel.naming`; the certified structure is untouched). `0` forces it off even when requested; otherwise it runs when `name=True` and an LLM provider/key resolves (`goldenmatch[llm]`), abstaining to no names otherwise.
`GOLDENMATCH_LOCAL_LLM_PATH`	unset	Explicit path to a local GGUF for `provider="local"`, bypassing the pinned-model download.
`GOLDENMATCH_LOCAL_LLM_CACHE`	`~/.cache/goldenmatch/models`	Directory the pinned GGUF is downloaded into (from its GitHub Release asset) and cached across runs.
`GOLDENMATCH_LOCAL_LLM_GPU_LAYERS`	`0`	Layers to offload to GPU for the in-process local model (`0` = CPU-only).
`GOLDENMATCH_WEAKNESS_LLM`	`0` (off)	Opt-in: let the `config_weaknesses` tool phrase its one-paragraph `summary_plain` with an LLM (needs `OPENAI_API_KEY`/`ANTHROPIC_API_KEY`). Only a compact structured digest of the findings (ids/severities/short labels) is sent: never raw rows or data. The structured findings themselves are always deterministic and never depend on the LLM; off = deterministic template summary.
`GOLDENMATCH_WEAKNESS_LLM_MODEL`	`gpt-4o-mini` (OpenAI) / `claude-3-5-haiku-latest` (Anthropic)	Model id for the `GOLDENMATCH_WEAKNESS_LLM` summary, per detected provider.
`GOLDENMATCH_UDF_IMPORTS`	unset	Snowflake UDF import asset directory.
`GOLDENMATCH_SYNC_STREAMING_THRESHOLD` / `_STREAMING_BLOCK_WORKERS`	`500000` / auto	Postgres streaming-store thresholds.
`GOLDENMATCH_ANALYTICS`	`0` (off)	Opt-in anonymous, PII-free usage analytics to PostHog (needs `POSTHOG_API_KEY`). Off = nothing collected.

Not a GOLDENMATCH_* var but relevant: set OPENAI_API_KEY or ANTHROPIC_API_KEY for LLM scoring and the LLM auto-config fallback, and POLARS_SKIP_CPU_CHECK=1 on Windows to skip the Polars import-time CPU probe.

Backends and scale

The measured numbers behind each backend choice.

Scale envelope

The block-size failure modes that matter more than the backend.

Distributed routing

The planner decides scoring, clustering, and golden routing per stage, keyed off driver RAM (not cluster total). The lint_routing / explain_routing MCP tools surface these decisions and flag overrides that force a slow path; a slow-path override at scale refuses unless allow_slow_path=true.

Single box

With no Ray cluster connected, every stage runs in-memory / in-process. There is nothing to distribute to, so the projections below do not apply.

Driver-RAM projection

When a cluster is present, each materializing stage distributes only when its working set exceeds the driver-RAM budget (available_ram_gb × 0.6):

Clustering / WCC distributes when the projected edge set (estimated_pair_count × 16 bytes) exceeds the budget. At 100M rows the 609.4M-pair edge set fits a large head, so WCC runs in-memory (scipy): forcing the distributed path there is a multi-hour tail.
Scoring and golden distribute when the row-frame estimate (n_rows × 512 bytes) exceeds the budget. The same data therefore distributes scoring on a 48 GB-driver cluster but keeps it in-memory on a 256 GB box.

Overrides

distributed_routing.* config pins and the legacy GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD env var are honored but linted. CLUSTERING_THRESHOLD=0 forces distributed WCC regardless of the projection; at scale the linter flags it ERROR and the runtime refuses unless allow_slow_path=true.

​Recommended setups

​Local / interactive (up to ~500K rows)

​Single box at scale (1M–200M rows, 16-core / 64 GB class)

​Distributed (Ray cluster, 50M+ rows)

​Native acceleration

​GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE

​GOLDENMATCH_NATIVE_RAYON_MIN_PAIRS

​GOLDENMATCH_NATIVE_SKETCH_RAYON_MIN_ROWS

​GOLDENMATCH_ER_MODEL_DIR

​Backend selection

​ANN blocker backend (embedding/semantic blocking)

​Distributed pipeline

​Fast path at scale (distribute only what doesn’t fit)

​Prep & scoring perf opt-ins

​Fused kernel auto-routing

​Auto-config & planning

​Config-suggestion (healer) knobs

​Servers, security & storage

​Gotchas that cost a day

​Throughput tier (sketch-then-verify)

​Enabling it

​What it does

​Posture and honest reporting

​Measured throughput (benchmark)

​Scope

​Configuration precedence

​Full environment-variable reference

​Everyday & scale

​Distributed sub-knobs

​Perf opt-ins & diagnostics

​Auto-config & blocking (advanced)

​Probabilistic Fellegi-Sunter (advanced)

​Servers, security, storage, LLM/embedding

Backends and scale

Scale envelope

​Distributed routing

​Single box

​Driver-RAM projection

​Overrides

Recommended setups

Local / interactive (up to ~500K rows)

Single box at scale (1M–200M rows, 16-core / 64 GB class)

Distributed (Ray cluster, 50M+ rows)

Native acceleration

`GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE`

`GOLDENMATCH_NATIVE_RAYON_MIN_PAIRS`

`GOLDENMATCH_NATIVE_SKETCH_RAYON_MIN_ROWS`

`GOLDENMATCH_ER_MODEL_DIR`

Backend selection

ANN blocker backend (embedding/semantic blocking)

Distributed pipeline

Fast path at scale (distribute only what doesn’t fit)

Prep & scoring perf opt-ins

Fused kernel auto-routing

Auto-config & planning

Config-suggestion (healer) knobs

Servers, security & storage

Gotchas that cost a day

Throughput tier (sketch-then-verify)

Enabling it

What it does

Posture and honest reporting

Measured throughput (benchmark)

Scope

Configuration precedence

Full environment-variable reference

Everyday & scale

Distributed sub-knobs

Perf opt-ins & diagnostics

Auto-config & blocking (advanced)

Probabilistic Fellegi-Sunter (advanced)

Servers, security, storage, LLM/embedding

Distributed routing

Single box

Driver-RAM projection

Overrides