> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bensevern.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Tuning & opt-ins

> The single reference for every GoldenMatch runtime knob: native acceleration, backend selection, the distributed pipeline, perf opt-ins, and every GOLDENMATCH_* environment variable — what each does, its default, and when to use or avoid it.

This is the one place that lists every runtime knob GoldenMatch reads — native acceleration, backend selection, the distributed pipeline, the prep/scoring perf opt-ins, and the full `GOLDENMATCH_*` environment-variable table. For each, you get **what it does, its default, and when to set it (or leave it alone)**.

If you are tuning a large run and just want the answer, jump to [Recommended setups](#recommended-setups). If something ran far slower than you expected, read [Gotchas that cost a day](#gotchas-that-cost-a-day) first — every item there is a real footgun.

<Note>
  Most GoldenMatch knobs are environment variables, not config fields, because they switch a code path (native vs Python, in-memory vs distributed) rather than describe your data. Data-shape settings — matchkeys, blocking keys, golden rules — live in [Configuration](/docs/goldenmatch/configuration). This page is the operational layer on top of that.
</Note>

## Recommended setups

You almost never need to set anything for correctness — auto-config and the planner choose sane defaults. These blocks are about **speed at scale** and **reproducibility**, not behavior.

### Local / interactive (up to \~500K rows)

Set nothing. Zero-config is the supported path. If you want the compiled kernels for a measurable speed-up on the scoring/clustering hot loops, install the optional native wheel:

```bash theme={null}
pip install goldenmatch[native]
```

With the wheel present, the default `GOLDENMATCH_NATIVE=auto` already runs every component that has a kernel natively (native is the reference; pure-Python is the lossy fallback). You do not need to set `GOLDENMATCH_NATIVE=1`.

### Single box at scale (1M–200M rows, 16-core / 64 GB class)

```bash theme={null}
# Make a missing/broken native wheel a LOUD error instead of a silent slow fallback.
export GOLDENMATCH_NATIVE=1
# Turn on the Polars-native address-normalize fast path (skips the per-row Python plugin).
export GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE=1
# Reproducible runs: don't read cross-run auto-config memory.
export GOLDENMATCH_AUTOCONFIG_MEMORY=0
# These two are already ON by default; set them explicitly in a bench harness for the record.
export GOLDENMATCH_BUCKET_SLIM_PROJECTION=1
export GOLDENMATCH_GOLDEN_SLIM_MULTIDF=1
# Windows only: skip the Polars CPU probe that can hang at import.
# export POLARS_SKIP_CPU_CHECK=1
```

The planner picks `backend=bucket` for this lane automatically when the native kernel is enabled. This is the exact env used by the quality-invariant scale ladder (1K → 100M), so it is the validated single-box recipe.

<Warning>
  **Setting `backend="ray"` does NOT distribute the run.** On its own it falls back to an in-memory path that materializes the whole frame on the driver. Real distribution needs `GOLDENMATCH_DISTRIBUTED_PIPELINE=2` plus a Ray Dataset input — see [Distributed pipeline](#distributed-pipeline) and [Gotchas](#gotchas-that-cost-a-day). At single-box scale, prefer `bucket`.
</Warning>

### Distributed (Ray cluster, 50M+ rows)

```bash theme={null}
pip install goldenmatch[ray]
export GOLDENMATCH_DISTRIBUTED_PIPELINE=2     # Phase 5 streaming — the real distributed path
export GOLDENMATCH_ENABLE_DISTRIBUTED_RAY=1   # let the planner pick ray for backend selection
export GOLDENMATCH_NATIVE=1
# Multi-node only: a SHARED scratch the WCC checkpoints round-by-round (node-local breaks cross-node reads).
export GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH=gs://your-bucket/rc_scratch
```

Phase 5 has a hard contract: a Ray Dataset input with a **global `__row_id__` column**, and it **writes golden records to `output_path`** rather than returning an in-memory clusters dict. The recall-complete path (block-shuffle scoring + randomized-contraction WCC) is **default-on** and was validated end-to-end at **100M rows on a real 5-node GCP cluster (9.2 min, 20M clusters recovered exactly, 0.36 GB driver RSS)**. See [Distributed pipeline](#distributed-pipeline) for the full contract. At single-box scale, prefer `bucket` — only reach for a Ray cluster when the data genuinely won't fit one box.

***

## Native acceleration

**Rust is the reference implementation** (as of the 2026-07 reference-mode change). The Rust/PyO3 kernel is the source of truth for GoldenMatch's hot-path primitives; **pure-Python is the lossy fallback** used only when a native wheel isn't installed (the wheel ships as `pip install goldenmatch[native]`). A central gate (`core/_native_loader.py`) selects per component.

`GOLDENMATCH_NATIVE` (`core/_native_loader.py`):

| Value                      | Behavior                                                                                                                                                                                                           |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `auto` (default, or unset) | Run native **wherever the component's kernel symbol exists on the wheel** — native is the reference. Pure-Python is the fallback for a missing wheel/symbol.                                                       |
| `1`                        | Require native. Raises `RuntimeError` at the first kernel call if the wheel isn't importable. Use this on scale/bench runs so a missing wheel fails loudly instead of running 100x+ slower in the Python fallback. |
| `0`                        | Force the pure-Python fallback everywhere. Use for parity debugging or if you suspect a kernel bug.                                                                                                                |

Under `auto`, native runs for every component that has a kernel symbol — `clustering`, `block_scoring`, `pairs`, `featurize`, `hashing`, `autoconfig`, `sketch`, `pprl_bloom`, and the multimodal `perceptual` hashes. Each cleared bit-/ULP-level parity against the Python reference, so the fallback is output-identical (golden-vector verified). (The former per-component allowlist, `_GATED_ON`, is retained in the loader as sign-off documentation but no longer governs `auto`.) The `sketch` component covers both MinHash/LSH (#1081) and SimHash-over-embeddings (#1090) band hashing (\~29x faster; the single source of truth across Python/Rust/TS, so semantic blocking runs on the Rust `sketch-core` kernel by default). Fellegi-Sunter block scoring is **native-authoritative by default as of goldenmatch 2.6.0** (`GOLDENMATCH_FS_NATIVE=0` opts out to the reproducible numpy path); it is scoped to the probabilistic path only.

The `autoconfig` component is the shared, pyo3-free decision core (`goldenmatch-autoconfig-core`) behind auto-config — the planner, column classifier (with the adaptive `1 − 1/√n` identifier-cardinality floor), the sample→full pair-count extrapolation (the corrected `ratio²` + Chao1 estimate), and the adaptive decision thresholds (the sparse-match floor and the per-type exact-matchkey floor). It runs native under `auto` and is byte-identical to the pure-Python oracle (golden-vector parity across Python / native / wasm-TS). The same crate compiles to WebAssembly so the TypeScript surface consumes the identical decision logic — one source of truth, zero cross-surface drift.

**When to set `=1`:** any large or benchmarked run where a silent Python fallback would be a disaster. The cost of a missing wheel is enormous (the scoring kernel is the dominant stage), and `auto` hides it.

**When to leave it `auto`:** normal local use. If the wheel is installed you already get native; if it isn't, you get working Python.

### `GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE`

**Default off.** When set to `1`, address-normalize transforms run as a Polars-native expression instead of the per-row Python refdata plugin. On address-heavy data at scale the per-row plugin is a real bottleneck in the prep stage; the native path removes it.

* **Set it (`=1`)** when you have `address` columns and are running at scale. Output is parity-tested on representative inputs.
* **Leave it off** for small runs, or if you hit an address-canonicalization edge case (it is opt-in precisely because one integration-level cluster-count drift hasn't been fully root-caused).

### `GOLDENMATCH_NATIVE_RAYON_MIN_PAIRS`

Default `20000000`. Candidate-pair count per kernel call below which the native block-scorer runs in the calling thread instead of fanning out to rayon. This exists to avoid a futex-park stall observed on some 8-core Linux CPUs (issue #688). The Python caller already parallelizes across buckets, so the sequential path loses nothing on small calls. Leave it at the default unless you are reproducing #688; `0` forces rayon always, a huge value forces sequential always.

### `GOLDENMATCH_NATIVE_SKETCH_RAYON_MIN_ROWS`

Default `10000`. Row count below which the native MinHash/LSH batch sketcher (#1081) runs in the calling thread instead of fanning out to rayon — the same #688 calling-thread-vs-fanout guard, applied to the `sketch` kernel. The SimHash batch sketcher (#1090) shares this threshold. Relevant whenever the `sketch` kernel runs native — which is the default under `auto` since #1090 (set `GOLDENMATCH_NATIVE=0` to force the Python path), or under `GOLDENMATCH_NATIVE=1`. `0` forces rayon always, a huge value forces sequential always.

***

## Backend selection

Same pipeline, different execution engine. Auto-config picks one from your row count; you can override.

| Backend            | Range                       | How it works                                                                                                   |
| ------------------ | --------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `polars` (default) | \< 500K rows                | In-memory Polars. Fastest per record.                                                                          |
| `bucket`           | 1M–200M, single box         | Per-bucket native scoring. The validated single-box path on 16c/64GB. Planner picks it when native is enabled. |
| `chunked`          | 5M+, memory-constrained box | Streaming reader + cross-chunk join + block-keyed index. The 4c/16GB fallback.                                 |
| `duckdb`           | 500K–50M                    | Out-of-core pair store, spills to disk so RAM isn't the ceiling.                                               |
| `ray`              | ≥ 50M, distributed          | Per-block remote tasks. **See the distribution caveat below — `backend="ray"` alone does not stream.**         |

Force one in config or on the CLI:

```python theme={null}
config = gm.GoldenMatchConfig(backend="duckdb")
```

```bash theme={null}
goldenmatch dedupe big.csv --backend ray
goldenmatch dedupe big.csv --chunked
```

**Auto-selection knobs:**

| Variable                      | Default           | Effect                                                                                                      |
| ----------------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_PLANNER_BUCKET`  | `1`               | The v3 planner may pick `bucket`. Set `0` to opt the bucket backend out.                                    |
| `GOLDENMATCH_DUCKDB_SCORE_DB` | unset (in-memory) | Path for the DuckDB pair store to spill to disk. Point it at scratch with room when running DuckDB at 50M+. |

<Note>
  The old single-threshold `GOLDENMATCH_AUTOCONFIG_BACKEND` / `GOLDENMATCH_AUTOCONFIG_BACKEND_THRESHOLD` env overrides were removed in v2.0. Backend selection is now the v3 planner's job — it decides from `(ComplexityProfile, RuntimeProfile)`. To force a backend, set it explicitly on `GoldenMatchConfig(backend=...)` or the CLI `--backend`.
</Note>

### ANN blocker backend (embedding/semantic blocking)

The embedding ANN blocker (`ANNBlocker`) resolves one of three inner-product backends per index build: **native HNSW → FAISS → numpy**. Native HNSW is the optional [`goldenmatch-hnsw`](https://pypi.org/project/goldenmatch-hnsw/) wheel — a pure-Rust `IndexHNSWFlat` with zero C dependencies (no FAISS); FAISS (`goldenmatch[embeddings]`) is exact `IndexFlatIP`; numpy is the always-available all-pairs fallback. Scores are the raw inner product on the HNSW/FAISS paths and cosine on numpy — identical when vectors are L2-normalized (the default embedder output).

In `auto` mode HNSW is chosen only where it wins: a large corpus **and** a small neighbor count per probe. Below that gate, brute force is both faster and exact, so small indexes (and the retrieve-all pattern behind the persistent `VectorIndex`) stay on the exact path unchanged.

| Variable                               | Default | Effect                                                                                                                                                                    |
| -------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_ANN_BACKEND`              | `auto`  | Force a backend: `auto` (size-gated HNSW→FAISS→numpy), `hnsw` (native, ignores the size gate), `faiss`, or `numpy`. Degrades gracefully when the named library is absent. |
| `GOLDENMATCH_ANN_HNSW_MIN`             | `4096`  | In `auto`, the minimum corpus size for HNSW. Below it, the exact backend is used (HNSW's win is asymptotic).                                                              |
| `GOLDENMATCH_ANN_HNSW_MAX_K`           | `512`   | In `auto`, the maximum `top_k` for HNSW. Above it (retrieve-nearly-all), the exact backend is used instead.                                                               |
| `GOLDENMATCH_ANN_HNSW_M`               | `16`    | HNSW graph degree (`M`): max neighbors per node on upper layers (`2M` on layer 0). Higher = better recall, more memory.                                                   |
| `GOLDENMATCH_ANN_HNSW_EF_CONSTRUCTION` | `200`   | Candidate-list size during index build. Higher = better graph quality, slower build.                                                                                      |
| `GOLDENMATCH_ANN_HNSW_EF_SEARCH`       | `64`    | Candidate-list size during search (raised to at least `top_k`, capped at the corpus size). Higher = better recall, slower query.                                          |

<Note>
  `goldenmatch-hnsw` is an **optional** wheel, installed separately (`pip install goldenmatch-hnsw`) exactly like FAISS. Without it, the ANN blocker uses FAISS if present, else the numpy fallback — no behavior change.
</Note>

***

## Distributed pipeline

This is the knob most people get wrong, so read the contract before you run a 100M job.

`GOLDENMATCH_DISTRIBUTED_PIPELINE` selects how `run_dedupe_pipeline_distributed` (the Ray Dataset entry point) executes:

| Value           | Phase                | What actually happens                                                                                                                     |
| --------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| unset (default) | Phase 2 "cheat-line" | Calls `take_all()` on the input and runs `dedupe_df()` **in memory** on the driver. Back-compat default. **Does not distribute scoring.** |
| `1`             | Phase 4 scaffold     | Clustering/golden stages are Ray-ready, but auto-config and scoring still materialize on the driver. Kept for compatibility.              |
| `2`             | Phase 5 streaming    | The real distributed path: scoring → clustering → golden → write all stay Ray Datasets. No `take_all()` on the input.                     |

**Phase 5 (`=2`) contract — different from in-memory dedupe:**

* **Input must carry a global `__row_id__` column.** Without it, each partition synthesizes local IDs that collide and the weakly-connected-components step merges unrelated clusters.
* **Output is written to `output_path` as parquet golden records.** It does **not** return an in-memory clusters dict the way `dedupe_df` does.
* It still materializes two things on the driver: the \~5K-row auto-config sample, and cluster aggregates for golden-field synthesis. These are the last "cheat-lines" being retired.

`GOLDENMATCH_ENABLE_DISTRIBUTED_RAY` (default `0`) is **separate**: it only lets the v3 planner *pick* `ray` during backend selection. It does **not** turn on the streaming pipeline. Explicit `backend="ray"` always works without it. The planner stopped auto-picking ray after the Distributed Plan v1 stack failed the 5M kill criterion against the bucket backend, so ray is opt-in.

**Recall-complete clustering at scale (#844, shipped 2026-06-11).** Phase 5's default path now block-shuffles scoring so true duplicates co-locate, then runs a relational **randomized-contraction** weakly-connected-components pass that clusters correctly across input-partition boundaries. This replaced two earlier WCC implementations that died at 100M (a driver head-wedge and a Ray streaming-executor deadlock). It is **default-on** and validated end-to-end at 100M (9.2 min, 20M clusters exact). On a **multi-node** cluster the WCC checkpoints each round to `GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH`, which **must be a shared path** (e.g. `gs://...`) — a node-local scratch breaks the cross-node parquet reads and now raises rather than silently under-merging.

<Note>
  **Only scoring needs distribution at this scale; clustering usually doesn't.** The scored pair set is small (\~`(id_a, id_b, score)` per match), so at 100M rows it's \~1.8 GB — it fits one node. Clustering therefore routes to the in-memory scipy connected-components pass (seconds) automatically whenever the pairs fit driver RAM, and only falls back to the distributed WCC (a multi-hour tail) when they genuinely don't. You don't need to set `GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD` for this — it's the default.
</Note>

<Warning>
  **Correlated survivorship (`field_groups` and conditional `field_rules`) does not run on the distributed or Sail paths.** The golden builder that resolves lock-step groups and conditional predicates runs on the driver. If your config includes `field_groups` or list-form `field_rules`, the Ray Phase-5 streaming pipeline (`GOLDENMATCH_DISTRIBUTED_PIPELINE=2`) and the Sail backend will refuse with a clear error rather than silently mis-merging. Run those configs on the in-memory path.
</Warning>

**Distributed sub-knobs** (only relevant on a real Ray cluster):

| Variable                                       | Default                   | Effect                                                                                                                                                                                                                                                                                                                       |
| ---------------------------------------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_DISTRIBUTED_BLOCK_SHUFFLE`        | `1` (on)                  | Block-shuffle scoring so duplicates co-locate before clustering. The recall-complete path. `0` restores the legacy per-partition `local_cc` (under-merges across partitions at scale).                                                                                                                                       |
| `GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH`          | unset                     | Scratch directory the randomized-contraction WCC checkpoints to. **Must be shared (e.g. `gs://...`) on a multi-node cluster** or the run raises.                                                                                                                                                                             |
| `GOLDENMATCH_DISTRIBUTED_WCC`                  | `two_phase`               | WCC algorithm for `build_clusters_distributed`'s *own* callers (the at-scale Phase-5 path forces `randomized_contraction` and ignores this). `two_phase` is chain-safe; avoid `pointer_jump` (can deadlock).                                                                                                                 |
| `GOLDENMATCH_DISTRIBUTED_WCC_SEED`             | unset                     | Seed for the randomized-contraction affine hash (reproducible cluster IDs across runs).                                                                                                                                                                                                                                      |
| `GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD` | unset (memory-aware)      | Pair count above which clustering goes distributed. **Unset (default): memory-aware** — clustering stays on the fast driver-side scipy.csgraph path whenever the scored pair set fits available driver RAM, and only distributes when it genuinely won't. Set an explicit integer to force a pair-count boundary either way. |
| `GOLDENMATCH_DISTRIBUTED_GOLDEN_THRESHOLD`     | `5000000`                 | Multi-member row count above which golden-record building goes distributed.                                                                                                                                                                                                                                                  |
| `GOLDENMATCH_DISTRIBUTED_SCORE_NUM_CPUS`       | `2`                       | CPUs requested per Ray scoring task.                                                                                                                                                                                                                                                                                         |
| `GOLDENMATCH_DISTRIBUTED_SCORE_PROJECT`        | `1` (on)                  | Project to scoring-relevant columns before the block-shuffle so it moves narrow blocks, not full records (#957). Output-invariant. `0` disables.                                                                                                                                                                             |
| `GOLDENMATCH_DISTRIBUTED_SHUFFLE_PARTS`        | `min(256, cpu×4)`         | Block-shuffle partition count. **Raise it** to shrink per-partition blocks so more `_score` tasks fit the object-store budget (relieves `_score` `ResourceBudget` backpressure). Co-location stays correct at any value.                                                                                                     |
| `GOLDENMATCH_DISTRIBUTED_OP_RESERVATION`       | unset (Ray default \~0.5) | Ray Data per-op object-store reservation ratio. **Lower it** (e.g. `0.2`) to hand the running `_score` op more object-store budget when it's `ResourceBudget`-backpressured below CPU capacity (idle workers, object store not full). Tune at scale.                                                                         |
| `GOLDENMATCH_DISTRIBUTED_SCORE_CONCURRENCY`    | unset                     | Explicit `concurrency=` on the `_score` map\_batches. Pins the task-pool size once blocks are narrow; the object-store reservation/shuffle-parts knobs above govern the actual `ResourceBudget` cap.                                                                                                                         |

<Note>
  **If the distributed score stage leaves workers idle** (`MapBatches(_score): … [backpressured:tasks(ResourceBudget)]` with the object store well under capacity), `_score` is reservation-capped, not CPU-bound. Lower `GOLDENMATCH_DISTRIBUTED_OP_RESERVATION` (→ `0.2`) and/or raise `GOLDENMATCH_DISTRIBUTED_SHUFFLE_PARTS`, then re-measure — the optimal pair is cluster-shaped. (Shrinking Ray Data's `target_max_block_size` is **not** a safe lever: `_score` keeps each co-located partition whole via `batch_size=None`, so a split block would split a group and under-score.)
</Note>

<Note>
  You bring the Ray cluster — GoldenMatch ships the pipeline, not the bootstrap. The GCP recipe is in [`docs/distributed-ray-cluster-setup.md`](https://github.com/benseverndev-oss/goldenmatch/blob/main/docs/distributed-ray-cluster-setup.md). At single-box scale the `bucket` backend is validated through 200M and is simpler — use it unless the data won't fit one box.
</Note>

### Fast path at scale (distribute only what doesn't fit)

The single most expensive distributed-tuning mistake is distributing a *stage* that fits in memory. **"The job is big" does not mean "every stage must be distributed."** Ask it per stage:

| Stage            | Fits one node at 100M?                                                  | Fast path                                                                                                                    | Slow trap                                                                                                                                     |
| ---------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| Scoring          | No — 100M rows                                                          | **Distributed** across workers, **native kernel on** (`GOLDENMATCH_NATIVE=1` + `goldenmatch-native` installed on every node) | pure-Python (cluster installs plain `goldenmatch`); \~6 of 80 CPU if the shuffle blocks are oversized                                         |
| Clustering (WCC) | **Yes** — the scored pair set is small (\~1.76 GB / 110M edges at 100M) | **In-memory** driver-side `scipy.csgraph` connected-components (\~60s)                                                       | the distributed randomized-contraction WCC — its O(log N) `gs://`-checkpoint rounds are a multi-hour tail when forced on a pair set that fits |
| Golden           | depends                                                                 | distributed groupby, or **skip it** if you only need cluster membership                                                      | building golden you then discard                                                                                                              |

**Concrete: don't force the distributed WCC on a pair set that fits.** By default `GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD` is **unset and the routing is memory-aware** — clustering stays on the fast in-memory `scipy.csgraph` path whenever the scored pair set fits available driver RAM, and only distributes when it genuinely won't. So at 100M (\~110M pairs / \~1.76 GB) the default keeps clustering in-memory on a 64–256 GB head. Set an explicit integer only if you want to force a hard pair-count boundary either way:

```bash theme={null}
export GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD=2000000000  # force in-memory WCC below 2B pairs
```

Setting it to `0` does the **opposite** — it forces the slow distributed WCC on everything.

**Recover a result without re-running.** The Phase-5 connected-components clustering is independent of the WCC algorithm, so a completed run's cluster *assignments* (written to gs\:// via the `assignments_output_path` hook) **are** the result. If a run finishes but the output is lost, or you want to re-score, download the assignments and score them against your ground truth on a 64 GB box — no cluster, no re-run. The 100M `group_by` for that scoring OOMs a 16 GB box, so run it on a large runner, not your laptop.

***

## Prep & scoring perf opt-ins

These trade nothing for speed/memory at scale and are safe to leave at their defaults. Two are already on.

| Variable                                 | Default  | Effect                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | When to change                                                                                                                                   |
| ---------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `GOLDENMATCH_BUCKET_SLIM_PROJECTION`     | `1` (on) | After bucket scoring, drop columns downstream stages don't need (smaller per-bucket frames). Measured −3.8 GB peak RSS at 10M, output invariant.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | Set `0` only if a custom downstream consumer needs a column it drops.                                                                            |
| `GOLDENMATCH_GOLDEN_SLIM_MULTIDF`        | `1` (on) | Slim internal columns before the multi-DataFrame golden build.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Rarely; `0` restores the wider frame.                                                                                                            |
| `GOLDENMATCH_IDENTITY_BATCH_FINGERPRINT` | `1` (on) | Compute no-PK identity hashes in one Arrow batch instead of per row (2.6x on the fingerprint step).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | `0` restores per-row, for parity debugging.                                                                                                      |
| `GOLDENMATCH_IDENTITY_WRITE_PIPELINE`    | `1` (on) | psycopg pipeline mode for the per-record identity write path (`resolve_clusters` absorb/merge). Streams `upsert`/`emit_event`/`add_edge` statements without a per-statement network round-trip — the dominant cost of a re-resolve against a remote Postgres (Cloud SQL), where a single transaction removed the fsync but not the round-trips. Postgres-only; no-op on SQLite.                                                                                                                                                                                                                                                                                                                                          | `0` restores per-statement writes, for parity debugging.                                                                                         |
| `GOLDENMATCH_ENSEMBLE_KERNEL`            | `1` (on) | The `ensemble` scorer rides the bucket fast path (per-pair + the float64 `_score_block_vec` vec lane + the native `score_one` id 12 arrow kernel), measured **1.47× faster** end-to-end on Febrl3 with byte-identical recall. `0` (`off`/`false`) is the kill-switch → the historical float32 `find_fuzzy_matches` decline path. The earlier default-on attempt regressed on **name-scorer matchkeys** (`name_freq_weighted_jw` / `given_name_aliased_jw` — neither vec nor native, so they ran the O(N²) per-pair Python loop, 19× slower than `find_fuzzy_matches`); that is now fixed at the source by a perf guard in `_resolve_fast_path` that declines the fast path whenever a field would force per-pair Python. | Leave on. Set `0` only to reproduce pre-2026-07-21 output byte-for-byte. See `docs/superpowers/specs/2026-07-21-ensemble-kernel-measurement.md`. |

**Diagnostic-only — these can SLOW the run:**

| Variable                          | Default | Effect                                                                                                                                            |
| --------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_BUCKET_DEBUG`        | `0`     | Print per-bucket prep / kernel / post-filter timing for `backend=bucket`. Output-invariant, near-zero cost, but it's a debug aid, not a speed-up. |
| `GOLDENMATCH_PREP_STAGED_COLLECT` | `0`     | Collect prepared records in stages (memory-management experiment). Can be slower; leave off unless diagnosing prep RSS.                           |
| `GOLDENMATCH_STANDARDIZE_STAGED`  | `0`     | Staged standardization pipeline (experimental). Same caveat.                                                                                      |

***

## Fused kernel auto-routing

The pipeline can route covered slow-path runs to the fused Arrow-native kernels, byte-identically, with no config change. There are two seams with very different maturity: **golden routing is a broad default-on win; match routing is a narrow capacity-survival path that fires only under memory pressure.**

| Variable                                                                                     | Default                     | Effect                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | When to change                                                                                                |
| -------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_GOLDEN_FUSED`                                                                   | on                          | **Golden seam auto-routing (default-on, byte-identical).** Tries the fused Arrow-native golden kernel on every covered slow-path config, reaching golden output at \~2x lower peak RSS. Declines to the classic builder (unchanged output) when the config is uncovered, the native kernel is absent, full `ClusterProvenance` lineage is requested, or the run is fast-path-eligible. Surfaced on `DedupeResult.golden_fused_used`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `0` / `false` / `off` is the kill-switch (forces the classic golden builder everywhere).                      |
| `GOLDENMATCH_MATCH_FUSED`                                                                    | on                          | **Match seam auto-routing — a narrow capacity-survival path.** A controller post-step short-circuits the whole match stage to the fused match kernel under a memory-pressure gate AND a narrow config profile: `auto_split=False`, no identity/adaptive/memory/llm\_boost/confidence\_majority/full-provenance, not across-files-only, and a covered matchkey — **weighted** (single-key or multi-pass) **or Fellegi-Sunter** (single-key or multi-pass; the FS path fits EM inside the short-circuit and was wired in #1892). Zero-config auto-config commits `auto_split=True` and explicit configs bypass the controller, so it stays dormant unless you configure that artifact-free profile and hit the RSS gate. When it fires it is a capacity mode that sheds `scored_pairs`/`review_pairs` + cluster-confidence/lineage to survive; cluster membership + golden stay byte-identical. Marked `DedupeResult.match_fused_capacity_mode` + telemetry `rule_name` `+fused_match_post_step`. Measured 2–19× leaner peak RSS on covered FS shapes. | `0` / `false` / `off` is the kill-switch.                                                                     |
| `GOLDENMATCH_FUSED_PRESSURE_FRACTION`                                                        | `0.65`                      | Fraction of available RAM the estimated classic-path peak RSS must exceed before match routing's pressure gate fires. Only relevant to the pressure-gated match seam.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Lower to make the capacity path trip earlier; raise toward `1.0` to make it trip only under extreme pressure. |
| `GOLDENMATCH_FUSED_RSS_SCALE` / `_BYTES_PER_PAIR` / `_BYTES_PER_CELL` / `_BLOCK_CONCURRENCY` | `0.763` / `64` / `40` / `4` | Calibration coefficients for the est-peak-RSS model that drives the match pressure gate. Defaults are fit against measured runs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Only when re-calibrating the est-RSS model against a new box/workload; leave alone otherwise.                 |

***

## Auto-config & planning

| Variable                                                  | Default   | Effect                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | When to set                                                                                                                                                                                   |
| --------------------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_AUTOCONFIG_MEMORY`                           | `1` (on)  | Read/write cross-run auto-config memory at `~/.goldenmatch/autoconfig_memory.db`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | Set `0` in CI or any run that must be reproducible and not influenced by prior runs.                                                                                                          |
| `GOLDENMATCH_PLANNING_EFFORT`                             | `normal`  | Iteration budget for auto-config: `fast` / `normal` / `thinking` / `einstein`. Higher = more thorough, slower; `thinking`+ measure real full-frame pair counts.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `fast` to speed up config on huge inputs; `thinking` when auto-config picks a poor blocking key. `normal` is byte-identical to the historical default.                                        |
| `GOLDENMATCH_CONFIG_LINT`                                 | `warn`    | Pre-flight [config linter](/docs/goldenmatch/config-linter) on the *resolved* config (zero-config or explicit). `warn` logs data-shape findings and runs; `strict` raises `ConfigLintError` on an error finding before the run; `off` skips. Findings attach to `DedupeResult.lint_findings`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | `strict` in CI/production to fail fast on a degenerate config (e.g. a blocking key that will OOM). `off` to silence.                                                                          |
| `GOLDENMATCH_AUTOCONFIG_LLM`                              | `0`       | Let an LLM propose a config when heuristics are exhausted on a RED profile. Needs `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Opt in only when heuristic auto-config can't find a healthy config and you accept the API cost.                                                                                               |
| `GOLDENMATCH_AUTOCONFIG_INDICATOR_BUDGET`                 | unset     | Set `fast` to skip the expensive complexity indicators.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | Speeding up config on very large inputs.                                                                                                                                                      |
| `GOLDENMATCH_AUTOCONFIG_FORCE_INCLUDE` / `_FORCE_EXCLUDE` | unset     | Comma-separated columns to force into / out of auto-config analysis.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | Rescue a column auto-config drops, or exclude a noisy one, without writing a full explicit config.                                                                                            |
| `GOLDENMATCH_FIELD_GROUP_SURVIVORSHIP`                    | `0` (off) | `1` enables auto-detection of correlated field groups for survivorship. When on, auto-config proposes lock-step field groups (address, person name, contact) so related columns are always promoted from the same source record. Explicitly declared `field_groups` in YAML are always honored regardless of this flag. Off by default -- output is byte-identical to prior behavior when unset. Group promotions are visible in the lineage `golden_records` section, `explain --cluster`, and the MCP `lineage` tool (see [Surfacing the group decision](/docs/goldenmatch/configuration#surfacing-the-group-decision)).                                                                                                                                        | Set when you want auto-config to detect field groups without writing explicit `field_groups` YAML. Equivalent to `field_group_detection: true` in the `golden_rules` config block.            |
| `GOLDENMATCH_PERCEPTUAL_AUTOCONFIG`                       | `0` (off) | `1` enables multimodal-ER media-as-evidence wiring (ADR 0022). When on, auto-config detects columns holding fixed-width hex perceptual hashes (a 16-char image pHash from `core.perceptual.phash_hex`, a 96-char rotation/crop-aware radial-variance profile from `radial_hex`, or a multi-word audio fingerprint from `audio_fp_hex`) and appends a `phash` / `radial` / `audio_fp` matchkey, plus `perceptual` banded-hamming-LSH blocking for an image column when the committed config has nothing else to block on (the rotation-aligned `radial` feature gets no LSH blocking). Additive + idempotent + fail-open; output is byte-identical to prior behavior when unset.                                                                              | Set when a dataset carries a precomputed perceptual-hash column you want treated as a match feature without writing explicit matchkey YAML.                                                   |
| `GOLDENMATCH_AUTOCONFIG_ROUTE_PROBABILISTIC`              | `0` (off) | `1` lets zero-config auto-config route a *probabilistic-shaped* dataset to the Fellegi-Sunter path instead of the default exact+weighted matchkeys. The trigger fires when the emitted config has **no surviving exact matchkey backed by a strong-identity column** (`identifier` / `email` / `phone`) **and ≥2 fuzzy fields** — i.e. there's no clean key carrying the dedup, so EM-weighted comparison wins. It delegates to `auto_configure_probabilistic_df` (which also diversifies blocking onto orthogonal stable keys). A behavior change, so default-off pending a broader regression sweep; measured lift on biographical/PII data (historical\_50k F1 0.466 → 0.83, recall 0.39 → 0.75) with no regression on datasets that retain a strong key. | Opt in on no-strong-identifier, error-heavy datasets (historical/biographical records, dirty PII) where exact matching under-recalls. Leave off when your data has a reliable id/email/phone. |

If auto-config raises `ControllerNotConfidentError`, that's by design at ≥100K rows on a RED profile — it's refusing to run a noise-result dedupe. See [Passing an explicit config](/docs/goldenmatch/configuration) for recovery (the in-repo `docs/explicit-config.md` has the failing-sub-profile table).

There are also a number of experimental blocking and probabilistic knobs (`GOLDENMATCH_BLOCKING_*`, `GOLDENMATCH_FS_*`, `GOLDENMATCH_NOISE_AWARE_*`, `GOLDENMATCH_QUALITY_AWARE_BLOCKING`, `GOLDENMATCH_FD_NEGATIVE_EVIDENCE`). These are research/tuning levers, default-safe, and listed in the [full table](#full-environment-variable-reference) below. Most users never touch them.

***

<h2 id="config-suggestion-healer-knobs">
  Config-suggestion (healer) knobs
</h2>

These tune the [healer](/docs/goldenmatch/config-suggestions) — the `review_config` config-suggestion loop, which is now wired into the default `dedupe_df` pipeline as an advisory surface (a free controller signal gates a cheap attach; see `GOLDENMATCH_SUGGEST_ON_DEDUPE`). The healer needs `goldenmatch[native]` and degrades gracefully without it. You almost never need to change these — the defaults are the values the suggester gym selected.

| Variable                              | Default          | Effect                                                                                                                                                                                                                                                                                                                        | When to set                                                                                                                                                 |
| ------------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_SUGGEST_ON_DEDUPE`       | on               | Master switch for the **default-pipeline** healer surface (Python `dedupe_df` / CLI / servers and the TS/WASM `dedupe(rows, { ... })`): attaches advisory `suggestions` when a free trigger fires. Set `0` to disable the trigger + attach entirely (the explicit `suggest=`/`heal=` opt-ins and `review_config` still work). | Set `0` to opt out of the advisory default-run surface.                                                                                                     |
| `GOLDENMATCH_HEAL_STEP_CAP`           | `5`              | Max healer iterations -- the coarsest cost lever (each step is a full dedupe plus up to `SUGGEST_MAX_VERIFY` verification re-runs). Clamped to >= 1; invalid input falls back to the default.                                                                                                                                 | Lower to cap healer cost on large inputs; raise to allow more repair steps.                                                                                 |
| `GOLDENMATCH_HEAL_MIN_HEALTH_GAIN`    | unset (disabled) | Marginal-gain early stop (#1404): stop the heal loop once an iteration's cluster-health gain falls below this float. Unset = run up to `HEAL_STEP_CAP`; `0.0` stops on the first non-improving step.                                                                                                                          | Set a small positive value to stop early once suggestions stop paying off.                                                                                  |
| `GOLDENMATCH_SUGGEST_MAX_VERIFY`      | `8`              | Max suggestions verified by a full pipeline re-run before the remaining tail passes through unverified. Each verified candidate is one extra re-run. Values `< 1` are clamped to `1`.                                                                                                                                         | Lower to cut healer cost; raise to verify more candidates end-to-end.                                                                                       |
| `GOLDENMATCH_SUGGEST_HEALTH`          | `cohesion`       | Self-verify health proxy the gate uses to keep only non-worsening suggestions. `cohesion` (precision-sensitive `min_edge` × saturating coverage) is the default since 2026-06; `legacy` restores the older `matched_rate × avg_conf − HHI` proxy.                                                                             | `legacy` only to reproduce pre-2026-06 suggestion behavior. The bake-off showed `cohesion` lifts live recovery 0.15 → 0.54 with no real-pair net-negatives. |
| `GOLDENMATCH_SUGGEST_COHESION`        | `min_edge`       | Cohesion statistic: `min_edge`, `mean_bottomk_edge`, or `edge_below_cutoff_fraction`.                                                                                                                                                                                                                                         | Leave alone; `min_edge` won the bake-off.                                                                                                                   |
| `GOLDENMATCH_SUGGEST_COVERAGE_CAP`    | `0.50`           | Coverage-saturation cap for the cohesion proxy (matched-rate at which coverage saturates to 1.0).                                                                                                                                                                                                                             | Leave alone; `0.50` is the safe operating point — tighter caps let a real-pair net-negative through.                                                        |
| `GOLDENMATCH_SUGGEST_VERIFY`          | on               | The self-verify gate. Set `0` to return raw kernel suggestions **unfiltered** (no no-net-negative guarantee).                                                                                                                                                                                                                 | Diagnostics only — to see what the kernel would propose before the gate.                                                                                    |
| `GOLDENMATCH_SUGGEST_FULL_DIST`       | `0` (off)        | Source the kernel's score distribution from a threshold-0 diagnostic run so the dip (`lower_threshold`) rule sees the full pre-threshold tail.                                                                                                                                                                                | Set `1` when you want the dip rule to fire on configs whose threshold is well above the score valley.                                                       |
| `GOLDENMATCH_SUGGEST_RECALL_MIN_SHED` | `0.02`           | Min recall the tighten-key rule must shed before it will even consider vetoing a matchkey.                                                                                                                                                                                                                                    | Leave alone; raising it makes the recall-shed guard more conservative.                                                                                      |
| `GOLDENMATCH_SUGGEST_RECALL_COH_ABS`  | `0.15`           | Max cohesion gain that still counts as "small" when weighing a recall shed.                                                                                                                                                                                                                                                   | Leave alone; the gym-selected operating point.                                                                                                              |
| `GOLDENMATCH_SUGGEST_RECALL_RATIO`    | `2.0`            | Min cohesion-gain-per-recall-shed ratio required to justify shedding recall.                                                                                                                                                                                                                                                  | Leave alone; higher = demands more cohesion payoff per unit recall lost.                                                                                    |

***

## Servers, security & storage

When you expose any server (MCP, REST, A2A, web UI) on a non-loopback host, it is **fail-closed**: it refuses to start without the matching bearer token.

| Variable                           | Used by                    | Notes                                                                                                                                                                                                                                                                                                                                        |
| ---------------------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_MCP_TOKEN`            | MCP server                 | Required on non-loopback binds. `/.well-known/` card stays public.                                                                                                                                                                                                                                                                           |
| `GOLDENMATCH_MCP_ALLOW_PUBLIC`     | MCP server                 | Opt-out of the fail-closed default: set `1`/`true` to intentionally run an **open, unauthenticated public** MCP server (e.g. a showcase deployment) without `GOLDENMATCH_MCP_TOKEN`. Unset/`0` = fail closed (a non-loopback bind without a token is refused). A set `GOLDENMATCH_MCP_TOKEN` still takes precedence.                         |
| `GOLDENMATCH_API_TOKEN`            | REST API                   | Required on non-loopback binds (all except `/health`).                                                                                                                                                                                                                                                                                       |
| `GOLDENMATCH_AGENT_TOKEN`          | A2A agent                  | Required on non-loopback binds (except `/health` + agent card).                                                                                                                                                                                                                                                                              |
| `GOLDENMATCH_WEB_TOKEN`            | Web UI                     | Required on non-loopback binds (except `/api/v1/healthz`).                                                                                                                                                                                                                                                                                   |
| `GOLDENMATCH_API_CORS_ORIGINS`     | REST API                   | Comma-separated allow-list; replaces wildcard CORS.                                                                                                                                                                                                                                                                                          |
| `GOLDENMATCH_ALLOWED_ROOT`         | All file-taking surfaces   | Opt-in path sandbox. When set, every user-supplied path must resolve under this root or the call is rejected. Recommended for deployed/shared instances (e.g. the Railway MCP server sets it to `/data`). Unset = no containment (correct local-first default).                                                                              |
| `GOLDENMATCH_MCP_MAX_UPLOAD_BYTES` | MCP server (inline upload) | Max decoded size for `file_content` / `upload_dataset` inline uploads. Default `67108864` (64 MB). Over the cap the call is rejected — pass a public `http(s)://` URL as `file_path` for larger datasets.                                                                                                                                    |
| `GOLDENMATCH_MCP_UPLOAD_TTL`       | MCP server (inline upload) | Age in seconds after which an inline-uploaded temp file is reaped. Default `86400` (24 h). Uploaded paths are ephemeral scratch; re-upload if you need one older than this.                                                                                                                                                                  |
| `GOLDENMATCH_MCP_SESSION_MAX`      | MCP server (session state) | Max number of per-MCP-session dedupe/match runs kept in memory so the stateful tools (`list_clusters`, `get_cluster`, `get_golden_record`, `explain_match`, `evaluate`, `export_results`, `match_record`, `find_duplicates`) can read a run made earlier in the same session. Default `64`; least-recently-used sessions evict past the cap. |
| `GOLDENMATCH_MCP_SESSION_TTL`      | MCP server (session state) | Seconds a stored session run stays readable (anchored to when the run was made, not last access). Default `3600` (1 h); past it the stateful tools return a clean "no run loaded" error until you re-run `agent_deduplicate`/`match_sources`.                                                                                                |

**Storage / persistence:**

| Variable                                                | Default                | Effect                                                                                                                                                                                                                      |
| ------------------------------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_DATABASE_URL` / `GOLDENMATCH_IDENTITY_DSN` | unset                  | Postgres/SQLite connection for the identity graph + review queue / Alembic migrations.                                                                                                                                      |
| `GOLDENMATCH_PREPARED_RECORD_STORE_DIR`                 | unset                  | Directory for the disk-backed prepared-record store.                                                                                                                                                                        |
| `GOLDENMATCH_PREPARED_RECORD_STORE_PERSIST`             | `0`                    | `1` keeps the prepared-record store across calls instead of cleaning it up.                                                                                                                                                 |
| `GOLDENMATCH_VECTOR_INDEX_DIR`                          | `.goldenmatch_vectors` | Default on-disk directory for the persistent semantic `VectorIndex` (embeddings + records + manifest) when a path isn't passed explicitly. The index survives across runs/processes so embeddings aren't rebuilt every run. |

**Product analytics (opt-in):**

Anonymous usage analytics — **off by default**. Emits only when `GOLDENMATCH_ANALYTICS` is on **and** a `POSTHOG_API_KEY` is set, so a user's machine stays silent unless they opt in (the project's own hosted services set the flag server-side). PII-free by construction: only coarse scalars (scale bands, backend, version) ever leave, never record values, column names, or paths. Full collection list + guarantees in `packages/python/goldenmatch/ANALYTICS.md`.

| Variable                | Default                    | Effect                                                                                           |
| ----------------------- | -------------------------- | ------------------------------------------------------------------------------------------------ |
| `GOLDENMATCH_ANALYTICS` | `0` (off)                  | `1`/`true`/`on` opts in to anonymous, PII-free usage events. Off = nothing is collected or sent. |
| `POSTHOG_API_KEY`       | unset                      | Destination PostHog project key. Required for any event to send; absent = silent no-op.          |
| `POSTHOG_HOST`          | `https://us.i.posthog.com` | PostHog ingestion host override.                                                                 |

***

## Gotchas that cost a day

Read these before a big run. Each is a real trap that produces a slow or wrong result with no error.

<AccordionGroup>
  <Accordion title="backend='ray' ran in-memory and never used the cluster">
    `backend="ray"` (or `dedupe_df(backend="ray")`) **alone runs the Phase 2 cheat-line**: it collects the whole frame to the driver and dedupes in memory. The cluster sits idle. To actually stream across workers you need `GOLDENMATCH_DISTRIBUTED_PIPELINE=2`, a Ray Dataset input with a global `__row_id__`, and you read golden output from `output_path` — not the return value. On a multi-node cluster also set a shared `GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH` (e.g. `gs://...`). For single-box scale, use `bucket`, not `ray`.
  </Accordion>

  <Accordion title="It ran 100x slower than the docs because native silently fell back to Python">
    Under the default `GOLDENMATCH_NATIVE=auto`, if the native wheel isn't importable (not installed, ABI mismatch, stale build) the scoring kernel **silently** runs in Python — orders of magnitude slower. On any run where speed matters, set `GOLDENMATCH_NATIVE=1` so a missing wheel raises instead of crawling. Confirm with `python -c "import goldenmatch_native"` or check `goldenmatch.core._native_loader.native_available()`.
  </Accordion>

  <Accordion title="The prep stage dominated wall on address-heavy data">
    The per-row Python address-normalize plugin is a real bottleneck at scale. Set `GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE=1` to use the Polars-native expression instead. It's off by default, so you have to opt in.
  </Accordion>

  <Accordion title="A distributed 100M run took hours instead of minutes">
    The scored pair set is small (\~1.76 GB at 100M) and fits one node. By default `GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD` is unset and routing is **memory-aware** — clustering stays in-memory (`scipy.csgraph`, \~60s) whenever the pair set fits driver RAM, so a 100M run should already keep clustering on the fast path. The multi-hour tail happens if something forces the *distributed* WCC (whose `gs://`-checkpoint rounds are slow): check you haven't set the threshold to `0` (forces the slow path on everything) or to a value **below** your pair count. See [Fast path at scale](#fast-path-at-scale-distribute-only-what-doesnt-fit). Distribute the *scoring* (the rows don't fit), not the *clustering* (the pairs do).
  </Accordion>

  <Accordion title="Results changed between runs for no reason">
    Cross-run auto-config memory (`GOLDENMATCH_AUTOCONFIG_MEMORY=1`, the default) lets a prior run influence the next. For reproducible benchmarks set `GOLDENMATCH_AUTOCONFIG_MEMORY=0`.
  </Accordion>

  <Accordion title="Python hung at import on Windows">
    Polars runs a WMI CPU probe at import that can hang on some Windows boxes. Set `POLARS_SKIP_CPU_CHECK=1`. (Add `PYTHONIOENCODING=utf-8` if non-ASCII output mangles the console.)
  </Accordion>
</AccordionGroup>

***

## Throughput tier (sketch-then-verify)

*New in #1083, builds on #1080.* An opt-in high-throughput path for large text
corpora where you care about near-duplicate recall but can trade precision for
speed. Instead of the full fuzzy/Fellegi-Sunter pipeline, it blocks with
MinHash/LSH (`lsh`) or SimHash (`simhash`) and confirms candidate pairs by cheap
sketch distance before any field-level scoring. Think: training data dedup, web
crawl dedup, paper-corpus cleaning.

### Enabling it

Pass `throughput=` to `dedupe_df`:

```python theme={null}
import goldenmatch as gm

# Float: sets recall_target (0 to 1). Adjusts LSH banding.
result = gm.dedupe_df(df, throughput=0.95)

# True: uses the default recall_target (0.95).
result = gm.dedupe_df(df, throughput=True)

# None (default): the throughput tier is off. Byte-identical to today.
result = gm.dedupe_df(df)

# ThroughputConfig: full control.
from goldenmatch import ThroughputConfig
result = gm.dedupe_df(df, throughput=ThroughputConfig(
    recall_target=0.90,
    similarity_threshold=0.75,
))
```

Or via environment variables:

| Variable                            | Default   | Effect                                                                                     |
| ----------------------------------- | --------- | ------------------------------------------------------------------------------------------ |
| `GOLDENMATCH_THROUGHPUT`            | `0` (off) | `1` enables the throughput tier with default settings.                                     |
| `GOLDENMATCH_THROUGHPUT_RECALL`     | `0.95`    | LSH recall target (tunes banding).                                                         |
| `GOLDENMATCH_THROUGHPUT_SIMILARITY` | unset     | Override the near-dup similarity threshold (Jaccard default `0.8`; cosine default `0.85`). |

### What it does

1. **Auto-selects blocking strategy.** Picks `simhash` (cosine/semantic) when an
   embedder is reachable, otherwise falls back to `lsh` (MinHash/Jaccard on
   shingles). Uses the longest text column in the frame.
2. **Blocks with sketch buckets.** Records that share at least one LSH band become
   candidates. The `recall_target` controls the banding split via the LSH
   probability curve.
3. **Verifies by sketch distance only.** Pairs that clear `similarity_threshold`
   are kept; the field-level fuzzy/FS scorer does not run.
4. **Raises `ThroughputNotApplicableError`** when the frame has no text column.
   Fix: pass an explicit `BlockingConfig` to direct it to a column.

### Posture and honest reporting

The throughput tier does **not** measure F1 or precision. `DedupeResult.throughput_posture`
(and the controller telemetry `throughput` block) carry the metrics the tier can
honestly report:

| Key                    | Meaning                                                                                                                                                                          |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `metric`               | `"jaccard"` (LSH) or `"cosine"` (SimHash)                                                                                                                                        |
| `recall_target`        | The recall target you passed (tunes banding)                                                                                                                                     |
| `similarity_threshold` | The sketch-distance threshold applied at verify                                                                                                                                  |
| `bands`                | LSH band count selected for the recall target                                                                                                                                    |
| `rows_per_band`        | Rows per band                                                                                                                                                                    |
| `expected_recall`      | LSH-theoretic recall at the banding: `1 - (1 - t^r)^b` where `t = similarity_threshold`, `r = rows_per_band`, `b = bands`. This is an analytic bound, not measured on your data. |
| `reduction_ratio`      | `1 - candidate_pairs / possible_pairs`. Observed.                                                                                                                                |
| `candidate_pairs`      | Pairs that passed blocking.                                                                                                                                                      |
| `verified_pairs`       | Pairs that passed sketch-distance verification.                                                                                                                                  |
| `notes`                | Human-readable posture string.                                                                                                                                                   |

`expected_recall` is the LSH-theoretic probability that a pair at exactly
`similarity_threshold` Jaccard/cosine collides in at least one band. It is
**not** a measured F1 or recall against ground truth. For pairs above threshold
the curve is tighter; for pairs below it, collision probability drops rapidly.
This is an honest bound, not a precision measurement.

### Measured throughput (benchmark)

The `bench-corpus-dedup` harness (`scripts/bench_corpus_dedup/`,
`.github/workflows/bench-corpus-dedup.yml`) measures the tier end-to-end against a
real public corpus, with injected ground-truth near-dups so recall is measurable.
On a slice of **FineWeb** (web crawl), the zero-config `dedupe_df(df, throughput=…)`
path measured:

|   docs | docs/sec | MB/sec | measured recall |
| -----: | -------: | -----: | --------------: |
| 14,000 |      275 |   0.84 |            0.44 |
| 70,000 |    1,192 |   3.64 |            0.43 |

docs/sec rises with scale because the wall is dominated by a fixed \~50s of
zero-config auto-config at these sizes; the sketch dedup itself is \~9s at 70k
(≈7,800 docs/sec raw). **Measured recall ≈ 0.43** is the honest LSH recall over a
mix of exact / partial-overlap / paraphrase near-dups — the paraphrase and partial
dups often do not collide in the LSH bands. (Contrast `expected_recall` above: the
per-pair collision probability *at* the similarity threshold, not a recall over a
real corpus — which is exactly why the bench measures recall directly.)

A deterministic **per-PR CI perf gate** (`throughput-gate` job, via
`scripts/bench_corpus_dedup/throughput_perf_gate.py`) guards regression on the
machine-independent cost — candidate pairs, reduction ratio, and measured recall on
a vendored offline corpus — against a committed baseline, so a change can't quietly
blow up the candidate count or drop recall.

> **Scale note:** the published numbers are the validated 14k–70k range. 100k+
> corpora currently bottleneck on the survivorship/golden stage; lifting that
> ceiling (and a datatrove head-to-head) are tracked follow-ups.

### Scope

The throughput tier is **single-node only**. Distributed throughput (#1084) and the
product surface (#1085) are follow-on issues. Default-off (`throughput=None`) is
byte-identical to today in every metric.

***

## Configuration precedence

When the same setting is reachable multiple ways, the more specific wins:

1. **Explicit API / CLI argument** (`dedupe_df(..., backend=...)`, `--backend`) — highest.
2. **Config field** (`GoldenMatchConfig(backend=...)`, YAML).
3. **Environment variable** (`GOLDENMATCH_*`).
4. **Built-in default** (auto-config / planner choice) — lowest.

So an env var sets the default behavior for a process, a config field overrides it for a run, and an explicit call argument overrides it for one call.

***

## Full environment-variable reference

Every `GOLDENMATCH_*` variable the package reads (plus the one `GOLDEN_*` exception, `GOLDEN_DIAGNOSTICS`), grouped by who should care. Defaults verified against source. Most users only ever touch the [recommended setups](#recommended-setups).

### Everyday & scale

| Variable                                  | Default    | Values                                | Effect                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ----------------------------------------- | ---------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_NATIVE`                      | `auto`     | `auto` / `1` / `0`                    | Native kernel gate. `1` requires native (raises if missing); `0` forces Python.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `GOLDENMATCH_NATIVE_ADDRESS_NORMALIZE`    | off        | `1`                                   | Polars-native address-normalize fast path.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `GOLDENMATCH_FRAME`                       | `arrow`    | `arrow`/`polars`                      | Frame backend. `arrow` is the DEFAULT since v3.0.0 (measured \~36% faster end-to-end on the 100K zero-config A/B): file ingest via pyarrow, the whole engine runs on the Frame seam, and result frames are `pyarrow.Table`. Since v3.1.0 polars is an OPTIONAL dependency and the polars-free install is the FAST configuration (measured head-to-head at 500K: 7.11s polars-free vs 7.55s polars-present, identical outputs -- the Rust fused kernels carry the hot paths). `polars` here requires `pip install 'goldenmatch[polars]'` (a clear error names the extra otherwise); the extra is a COMPATIBILITY surface (this classic lane, the kernel-absent golden fast replay, goldencheck cell-quality weighting), not an accelerator. Output-equivalence enforced by a differential harness + frozen fixtures. |
| `GOLDENMATCH_FRAME_LANE`                  | `1` (on)   | `0`/`1`                               | Arrow "frame lane" for the output-writing pipeline (keeps the spine arrow through the final collect). `0` restores the classic polars shim. Ignored when `GOLDENMATCH_COLUMNAR_PIPELINE=1`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `GOLDENMATCH_AUTOCONFIG_MEMORY`           | `1`        | `0`/`1`                               | Cross-run auto-config memory. `0` for reproducible/CI runs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `GOLDENMATCH_PLANNING_EFFORT`             | `normal`   | `fast`/`normal`/`thinking`/`einstein` | Auto-config iteration budget.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `GOLDENMATCH_DISTRIBUTED_PIPELINE`        | unset      | `1`/`2`                               | Distributed execution phase. `2` = real streaming (needs `__row_id__`, writes to `output_path`).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `GOLDENMATCH_ENABLE_DISTRIBUTED_RAY`      | `0`        | `0`/`1`                               | Let the planner pick `ray`. Does not enable streaming on its own.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `GOLDENMATCH_PLANNER_BUCKET`              | `1`        | `0`/`1`                               | Allow the planner to pick `bucket`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `GOLDENMATCH_BUCKET_DEFAULT`              | `1` (on)   | `0`/`off`/`false`/`no`                | Default in-memory **bucket** FS route on runs up to 750K rows (the #1798-safe path). Opt out to force the legacy per-block path. Distinct from `GOLDENMATCH_FS_DEFAULT_BUCKET`, which routes the FS bucket path itself.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `GOLDENMATCH_ATTRIBUTE_DEMOTION`          | `1` (on)   | `0`/`1`                               | Demote an EXACT matchkey on a shared group/list/facility value (shared clinic phone, campaign id, facility NPI) when its large shared-value groups do not co-agree on the person name. `0` restores the pre-2.7.0 behavior.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `GOLDENMATCH_DISCRIMINATIVE_VETO`         | `1` (on)   | `0`/`1`                               | The #1351 exact-key discriminative-power veto (demote a shared-locality exact key). `0` disables it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `GOLDENMATCH_DISCRIMINATIVE_TAU`          | `0.5`      | float \[0,1]                          | Co-agreement floor for the discriminative veto: an exact key whose large shared-value groups co-agree on the person name below this fraction is vetoed. Out-of-range values fall back to the default.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `GOLDENMATCH_TF_NAME_WEIGHTING`           | `1` (on)   | `0`/`1`                               | Data-driven per-dataset name-frequency downweight on `name_freq_weighted_jw` fields. `0` restores the static-census behavior.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `GOLDENMATCH_BASE_STORE`                  | `auto`     | `auto`/`memory`/`lance`               | Candidate base store for the ANN `match_one` path. `auto` picks Lance only above \~2M rows when installed.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `GOLDENMATCH_DUCKDB_SCORE_DB`             | unset      | path                                  | DuckDB pair-store spill path.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `GOLDENMATCH_NATIVE_RAYON_MIN_PAIRS`      | `20000000` | int                                   | Native kernel sequential-vs-rayon threshold.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `GOLDENMATCH_NATIVE_RAYON_MIN_BLOOM_ROWS` | `10000`    | int                                   | Native PPRL bloom-filter sequential-vs-rayon threshold (row count). `0` = always parallel; a very large value = always sequential. Sibling of `_RAYON_MIN_PAIRS` (the #688-style futex-park guard).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `GOLDENMATCH_CONFIG_LINT`                 | `warn`     | `warn`/`strict`/`off`                 | Pre-flight config linter: warn-and-run, refuse on error, or skip.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `GOLDENMATCH_THROUGHPUT`                  | `0` (off)  | `0`/`1`                               | Enable the sketch-then-verify throughput tier. Equivalent to `dedupe_df(throughput=True)`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `GOLDENMATCH_THROUGHPUT_RECALL`           | `0.95`     | float                                 | LSH recall target; tunes the band/row split.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `GOLDENMATCH_THROUGHPUT_SIMILARITY`       | unset      | float                                 | Override the near-dup similarity threshold (Jaccard `0.8` / cosine `0.85` when unset).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `GOLDENMATCH_AUTO_SEMANTIC_BLOCKING`      | on         | `0`/`1`                               | Auto-enable SimHash-over-embeddings blocking for text-heavy data (#1090). Default-on; a no-op when no embedder is reachable. `0` disables.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `GOLDENMATCH_SEMANTIC_BLOCKING_THRESHOLD` | `0.6`      | float (0,1)                           | Recall threshold for auto semantic blocking; lower = more bands = higher recall. Drives the SimHash band/row split.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |

### Distributed sub-knobs

| Variable                                       | Default              | Effect                                                                                                                       |
| ---------------------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_DISTRIBUTED_BLOCK_SHUFFLE`        | `1` (on)             | Block-shuffle scoring (recall-complete path). `0` = legacy `local_cc`.                                                       |
| `GOLDENMATCH_DISTRIBUTED_WCC_SCRATCH`          | unset                | Randomized-contraction WCC scratch. **Must be shared (`gs://...`) on multi-node.**                                           |
| `GOLDENMATCH_DISTRIBUTED_WCC`                  | `two_phase`          | WCC for `build_clusters_distributed`'s own callers (Phase-5-at-scale forces `randomized_contraction`). Avoid `pointer_jump`. |
| `GOLDENMATCH_DISTRIBUTED_WCC_SEED`             | unset                | Randomized-contraction hash seed (reproducible cluster IDs).                                                                 |
| `GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD` | unset (memory-aware) | Pair count → distributed clustering. Unset default routes by driver-RAM fit; set an integer to force a hard boundary.        |
| `GOLDENMATCH_DISTRIBUTED_GOLDEN_THRESHOLD`     | `5000000`            | Row count → distributed golden.                                                                                              |
| `GOLDENMATCH_DISTRIBUTED_SCORE_NUM_CPUS`       | `2`                  | CPUs per Ray scoring worker.                                                                                                 |

### Perf opt-ins & diagnostics

| Variable                                                                                     | Default                     | Effect                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| -------------------------------------------------------------------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_BUCKET_SLIM_PROJECTION`                                                         | `1` (on)                    | Drop unneeded columns after bucket scoring.                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `GOLDENMATCH_GOLDEN_SLIM_MULTIDF`                                                            | `1` (on)                    | Slim internal columns before golden build.                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `GOLDENMATCH_GOLDEN_FUSED`                                                                   | on                          | Golden-seam fused auto-routing (default-on, byte-identical, \~2x lower peak RSS). `0` kills it. See [Fused kernel auto-routing](#fused-kernel-auto-routing).                                                                                                                                                                                                                                                                                                              |
| `GOLDENMATCH_MATCH_FUSED`                                                                    | on                          | Match-seam fused auto-routing (weighted + Fellegi-Sunter) - a narrow capacity-survival path that fires only under memory pressure on an artifact-free config. `0` kills it. See [Fused kernel auto-routing](#fused-kernel-auto-routing).                                                                                                                                                                                                                                  |
| `GOLDENMATCH_FUSED_PRESSURE_FRACTION`                                                        | `0.65`                      | RAM-fraction pressure gate for the match-seam routing.                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `GOLDENMATCH_FUSED_RSS_SCALE` / `_BYTES_PER_PAIR` / `_BYTES_PER_CELL` / `_BLOCK_CONCURRENCY` | `0.763` / `64` / `40` / `4` | est-peak-RSS calibration coefficients for the match pressure gate.                                                                                                                                                                                                                                                                                                                                                                                                        |
| `GOLDENMATCH_IDENTITY_BATCH_FINGERPRINT`                                                     | `1` (on)                    | Batch no-PK identity hashing.                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `GOLDENMATCH_IDENTITY_WRITE_PIPELINE`                                                        | `1` (on)                    | Pipeline the per-record identity writes on remote Postgres (#1912).                                                                                                                                                                                                                                                                                                                                                                                                       |
| `GOLDENMATCH_BUCKET_DEBUG`                                                                   | `0`                         | Per-bucket timing breakdown (diagnostic).                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `GOLDEN_DIAGNOSTICS`                                                                         | `1` (on)                    | Anomaly diagnostics (note: no `MATCH` in the name). On states that are probably a GoldenMatch bug -- a native wheel-skew slow path, an unexpected `dedupe_df`/`match_df` crash, the config linter itself crashing, or a broken native install -- emit a warning with a prefilled GitHub issue URL plus a PII-safe environment report. Never fires on expected fallbacks or user errors, and sends nothing anywhere (a better error message, not telemetry). `0` disables. |
| `GOLDENMATCH_PREP_STAGED_COLLECT`                                                            | `0`                         | Staged prep collect (experimental, may slow).                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `GOLDENMATCH_STANDARDIZE_STAGED`                                                             | `0`                         | Staged standardization (experimental, may slow).                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `GOLDENMATCH_BUCKET_VEC_MIN` / `_MAX`                                                        | `32` / `2000`               | Block-size band for the vectorized float64 NxN cdist path in bucket scoring.                                                                                                                                                                                                                                                                                                                                                                                              |
| Fellegi-Sunter routing & scoring knobs (`GOLDENMATCH_FS_*`)                                  | —                           | Have their own home: see [Probabilistic / Fellegi-Sunter](#probabilistic-fellegi-sunter-advanced) for the path-routing map + the complete `FS_*` table.                                                                                                                                                                                                                                                                                                                   |
| `GOLDENMATCH_BENCH_DUMP_PAIRS`                                                               | unset                       | Dump per-stage pairs for benchmarking.                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `GOLDENMATCH_SKIP_MATCH_LOG` / `GOLDENMATCH_MATCH_LOG_FLUSH_PAIRS`                           | —                           | Streaming match-log controls.                                                                                                                                                                                                                                                                                                                                                                                                                                             |

### Auto-config & blocking (advanced)

| Variable                                                            | Default          | Effect                                                                                                                                                                                                                                                                                                                                                                  |
| ------------------------------------------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_AUTOCONFIG_LLM`                                        | `0`              | LLM config fallback (needs API key).                                                                                                                                                                                                                                                                                                                                    |
| `GOLDENMATCH_AUTOCONFIG_INDICATOR_BUDGET`                           | unset            | `fast` skips expensive indicators.                                                                                                                                                                                                                                                                                                                                      |
| `GOLDENMATCH_AUTOCONFIG_ARROW_NATIVE`                               | `1` (on)         | Keep auto-config on the arrow-native path instead of coercing the arrow input to polars. Auto-forced off (warn-once per feature) for the sub-features that still assume a polars target (throughput tier, cross-source match). `0` = always coerce arrow→polars.                                                                                                        |
| `GOLDENMATCH_AUTOCONFIG_FORCE_INCLUDE` / `_FORCE_EXCLUDE`           | unset            | Force columns into/out of analysis.                                                                                                                                                                                                                                                                                                                                     |
| `GOLDENMATCH_FIELD_GROUP_SURVIVORSHIP`                              | `0` (off)        | Auto-detect correlated field groups for survivorship. Equivalent to `field_group_detection: true` in `golden_rules`. Off = byte-identical to prior behavior. Group promotions are visible in the lineage `golden_records` section, `explain --cluster`, and the MCP `lineage` tool.                                                                                     |
| `GOLDENMATCH_AUTOCONFIG_SAMPLE_STRATEGY`                            | stratified       | `random` forces random sampling.                                                                                                                                                                                                                                                                                                                                        |
| `GOLDENMATCH_AUTOCONFIG_ZERO_LABEL_COMMIT`                          | `1`              | Zero-label commit tiebreaker. `0` = legacy.                                                                                                                                                                                                                                                                                                                             |
| `GOLDENMATCH_AUTOCONFIG_ZERO_LABEL_STABILITY`                       | `0`              | Perturbation re-runs (slow, experimental).                                                                                                                                                                                                                                                                                                                              |
| `GOLDENMATCH_MULTISOURCE_AUTOCONFIG`                                | `1` (on)         | Multi-source-aware auto-config. `0`/`false`/`disabled` = legacy single-source.                                                                                                                                                                                                                                                                                          |
| `GOLDENMATCH_BLOCKING_PAIRS_PER_ROW`                                | `50`             | Candidate-pairs-per-row budget the blocking selector targets.                                                                                                                                                                                                                                                                                                           |
| `GOLDENMATCH_BLOCKING_MAX_RATIO`                                    | `0.95`           | Skip blocking keys above this cardinality ratio.                                                                                                                                                                                                                                                                                                                        |
| `GOLDENMATCH_BLOCKING_MIN_RATIO`                                    | `0.01`           | Minimum cardinality ratio for a blocking key.                                                                                                                                                                                                                                                                                                                           |
| `GOLDENMATCH_BLOCKING_CARDINALITY_SCALER`                           | Chao1            | `observed` uses sample-observed cardinality.                                                                                                                                                                                                                                                                                                                            |
| `GOLDENMATCH_BLOCKING_DEGENERATE_THRESHOLD` / `_MAX_AVG_BLOCK_SIZE` | `0.95` / `10000` | Degenerate-blocking guard.                                                                                                                                                                                                                                                                                                                                              |
| `GOLDENMATCH_BLOCKING_PRUNE_PASSES` / `_PASS_MIN_WEAKPOS`           | `0` / `1`        | Weak-positive multi-pass pruning.                                                                                                                                                                                                                                                                                                                                       |
| `GOLDENMATCH_QUALITY_AWARE_BLOCKING`                                | `0`              | Add fuzzy passes for noisy columns (GoldenCheck door #1).                                                                                                                                                                                                                                                                                                               |
| `GOLDENMATCH_FD_NEGATIVE_EVIDENCE`                                  | `0`              | FD-driven negative evidence (GoldenCheck door #3).                                                                                                                                                                                                                                                                                                                      |
| `GOLDENMATCH_FACILITY_NAME_NE`                                      | `0`              | On company/location-mode data (shared phone/address/company demoted because the sharers don't co-agree on the person name), add the person full name (first+last) as `token_sort` negative evidence so distinct colleagues at one facility don't fuse. Opt-in; only fires when facility mode is detected, so person datasets (e.g. Febrl) are unaffected even when set. |
| `GOLDENMATCH_QUALITY_GATED_REVIEW`                                  | `0`              | Downgrade auto-merge on low-quality records.                                                                                                                                                                                                                                                                                                                            |

### Probabilistic Fellegi-Sunter (advanced)

A `type: probabilistic` matchkey trains EM once, then its candidate pairs are scored by **one of four paths, chosen automatically** — you rarely set these by hand, but this is the map when you need to:

* **bucket** (default) — memory-bounded field-hash buckets; the #1798-safe route for `static`/`multi_pass` blocking on the default backend.
* **external-blocks** — memory-bounded scorer for strategy-generated candidates (`lsh`/`ann`/`learned`/`canopy`/`sorted_neighborhood`) that buckets can't re-derive from field hashes.
* **batched** — the legacy per-block path. Reached by `GOLDENMATCH_FS_DEFAULT_BUCKET=0` or an explicit non-default backend. Eager `build_blocks`, so memory grows with dataset size.
* **fused** — a single Arrow-native FFI call (block + score + dedup + cluster) under est-RSS pressure on an artifact-free config; 2–19× leaner peak RSS. See [Fused kernel auto-routing](#fused-kernel-auto-routing).

Within whichever path is chosen, the per-block math runs on the native Rust FS kernel by default (`GOLDENMATCH_FS_NATIVE`), falling back to the vectorized numpy scorer (`GOLDENMATCH_FS_VECTORIZED`) and then the scalar loop. Every knob below is an escape hatch or an advanced tuning lever — the defaults are the recommended path.

**Path routing**

| Variable                                     | Default  | Effect                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| -------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_FS_DEFAULT_BUCKET`              | `1` (on) | Route `static`/`multi_pass` FS on the default backend to the memory-bounded **bucket** scorer (the #1798 OOM fix). `0` = legacy **batched** path (eager `build_blocks`, memory grows with N) — the literal "legacy" hatch; also disables the external-blocks route.                                                                                                                                                                                                     |
| `GOLDENMATCH_MATCH_FUSED`                    | on       | Whole-stage **fused** match kernel (weighted **and** FS). A covered FS run routes here under est-RSS pressure on an artifact-free config. `0`/`false`/`off` kill-switch. See [Fused kernel auto-routing](#fused-kernel-auto-routing).                                                                                                                                                                                                                                   |
| `GOLDENMATCH_AUTOCONFIG_ROUTE_PROBABILISTIC` | `0`      | Route zero-config auto-config to the **probabilistic (FS)** matchkey builder instead of the weighted one.                                                                                                                                                                                                                                                                                                                                                               |
| `GOLDENMATCH_FS_ROUTE_MIN_ROWS`              | `500`    | Small-N floor on FS routing: a probabilistic-shaped dataset with fewer rows stays on the robust **weighted** path (FS EM is data-starved at small N — it commits a RED config and its discrete-level comparison under-merges fuzzy-close variants). `0` disables the floor (route purely by shape). Every dataset that validated the FS-default win sits far above the floor, so routing there is unchanged; only tiny corpora (KG entity sets, small demos) fall back. |

**Scoring engine (native vs numpy vs scalar)**

| Variable                                | Default        | Effect                                                                                                                                                                                                                                |
| --------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_FS_NATIVE`                 | on (`auto`)    | Native Rust FS block scorer — the reference/answer under `auto`. NE-bearing matchkeys score natively when the wheel advertises `FS_SUPPORTS_NE` (`goldenmatch-native` ≥ 0.1.15). `0`/`false`/`off` = the reproducible numpy fallback. |
| `GOLDENMATCH_FS_BUCKET_NATIVE`          | `1` (on)       | Use the **batched** native scorer inside the bucket route. `0` = the per-block native loop (byte-identical parity hatch). Only gates the batched call; per-block nativeness still follows `GOLDENMATCH_FS_NATIVE`.                    |
| `GOLDENMATCH_FS_VECTORIZED`             | on             | Vectorized (`rapidfuzz.cdist` N×N) block scoring (\~9×). `0` = scalar per-pair loop. String scorers only — `embedding`/`record_embedding` are matrix-only and always run vectorized.                                                  |
| `GOLDENMATCH_FS_BATCH_ROWS`             | `256`          | Small-block coalescing cap for the batched vectorized scorer (amortizes per-call FFI/marshal overhead). Sweep knee; raising past it stops helping.                                                                                    |
| `GOLDENMATCH_FS_VEC_MAX_ELEMS`          | `50000000`     | Max N×N matrix elements before a block is split — the vectorized-scorer memory guard against a single mega-block.                                                                                                                     |
| `GOLDENMATCH_FS_WORKERS`                | `min(16, cpu)` | Thread-pool size for the batched FS scorer (both kernels release the GIL). `1` forces the deterministic sequential reference loop.                                                                                                    |
| `GOLDENMATCH_DISTRIBUTED_FS_TRAIN_ROWS` | `200000`       | EM training-sample row cap on the distributed FS path.                                                                                                                                                                                |

**Model & math**

| Variable                                   | Default     | Effect                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------------------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_FS_AUTOCONFIG_V2`             | `1` (on)    | FS auto-config v2 comparison-set + blocking curation. `0` = legacy field set.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `GOLDENMATCH_FS_DOMAIN_COMPARATORS`        | `0` (off)   | Admit `date` columns with the magnitude-aware `date_diff` scorer (day-distance bands: a full-year DOB gap is a weak partial) instead of `levenshtein`. Scale-neutral (just a new scorer — same blocks/pairs/EM/clustering). Default off is byte-identical. Spec: `docs/superpowers/specs/2026-07-23-fs-domain-comparators-design.md`.                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `GOLDENMATCH_FS_TF_ADJUSTMENT`             | `0` (off)   | Winkler term-frequency adjustment on skewed categorical fields (name/string/geo/zip): an exact agreement on a rare value out-weights one on a common value ("Zelinski" > "Smith"). Targets the over-merge regime (historical\_50k precision below Splink). Stays on the native FS path (scale-safe); default off is byte-identical.                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `GOLDENMATCH_FS_STRIP_HONORIFICS`          | `0` (off)   | Append the `strip_honorifics` transform to name comparison fields, so a title/rank token leaked into a name field ("Sir", "Baronet", "Bt.") stops carrying match weight (a honorific-only value becomes missing, not an empty-string agreement). Reaches the over-merge that `GOLDENMATCH_FS_TF_ADJUSTMENT` can't on historical\_50k (F1 +0.0108, precision +0.0245 in the A/B). Regnal numerals are kept. Default off is byte-identical.                                                                                                                                                                                                                                                                                                                                                           |
| `GOLDENMATCH_FS_CALIBRATED`                | `linear`    | `posterior` = true match-probability calibration (frontier-neutral — ranks pairs identically, so it can't change F1; it makes the score a real probability with a meaningful 0.5 boundary).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `GOLDENMATCH_FS_MONOTONIC`                 | `warn`      | Non-monotonic EM-weight handling. `warn` (default) = detect + warn, **don't modify** (Splink posture). `enforce`/`1`/`on` = isotonic (PAV) repair. `0`/`off` = silent (no detection).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `GOLDENMATCH_FS_MISSING`                   | per-dataset | Missing-value semantics (#1846). `unobserved` = textbook FS (a missing value is absence of evidence); `disagree` = evidence against a match. Env overrides `mk.missing`; by default auto-config picks per-dataset from profiled null rates.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `GOLDENMATCH_TF_NAME_WEIGHTING`            | `1` (on)    | Data-driven term-frequency down-weighting for `name_freq_weighted_jw` fields (shared with the weighted path). `0` = static-census behavior.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `GOLDENMATCH_FS_EM_SAMPLE_ROWS`            | `100000`    | Cap on the frame rows used to build EM-training blocks on the bucket route (scoring re-partitions internally, so only EM training sees the sample). Bounds the FS memory peak at scale; byte-identical F1 when `height <= cap`. `0` restores full-frame EM.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `GOLDENMATCH_FS_EM_BLOCK_SLIM`             | `1` (on)    | Project the EM-training block frame to `__row_id__` + the blocking-key columns before `build_blocks` (EM reads only those). Cuts the EM `build_blocks` memory peak; byte-identical output. `0` restores full-width blocks (the parity oracle).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `GOLDENMATCH_FS_EM_AGG_BLOCKS`             | `1` (on)    | Build the EM-training blocks as compact `int64` row-id arrays via one `group_by().agg()` per pass (no per-block frames) — supersedes the block-slim lever, killing the per-block-object floor. Byte-identical output on `static`/`multi_pass`. `0` restores the frame path.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `GOLDENMATCH_FS_OUT_OF_CORE`               | `0` (off)   | Opt-in single-box **out-of-core** FS path for datasets past the \~40M in-memory wall: the prepared frame spills to a DuckDB file and blocks stream one group at a time (bounded peak). Reached via `dedupe_to_parquet(..., out_dir=...)` on a single-probabilistic-matchkey `static`/`multi_pass` config, or the scoring branch inside `dedupe_df`. Byte-parity with the in-memory route absent oversized blocks.                                                                                                                                                                                                                                                                                                                                                                                   |
| `GOLDENMATCH_FS_OOC_ARROW_CLUSTER`         | `1` (on)    | Within the out-of-core path, keep the scored pairs as an Arrow `PAIR_STREAM` table (never `list[tuple]`), dedup them with the native `dedup_pairs_arrow` kernel, and cluster with the Rust `build_clusters_arrow_native` Union-Find — so pairs never enter Python as objects and no `dict[int,dict]` is built. `0` restores the `list[tuple]` + Python `build_clusters` path (rollback). Distinct from the in-memory `GOLDENMATCH_FS_ARROW_STREAM`.                                                                                                                                                                                                                                                                                                                                                 |
| `GOLDENMATCH_FS_OOC_DEBUG`                 | `0` (off)   | Print a per-phase progress line for the out-of-core FS scorer (load+index, and per blocking pass: block-map build time, scan+score wall, block-row count), so a long-running ≥25M streaming leg shows live progress instead of a blank spinner. Output-invariant.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `GOLDENMATCH_FS_MAX_PASS_PAIRS`            | `300000000` | Global total candidate-pair budget for probabilistic-blocking auto-config. Bounds the emitted passes so a low-cardinality pass can't compound into a megablock (the OOM/over-generation gate).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `GOLDENMATCH_FS_ARROW_STREAM`              | `0` (off)   | Opt the eligible single-matchkey FS bucket dedupe path into incremental per-bucket Arrow pair-stream accumulation (banks the exclude-set + scoring-accumulation peak). Byte-identical clusters.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `GOLDENMATCH_FS_COLUMNAR_CLUSTER`          | `0` (off)   | B2c (#1811): thread the Arrow pair stream STRAIGHT to the columnar cluster path (`build_clusters_columnar` over `_columnar_pairs_df`) on the eligible single-FS-matchkey in-memory bucket dedupe, so the driver-resident `all_pairs` Python `list[tuple]` is never built during scoring → clustering — the accumulator that runs to hundreds of millions of tuples before clustering starts at 14M (the late-stage OOM of #1811). The in-memory analogue of `GOLDENMATCH_FS_OOC_ARROW_CLUSTER`; superset of `GOLDENMATCH_FS_ARROW_STREAM` (also drops `matched_pairs`). Clusters are NOT byte-identical (the FS bucket pipeline is \~0.1%-nondeterministic run-to-run regardless). The post-cluster `scored_pairs` list is deduped columnar + shed above `GOLDENMATCH_FS_SCORED_PAIRS_MAX` (#2006). |
| `GOLDENMATCH_FS_SCORED_PAIRS_MAX`          | `50000000`  | B2c (#2006): cap on how many deduped pairs the FS columnar-cluster path materializes into `DedupeResult.scored_pairs`. Above the cap the driver-resident `list[tuple]` is SHED (empty list + `scored_pairs_shed=True`) — the last O(pairs) accumulator #1811 left; `clusters`/`golden`/`dupes`/`unique` are unaffected, only the steward-facing raw pair list drops, and never silently (the marker). `0` disables the cap (always materialize). Scoped to `GOLDENMATCH_FS_COLUMNAR_CLUSTER`; the weighted columnar lane is unchanged.                                                                                                                                                                                                                                                              |
| `GOLDENMATCH_FS_REQUIRE_POSITIVE_EVIDENCE` | `1` (on)    | Require strictly positive net evidence (summed match weight `W > 0`) to link a pair in **linear** mode — drops `W <= 0` pairs the min-max normalization would otherwise map onto a score `>= 0.5`, killing the scale-growing FS over-merge. `0` restores the legacy emit-at-neutral behavior. Linear-only (posterior calibration already folds the prior into the 0.5 Bayes cut); the wasm/DuckDB/Postgres surfaces pass `false`.                                                                                                                                                                                                                                                                                                                                                                   |

**Other advanced (scorer / golden / cluster — not FS-specific)**

| Variable                                                                            | Default        | Effect                                                                |
| ----------------------------------------------------------------------------------- | -------------- | --------------------------------------------------------------------- |
| `GOLDENMATCH_NOISE_AWARE_SCORERS`                                                   | `1` (on)       | Auto-upgrade `token_sort`→`jaro_winkler` on noisy text. `0` = legacy. |
| `GOLDENMATCH_NOISE_AWARE_TARGET`                                                    | `jaro_winkler` | Override the noise-aware target scorer.                               |
| `GOLDENMATCH_GOLDEN_STRATEGY_STRICT`                                                | `0`            | Strict golden survivorship.                                           |
| `GOLDENMATCH_GOLDEN_TUNER_MIN_CORRECTIONS` / `GOLDENMATCH_NE_TUNER_MIN_CORRECTIONS` | —              | Memory re-tuning floors.                                              |
| `GOLDENMATCH_CLUSTER_SPLIT_EDGE_BUDGET`                                             | ∝ n\_rows      | Oversized-cluster auto-split work budget.                             |
| `GOLDENMATCH_COLUMNAR_PIPELINE`                                                     | `0`            | Columnar pipeline experiment (Phase A opt-in).                        |

### Servers, security, storage, LLM/embedding

| Variable                                                               | Default                                                        | Effect                                                                                                                                                                                                                                                                                                                                                                                      |
| ---------------------------------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOLDENMATCH_MCP_TOKEN` / `_API_TOKEN` / `_AGENT_TOKEN` / `_WEB_TOKEN` | unset                                                          | Bearer tokens; required on non-loopback binds.                                                                                                                                                                                                                                                                                                                                              |
| `GOLDENMATCH_API_CORS_ORIGINS`                                         | unset                                                          | REST CORS allow-list.                                                                                                                                                                                                                                                                                                                                                                       |
| `GOLDENMATCH_ALLOWED_ROOT`                                             | unset                                                          | Opt-in path sandbox root.                                                                                                                                                                                                                                                                                                                                                                   |
| `GOLDENMATCH_DATABASE_URL` / `GOLDENMATCH_IDENTITY_DSN`                | unset                                                          | DB connection / migration DSN.                                                                                                                                                                                                                                                                                                                                                              |
| `GOLDENMATCH_SAIL_IDENTITY_ID_SCHEME`                                  | `h1`                                                           | Identity record-id scheme on the Sail (Spark Connect) tier.                                                                                                                                                                                                                                                                                                                                 |
| `GOLDENMATCH_PREPARED_RECORD_STORE_DIR` / `_PERSIST`                   | unset / `0`                                                    | Prepared-record store path / persistence.                                                                                                                                                                                                                                                                                                                                                   |
| `GOLDENMATCH_LLM_MODEL`                                                | provider default                                               | Override the LLM scorer model.                                                                                                                                                                                                                                                                                                                                                              |
| `GOLDENMATCH_EMBEDDING_PROVIDER` / `GOLDENMATCH_INHOUSE_MODEL`         | unset                                                          | Embedding provider / in-house model id.                                                                                                                                                                                                                                                                                                                                                     |
| `GOLDENMATCH_GPU_MODE` / `_GPU_ENDPOINT` / `_GPU_API_KEY`              | auto                                                           | GPU embedding mode / remote endpoint / key.                                                                                                                                                                                                                                                                                                                                                 |
| `GOLDENMATCH_LLAMA_GGUF`                                               | unset                                                          | Path to a local GGUF embedding model for the `model="llama"` / `"llama:/path.gguf"` backend — offline embeddings, no cloud/torch (`pip install goldenmatch[llama]`). Benchmarks: `scripts/bench_llama_embedder.py`, `scripts/bench_llama_product_matching.py`.                                                                                                                              |
| `GOLDENMATCH_LLM_BASE_URL`                                             | `https://api.openai.com/v1`                                    | OpenAI-compatible base URL for the LLM scorer — point at a local llama.cpp / llamafile server for offline, free LLM judging (also honors `OPENAI_BASE_URL`).                                                                                                                                                                                                                                |
| `GOLDENMATCH_WEAKNESS_LLM`                                             | `0` (off)                                                      | Opt-in: let the `config_weaknesses` tool phrase its one-paragraph `summary_plain` with an LLM (needs `OPENAI_API_KEY`/`ANTHROPIC_API_KEY`). Only a compact structured digest of the findings (ids/severities/short labels) is sent — never raw rows or data. The structured findings themselves are always deterministic and never depend on the LLM; off = deterministic template summary. |
| `GOLDENMATCH_WEAKNESS_LLM_MODEL`                                       | `gpt-4o-mini` (OpenAI) / `claude-3-5-haiku-latest` (Anthropic) | Model id for the `GOLDENMATCH_WEAKNESS_LLM` summary, per detected provider.                                                                                                                                                                                                                                                                                                                 |
| `GOLDENMATCH_UDF_IMPORTS`                                              | unset                                                          | Snowflake UDF import asset directory.                                                                                                                                                                                                                                                                                                                                                       |
| `GOLDENMATCH_SYNC_STREAMING_THRESHOLD` / `_STREAMING_BLOCK_WORKERS`    | `500000` / auto                                                | Postgres streaming-store thresholds.                                                                                                                                                                                                                                                                                                                                                        |
| `GOLDENMATCH_ANALYTICS`                                                | `0` (off)                                                      | Opt-in anonymous, PII-free usage analytics to PostHog (needs `POSTHOG_API_KEY`). Off = nothing collected.                                                                                                                                                                                                                                                                                   |

<Note>
  Not a `GOLDENMATCH_*` var but relevant: set `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` for LLM scoring and the LLM auto-config fallback, and `POLARS_SKIP_CPU_CHECK=1` on Windows to skip the Polars import-time CPU probe.
</Note>

<Card title="Backends and scale" icon="server" href="/docs/goldenmatch/backends-and-scale">
  The measured numbers behind each backend choice.
</Card>

<Card title="Scale envelope" icon="gauge-high" href="/docs/concepts/scale-envelope">
  The block-size failure modes that matter more than the backend.
</Card>

## Distributed routing

The planner decides scoring, clustering, and golden routing per stage, keyed off
**driver RAM** (not cluster total). The `lint_routing` / `explain_routing` MCP
tools surface these decisions and flag overrides that force a slow path; a
slow-path override at scale refuses unless `allow_slow_path=true`.

<h3 id="routing-single-box">
  Single box
</h3>

With no Ray cluster connected, every stage runs in-memory / in-process. There is
nothing to distribute to, so the projections below do not apply.

<h3 id="routing-driver-ram-projection">
  Driver-RAM projection
</h3>

When a cluster is present, each materializing stage distributes only when its
working set exceeds the driver-RAM budget (`available_ram_gb × 0.6`):

* **Clustering / WCC** distributes when the projected edge set
  (`estimated_pair_count × 16 bytes`) exceeds the budget. At 100M rows the
  \~1.76 GB edge set fits, so WCC runs in-memory (scipy) — forcing the
  distributed path there is a multi-hour tail.
* **Scoring** and **golden** distribute when the row-frame estimate
  (`n_rows × 512 bytes`) exceeds the budget. The same data therefore distributes
  scoring on a 48 GB-driver cluster but keeps it in-memory on a 256 GB box.

<h3 id="routing-overrides">
  Overrides
</h3>

`distributed_routing.*` config pins and the legacy
`GOLDENMATCH_DISTRIBUTED_CLUSTERING_THRESHOLD` env var are honored but linted.
`CLUSTERING_THRESHOLD=0` forces distributed WCC regardless of the projection; at
scale the linter flags it ERROR and the runtime refuses unless
`allow_slow_path=true`.