Skip to main content
The introspective AutoConfig controller is what lets GoldenMatch beat hand-tuned baselines with no input. It detects column types, selects scorers, picks a blocking strategy, then iterates on signals the pipeline emits until it converges on a defensible config.

​
What it does

Given a file, the controller:
  1. Detects column types (name, email, phone, zip, address, description).
  2. Selects appropriate scorers per column.
  3. Picks a blocking strategy.
  4. Runs the pipeline and reads back complexity signals.
  5. Applies refinement rules and repeats until health stops improving.

​
Signals it watches

The controller iterates on a ComplexityProfile:
  • Block-size distribution (p50 / p95 / p99).
  • Score histogram (bimodality detection).
  • Transitivity rate.
  • Borderline mass (pairs near the threshold).
  • Negative-evidence collision rate (v1.11+).

​
Refinement rules

The HeuristicRefitPolicy applies rules such as: Configs are ranked by a health metric: GREEN > YELLOW > RED, with the initial config as a virtual fallback. Precision-collapse detection. The precision-anchor rule doubles as a labels-free precision-collapse detector at commit time: once it has fired, commit selection rank-demotes any candidate config whose shape still trips the rule’s trigger, and the score-distribution unimodality (dip) gate only reads RED with at least 30 scored pairs behind it (a flat dip over fewer pairs is sampling noise, not evidence). Measured on the crafted over-merge fixture: precision 0.009 -> 0.9868 at recall 1.0, with NCVR unaffected.
At 100,000+ rows, auto-config raises ControllerNotConfidentError rather than silently committing a low-confidence (RED) config. Handle this explicitly instead of adding a silent fallback path.

​
Planning effort

The planning-effort tier controls how hard the controller searches for a config. Pass it as a planning_effort= kwarg to dedupe_df / match_df / auto_configure_df, set it on a GoldenMatchConfig, or use the GOLDENMATCH_PLANNING_EFFORT env var. The default normal is byte-for-byte the prior behavior.
Because block scoring is now ~5x faster (bucket+native), the higher tiers measure the true candidate-pair count on the full data rather than projecting it from a 2K–20K sample — which removes the wrong-backend-on-skewed-data failure. Any measurement failure falls back to extrapolation.

​
In-house embeddings

When you point auto-config at the local in-house embedding model (a matchkey field with model="inhouse:/path", or GOLDENMATCH_EMBEDDING_PROVIDER=inhouse + GOLDENMATCH_INHOUSE_MODEL), it is treated as a local, offline-safe scorer and is not demoted as a remote-asset drift risk. Cloud embedders (sentence-transformers / Vertex) still require allow_remote_assets=True.

​
Environment variables

​
Cross-run memory

By default the controller remembers what worked on prior runs and seeds future runs from it. Disable it with GOLDENMATCH_AUTOCONFIG_MEMORY=0.

Scale envelope

The block-size guards the controller respects, and how it picks a backend.