Identity graph

​

What problem this solves

• Cluster IDs are different.
• There is no record of “Alice Smith merged into entity 7 because of evidence X.”
• A second source pointing at the same person has no way to find the existing identity.
• The web/SQL/agent surfaces all reason from scratch each time.

​

Quickstart

matchkeys:
  - name: people
    type: weighted
    threshold: 0.85
    fields:
      - { field: name,  scorer: jaro_winkler, weight: 0.7 }
      - { field: email, scorer: exact,        weight: 0.3 }

blocking:
  strategy: static
  keys:
    - fields: [zip]

identity:
  enabled: true
  source_pk_column: id
  dataset: customers

import goldenmatch as gm
result = gm.dedupe("customers.csv", config="goldenmatch.yml")
print(result["identity_summary"])
# {'created': 12, 'absorbed_records': 0, 'merged': 0,
#  'split': 0, 'edges_added': 27, 'events_emitted': 12,
#  'records_upserted': 100}

with gm.IdentityStore(path=".goldenmatch/identity.db") as store:
    view = gm.find_by_record(store, "customers:1")
    print(view.node.entity_id)            # stable UUIDv7
    print([r.record_id for r in view.records])
    print([(e.kind, e.run_name) for e in view.events])

​

Storage model

​

How resolution works

1. Look up existing identities that already own any record in the cluster.
2. Decide what happened:
   • No overlap, mint a new identity (UUIDv7), emit created.
   • One existing identity covers all overlapping records, absorb the new records, emit absorbed_record per addition.
   • Multiple existing identities overlap, merge them. Winner = most members (tie-break: oldest created_at). Emit merged_with on winner, retire losers with status='merged_into', merged_into=<winner>.
3. Upsert every cluster record under the chosen identity.
4. Record evidence, one row in evidence_edges for every scored within-cluster pair, including matchkey name, per-field scores, negative-evidence penalties, and a controller-telemetry snapshot.

​

Surfaces: one shape, many faces

​

Python

from goldenmatch import IdentityStore, find_by_record, get_entity, identity_history, manual_merge

with IdentityStore(path=".goldenmatch/identity.db") as s:
    view = find_by_record(s, "crm:42")
    events = identity_history(s, view.node.entity_id)
    manual_merge(s, keep_entity_id="...", absorb_entity_id="...", reason="dup")

​

CLI

goldenmatch identity list --dataset customers --status active
goldenmatch identity show <entity_id>
goldenmatch identity resolve crm:42
goldenmatch identity history <entity_id>
goldenmatch identity conflicts --dataset customers
goldenmatch identity merge <keep_id> <absorb_id> --reason "dup confirmed"
goldenmatch identity split <entity_id> crm:42 crm:43 --reason "wrong merge"

​

REST

GET    /api/v1/identities                            # list (paginated)
GET    /api/v1/identities/stats                      # totals
GET    /api/v1/identities/{entity_id}                # full view
GET    /api/v1/identities/{entity_id}/360            # Customer 360 serving view
GET    /api/v1/identities/{entity_id}/history        # event log
GET    /api/v1/identities/{entity_id}/evidence       # edges
GET    /api/v1/identities/by-record/{record_id}      # resolve
GET    /api/v1/identities/conflicts                  # conflict edges
POST   /api/v1/identities/{entity_id}/merge          # manual merge
POST   /api/v1/identities/{entity_id}/split          # manual split

​

Web UI

​

MCP

• identity_resolve: look up by record_id
• identity_show: full payload by entity_id
• identity_list: list with filters
• identity_history: event log
• identity_conflicts: list conflicts_with edges
• identity_merge / identity_split: steward operations
• identity_claim: assign a record to a durable identity
• identity_resolve_conflict: resolve a flagged conflicts_with edge
• identity_profile / identity_stats / identity_worklist: inspection + steward triage
• identity_audit / identity_audit_seal / identity_audit_verify: tamper-evident audit log

​

A2A

​

SQL (Postgres + DuckDB)

SELECT goldenmatch_identity_resolve('crm:42', '/path/to/identity.db');
SELECT goldenmatch_identity_view('019e1f...', '/path/to/identity.db');
SELECT goldenmatch_identity_history('019e1f...', '/path/to/identity.db');
SELECT goldenmatch_identity_conflicts('customers', '/path/to/identity.db');
SELECT goldenmatch_identity_list('customers', 'active', '/path/to/identity.db');

​

”Why did these link?”: reading the evidence

from goldenmatch import IdentityStore, get_entity

with IdentityStore(path=".goldenmatch/identity.db") as s:
    view = get_entity(s, entity_id)

for edge in view.edges:
    print(f"{edge.record_a_id} <-> {edge.record_b_id}")
    print(f"  score: {edge.score:.3f}  matchkey: {edge.matchkey_name}")
    print(f"  fields: {edge.field_scores}")
    if edge.negative_evidence:
        print(f"  negative_evidence: {edge.negative_evidence}")
    if edge.controller_snapshot:
        print(f"  autoconfig: {edge.controller_snapshot.get('stop_reason')}")

​

Configuration reference

identity:
  enabled: false           # default off; set true to opt in
  backend: sqlite          # or "postgres"
  path: .goldenmatch/identity.db   # sqlite only
  connection: null         # postgres DSN; required when backend=postgres
  dataset: null            # namespace label that flows into every row
  source_pk_column: null   # column to derive {source}:{source_pk} record_id
                           # when null, derives {source}:h1:{12 hex} from the
                           # row fingerprint (un-fingerprintable rows fall back
                           # to {source}:hash:{12 hex} from the payload SHA-256)
  emit_singletons: true    # whether 1-record clusters become identities

​

Postgres setup

psql -d $DB -f packages/python/goldenmatch/goldenmatch/db/migrations/identity_v1.sql

gm.IdentityStore(backend="postgres", connection="postgres://user:pass@host/db")

​

Performance notes

• Resolve runs after clustering, before output. It is dominated by write throughput, and its cost tracks the identity graph, not the input frame. Roughly the number of identities created plus records upserted. A 1M-row dedupe that yields 50k multi-record identities does far less resolve work than a 200k-row dedupe that yields 200k.
• emit_singletons is the single biggest cost lever, and it defaults to true. With it on, every single-record cluster becomes its own identity, so resolve work scales with row count rather than with the number of real multi-record entities: on a 14M-row dedupe that is 14M identities, not ~500k. Turn it off unless you specifically need a durable id for records that matched nothing:
  identity:
    emit_singletons: false
  
  With emit_singletons: false the resolver also skips preparing rows no multi-record cluster references, so both wall time and resident memory drop with it. Concrete ceiling. On the SQLite backend, emit_singletons: true becomes impractical around ~100k rows. Every row is prepped and written one statement at a time, so resolve tracks the whole input frame. Measured on a single box: 250k rows with singletons on is already slow, while emit_singletons: false resolves a 1M-row dedupe in ~6 minutes because it only touches the multi-record identity graph. Above ~100k rows with singletons on, the resolver logs a one-time advisory pointing at emit_singletons: false and Postgres. (The advisory is SQLite-only and never blocks the run; Postgres’s bulk COPY write path makes the singleton row count a non-issue there.)
• SQLite ceiling. SQLite is the right default through the low millions of rows on a single box. Past that, prefer Postgres: brand-new identities take a bulk COPY fast path there, and it is the only backend the distributed (Ray) resolver supports. The tradeoff is network latency per run, not a throughput cliff.
• The per-partition (distributed) resolver is not reachable from a single-node in-memory run. resolve_identities_distributed needs both Ray and backend: postgres, it materializes cluster aggregates driver-side and resolves them against a pooled Postgres connection (true per-partition map_batches is still on the roadmap; see the Phase 6 notes). A plain single-node run on SQLite therefore has exactly two scale levers: emit_singletons: false, and, once you are past the low millions of rows, moving the store to Postgres. Adding Ray without a Postgres store does not change the single-node path. The SQLite gap is in our write path, not in SQLite: the stdlib sqlite3 driver has no COPY equivalent, so resolve writes go row-at-a-time (batched into transactions) rather than as columnar bulk ingest. Arrow-native bulk ingest into SQLite is available, the Apache Arrow ADBC SQLite driver supports adbc_ingest from Python and can be driven from Rust via adbc_core + adbc_driver_manager, and adopting it would let the SQLite backend share the same staging-table-then-upsert shape the Postgres bulk path uses. That is a planned follow-up, not something GoldenMatch does today.
• Resolution is gated and additive: if the store fails to open, the pipeline logs a warning and continues. Identity never blocks a dedupe.
• For multi-process writers, the SQLite store uses WAL + a 5s busy_timeout. Postgres relies on row-level locks. Single-tenant web UI / CLI invocations are the assumed model; for high-write multi-tenant graphs use Postgres.

​

When NOT to use it

• Single-shot ad-hoc dedupe where you only want golden records out and don’t care about the next run.
• Pipelines whose source has no stable PK and whose rows are duplicated character-for-character, the hash fallback will fold them together.

​

Migration / backfill

​

Migrating legacy record ids

​

Running the migration

# SQLite (default store)
goldenmatch identity migrate-ids --path .goldenmatch/identity.db

# Postgres
goldenmatch identity migrate-ids --dsn postgres://user:pass@host/db

# Preview counts without writing (dry run)
goldenmatch identity migrate-ids --path .goldenmatch/identity.db --dry-run

​

Python API

from goldenmatch.identity import migrate_record_ids

with gm.IdentityStore(path=".goldenmatch/identity.db") as store:
    report = migrate_record_ids(store, dry_run=False)
    print(report)
    # {'source_records': 1840, 'evidence_edges': 4120,
    #  'identity_aliases': 0, 'identity_events': 3680}

​

See also

• examples/python/08_identity_graph.py: end-to-end demo (two-run stability + absorb + merge + split + conflict)
• Pipeline architecture: where identity sits in the dedupe flow
• Learning Memory: the other persistent-state layer; complementary, not competing
• Configuration: full schema reference