> ## Documentation Index > Fetch the complete documentation index at: https://docs.bensevern.dev/docs/llms.txt > Use this file to discover all available pages before exploring further. # Config matrix > The full matrix of GoldenFlow config knobs and its transform vocabulary -- generated so it never drifts. The exhaustive, always-current reference for GoldenFlow configuration: the `GoldenFlowConfig` schema, the full transform-op vocabulary, the domain packs, and every `GOLDENFLOW_*` runtime knob. Built for humans and AI agents. Everything below the line is **generated from code** (`scripts/gen_config_matrix.py`) and verified in CI. The guidance above the line is hand-authored. See [Overview](/docs/goldenflow/overview) and [CLI](/docs/goldenflow/cli) for walkthroughs. ## How GoldenFlow is configured A GoldenFlow config (`GoldenFlowConfig`, loadable from YAML or learned from a sample via `learn_config`) is a declarative transform plan: * **`transforms`** -- the core: a list of `TransformSpec{column, ops}`, where `ops` is an ordered chain of transform-op names drawn from the **Transform ops** vocabulary below (e.g. `["strip", "lowercase", "email_normalize"]`). The op vocabulary is the real config surface; the model shape is thin. * **`splits` / `filters` / `renames` / `drop` / `dedup` / `mappings`** -- column reshaping around the transform chains. * **Domain packs** (`learn_config(domain=...)`) seed sensible op chains for a known domain (see the **Domain packs** vocabulary). `canonicalize` is a first-class op with its own `CanonicalizeKind` set. ## Config object reference Every config object in the pydantic tree(s), generated from the package schema. Nested objects link by name. ### `GoldenFlowConfig` | Field | Type | Default | Choices / notes | | ------------ | -------------------- | ------- | ---------------------------------------------------------------------------------------------------------------- | | `source` | str \| None | `None` | Path or identifier of the input dataset to read records from. | | `output` | str \| None | `None` | Path or identifier where the transformed dataset is written. | | `transforms` | list\[TransformSpec] | `[]` | [`TransformSpec`](#transformspec) -- Per-column transform specs that standardize values by applying ordered ops. | | `splits` | list\[SplitSpec] | `[]` | [`SplitSpec`](#splitspec) -- Split specs that break one column into several target columns. | | `renames` | dict\[str, str] | `{}` | Mapping of existing column names to new names to rename them to. | | `drop` | list\[str] | `[]` | Names of columns to remove from the output. | | `filters` | list\[FilterSpec] | `[]` | [`FilterSpec`](#filterspec) -- Filter specs that keep only rows satisfying each condition. | | `dedup` | DedupSpec \| None | `None` | [`DedupSpec`](#dedupspec) -- Optional deduplication spec that collapses duplicate rows by key columns. | | `mappings` | list\[MappingSpec] | `[]` | [`MappingSpec`](#mappingspec) -- Mapping specs that copy source columns to targets with optional transforms. | ### `TransformSpec` | Field | Type | Default | Choices / notes | | -------- | ---------- | ------------ | --------------------------------------------------------------------- | | `column` | str | **required** | Name of the column the transform operations are applied to. | | `ops` | list\[str] | **required** | Ordered list of transform-op names applied in sequence to the column. | ### `SplitSpec` | Field | Type | Default | Choices / notes | | -------- | ---------- | ------------ | ----------------------------------------------------------------- | | `source` | str | **required** | Name of the column whose values are split into multiple columns. | | `target` | list\[str] | **required** | Names of the output columns produced by the split. | | `method` | str | **required** | Splitting method that determines how the source value is divided. | ### `FilterSpec` | Field | Type | Default | Choices / notes | | ----------- | ---- | ------------ | ------------------------------------------------------------- | | `column` | str | **required** | Name of the column the filter condition is evaluated against. | | `condition` | str | **required** | Predicate expression that rows must satisfy to be kept. | ### `DedupSpec` | Field | Type | Default | Choices / notes | | --------- | ---------- | ------------ | ---------------------------------------------------------------------------------------------------- | | `columns` | list\[str] | **required** | Columns whose combined values define what counts as a duplicate row. | | `keep` | Literal | `'first'` | `first`, `last` -- Which duplicate to retain within each group, either the first or last occurrence. | ### `MappingSpec` | Field | Type | Default | Choices / notes | | ----------- | ------------------------- | ------------ | ------------------------------------------------------------------------------------ | | `source` | str | **required** | Name of the source column to read values from. | | `target` | str \| list\[str] | **required** | Name or names of the target column(s) the source values are written to. | | `transform` | str \| list\[str] \| None | `None` | Optional transform op or ordered list of ops applied while mapping source to target. | ## CLI Every command and its options/arguments, generated from the Typer app. `choice`-typed options list their allowed values. | Command | Option | Type | Default | Notes | | ------------- | ----------------- | ------- | ------------------- | ------------------------------------------------------------- | | `agent-serve` | `--port` | integer | `8150` | Port the A2A agent server listens on. | | `demo` | `--output-dir` | path | `Path('.')` | Directory where the sample data and config files are written. | | `diff` | `before` | path | **required** | Before file | | `diff` | `after` | path | **required** | After file | | `history` | `--limit` | integer | `20` | Number of recent runs to show | | `init` | `data` | path | `None` | Data file to profile | | `init` | `--output` | path | `'goldenflow.yaml'` | Path where the generated config file is written. | | `interactive` | `path` | path | `None` | Input data file | | `learn` | `path` | path | **required** | Input data file | | `learn` | `--output` | path | `'goldenflow.yaml'` | Output config path | | `map` | `--source` | path | **required** | Source data file | | `map` | `--target` | path | **required** | Target data file or schema | | `map` | `--config` | path | `None` | Mapping config | | `map` | `--output` | path | `None` | Save mapping config | | `mcp-serve` | `--transport` | text | `'stdio'` | Transport: stdio or http | | `mcp-serve` | `--host` | text | `'0.0.0.0'` | Host for HTTP transport | | `mcp-serve` | `--port` | integer | `8150` | Port for HTTP transport | | `profile` | `path` | path | **required** | Input data file | | `schedule` | `path` | path | **required** | Data file to transform | | `schedule` | `--every` | text | `'1h'` | Interval (e.g., 5m, 1h, 30s) | | `schedule` | `--config` | path | `None` | YAML config applied on each scheduled run. | | `schedule` | `--output-dir` | path | `None` | Directory where transformed files are written. | | `serve` | `--host` | text | `'0.0.0.0'` | Network interface the REST API server binds to. | | `serve` | `--port` | integer | `8000` | Port the REST API server listens on. | | `stream` | `path` | path | **required** | Input data file | | `stream` | `--chunk-size` | integer | `10000` | Rows per batch | | `stream` | `--config` | path | `None` | YAML config applied to each streamed batch. | | `stream` | `--output-dir` | path | `None` | Directory where the combined transformed file is written. | | `transform` | `path` | path | **required** | Input data file | | `transform` | `--config` | path | `None` | YAML config file | | `transform` | `--output-dir` | path | `None` | Output directory | | `transform` | `--domain` | text | `None` | Domain pack to use | | `transform` | `--from-findings` | boolean | `False` | Read findings from stdin | | `transform` | `--llm` | boolean | `False` | Enable LLM-enhanced transforms | | `transform` | `--strict` | boolean | `False` | Fail if any transform errors occur | | `validate` | `path` | path | **required** | Input data file | | `validate` | `--config` | path | `None` | YAML config file describing which transforms to apply. | | `watch` | `path` | path | `'.'` | Directory to watch | | `watch` | `--config` | path | `None` | YAML config applied to each new or changed file. | | `watch` | `--output-dir` | path | `None` | Directory where transformed files are written. | | `watch` | `--interval` | float | `2.0` | Poll interval in seconds | ## MCP tools 10 MCP tool(s) exposed by `goldenflow.mcp.server` -- the programmatic / agent surface. Config-bearing tools take the same knobs as above. | Tool | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `diff` | Compare two data files and show what changed (added, removed, modified rows). | | `explain_transform` | Describe what a specific transform does, its mode, and input types. | | `learn` | Generate a YAML config from data patterns. | | `list_domains` | List available domain packs (e.g., people\_hr, ecommerce, finance). | | `list_transforms` | List all registered transforms with their modes, input types, and auto-apply status. | | `map` | Auto-map schemas between source and target files. | | `profile` | Profile a data file showing column types, nulls, and patterns. | | `select_from_findings` | Map GoldenCheck findings to recommended GoldenFlow transforms. Bridge tool for Check-to-Flow handoff. | | `transform` | Clean / normalize a data file with GoldenFlow transforms (phone, date, unicode, casing, categorical, and more) and write the transformed file. Zero-config auto-detects and applies transforms; pass a config to control them. See list\_transforms for what's available. Does not deduplicate or match -- it only reshapes column values. | | `validate` | Dry-run transform on a file. Shows what would change without writing output. | ## Enumerated vocabularies Allowed values for the `str`-typed / registry-backed fields above. ### Transform ops *`list_transforms` -- `TransformSpec.ops`.* | Value | Meaning | | -------------------------- | ----------------------------------------------------------------------------------------- | | `aba_validate` | Validate a US ABA bank routing number: exactly 9 digits plus the | | `abs_value` | Return the absolute value. | | `account_mask` | Mask account numbers showing only last 4 digits. | | `address_expand` | Replace street abbreviations (St, Ave...) with full forms. | | `address_standardize` | Replace full street suffixes (Street, Avenue...) with abbreviations. | | `age_from_dob` | Compute age in years from a date of birth. | | `boolean_normalize` | Parse loose boolean-ish strings (yes/no/y/n/1/0/true/false/t/f). | | `carceral_abbreviate` | Expand carceral facility-type abbreviations and state-complex aliases. | | `carceral_name_normalize` | Full carceral name pipeline: org-strip + uppercase + punctuation strip | | `carceral_org_strip` | Strip leading operator-org prefix from a carceral facility name. | | `category_auto_correct` | Auto-correct categorical misspellings and case variants. | | `category_from_file` | Load mapping from a CSV/YAML file and standardize values. | | `category_llm_correct` | LLM-enhanced categorical correction. | | `category_standardize` | Map variant values to canonical values. mapping: \{canonical: \[variant1, variant2, ...]} | | `cc_brand` | Detect the card brand (visa/mastercard/amex/...) or null. Native-first. | | `cc_format` | Group a valid payment-card number (Amex 4-6-5, else 4-4-4-4...); | | `cc_mask` | Mask a payment-card number to stars + last 4 digits. | | `cc_validate` | Validate a payment-card number via the Luhn checksum. | | `clamp` | Clip values into \[min\_val, max\_val]. | | `collapse_whitespace` | Collapse runs of whitespace to single spaces. | | `comma_decimal` | Convert European decimal format (1.234,56) to float (1234.56). | | `company_extract_legal` | Extract the canonical legal-form token (`inc`/`llc`/...) or null. | | `company_normalize` | Composite company dedup key: lowercase, drop leading 'the', strip legal | | `company_strip_legal` | Strip trailing legal-form suffixes, preserving the core name's case. | | `country_standardize` | Normalize country names to ISO 3166-1 alpha-2 codes. | | `currency_strip` | Strip currency symbols and thousand separators, return numeric. | | `cusip_format` | Standardize CUSIP identifiers (9 chars, uppercase). | | `cusip_validate` | Validate a CUSIP. Native-first over goldenflow-core. | | `date_eu` | Parse a European (DD/MM/YYYY) date. | | `date_iso8601` | Parse/format a date as ISO-8601. | | `date_parse` | Auto-detect format and normalize to ISO 8601. | | `date_shift` | Shift dates by a number of days (positive = forward, negative = backward). | | `date_us` | Parse a US (MM/DD/YYYY) date. | | `date_validate` | Validate if value is a parseable date. Returns True/False/None. | | `datetime_iso8601` | Parse to ISO 8601 datetime (with time component). | | `double_metaphone_alt` | Alternate Double Metaphone code. Native-first. | | `double_metaphone_primary` | Primary Double Metaphone code (blocking key). Native-first. | | `ean_validate` | Validate an EAN-8, UPC-A, or EAN-13 via its GTIN mod-10 checksum. | | `ein_format` | Normalize EIN to XX-XXXXXXX format. Native-first. | | `email_canonical` | Full dedup key: email\_normalize + alias googlemail.com -> gmail.com so | | `email_extract_domain` | Extract the lowercased domain from an email address. | | `email_lowercase` | Lowercase the entire email address (trim + lowercase). | | `email_mask` | PII mask: keep the first local char, star the rest, keep @domain | | `email_normalize` | Normalize email: lowercase, strip +tags, strip dots from Gmail local part. | | `email_validate` | Validate email format. Returns True/False/None. | | `extract_day` | Extract the day of month as an integer (1-31). | | `extract_day_of_week` | Extract the day of week name (Monday, Tuesday, etc.). | | `extract_month` | Extract the month as an integer (1-12). | | `extract_numbers` | Extract all numbers from text, joined by spaces. | | `extract_quarter` | Extract the quarter (1-4) from a date. | | `extract_year` | Extract the year as an integer. | | `fill_zero` | Replace null values with 0. | | `fix_mojibake` | Fix common UTF-8/Latin-1 mojibake by re-encoding. | | `fraction_to_decimal` | Parse a fraction or mixed number (1/2, 3 3/4) to a float. Native-first. | | `gender_standardize` | Standardize gender strings to `M`/`F`; anything else passes | | `iban_format` | Group a valid IBAN into 4-char blocks; `null` for invalid input. | | `iban_validate` | Validate an IBAN via structural checks + the ISO 7064 mod-97 check. | | `icd10_format` | Standardize ICD-10 codes (uppercase, insert dot after 3rd char). | | `imei_validate` | Validate an IMEI: exactly 15 digits plus the Luhn checksum. | | `initial_expand` | Returns (series, flagged\_rows). Values with initials are unchanged but flagged. | | `isbn_normalize` | Canonicalize a valid ISBN-10/13 to its 13-digit form; `null` for | | `isbn_validate` | Validate an ISBN-10 or ISBN-13 via its checksum. | | `isin_validate` | Validate an ISIN (ISO 6166). Native-first over goldenflow-core. | | `latlng_pack` | Pack `lat` + `lng` into a single `latlng` column shaped | | `lowercase` | Lowercase the value. | | `luhn_validate` | Generic Luhn check-digit validation. Native-first over goldenflow-core. | | `merge_name` | Merge first\_name and last\_name columns into a full\_name column. | | `mls_normalize` | Normalize MLS listing IDs (uppercase, strip whitespace). | | `name_initials` | Initials of each whitespace token (letter-leading tokens only). | | `name_proper` | Proper-case a name (title-case + Mc/O' fixups). | | `name_script` | Detect the dominant Unicode script in a name: `Unknown` for empty | | `name_transliterate` | ASCII-fold a name via an explicit curated diacritic map. Non-ASCII | | `nickname_standardize` | Map common nicknames to formal first names. | | `normalize_line_endings` | Normalize \r\n and \r to \n. | | `normalize_quotes` | Replace smart/curly quotes with straight quotes. | | `normalize_unicode` | Normalize to Unicode NFC form. | | `npi_validate` | Validate a US NPI. Native-first over goldenflow-core. | | `null_standardize` | Map null-sentinel strings (n/a, null, none, na, nil, nan, -, empty) to | | `ordinal_to_int` | Parse an English ordinal (1st/2nd/3rd) to its integer. Native-first. | | `pad_left` | Left-pad strings to a fixed width. | | `pad_right` | Right-pad strings to a fixed width. | | `percentage_normalize` | Strip trailing %, parse to float, divide by 100. | | `phone_country_code` | Extract the country calling code as an integer. | | `phone_digits` | Keep only the phone number's digits. | | `phone_e164` | Format a phone number as E.164. | | `phone_national` | Format a phone number in national format. | | `phone_validate` | Flag whether the phone number is valid. | | `remove_digits` | Remove all digit characters from text. | | `remove_emojis` | Remove emoji characters from text. | | `remove_html_tags` | Strip HTML tags from text. | | `remove_punctuation` | Strip punctuation characters. | | `remove_urls` | Strip URLs (http/https) from text. | | `roman_to_int` | Parse a Roman numeral to its integer (canonical forms only, 1..=3999). | | `round` | Round to n decimal places (round-half-away-from-zero, see the | | `scientific_to_decimal` | Convert scientific notation (1.5e3) to decimal (1500.0). | | `sku_normalize` | Normalize SKU identifiers (uppercase, strip whitespace, remove special chars). | | `soundex` | Soundex phonetic key. Native-first over goldenflow-core. | | `split_address` | Parse 'street, city, state zip' into separate columns. | | `split_name` | Split 'First Last' into first\_name and last\_name columns. | | `split_name_reverse` | Split 'Last, First' into first\_name and last\_name columns. | | `ssn_format` | Normalize SSN to XXX-XX-XXXX format. Native-first over goldenflow-core. | | `ssn_mask` | Mask SSN to \***-**-XXXX (last 4 visible). Native-first. | | `ssn_validate` | Flag whether the value is a valid US Social Security Number. | | `state_abbreviate` | Normalize state name to a 2-letter abbreviation (unmatched -> original). | | `state_expand` | Expand a 2-letter state abbreviation to its full name (unmatched -> original). | | `strip` | Trim leading/trailing whitespace. | | `strip_middle` | Keep only the first and last whitespace tokens (drop the middle). | | `strip_suffixes` | Strip trailing professional suffixes (Jr/Sr/II/MD/PhD/etc.) from names. | | `strip_titles` | Strip leading personal titles (Mr/Mrs/Ms/Dr/Prof/etc.) from names. | | `swift_format` | Normalize a valid SWIFT/BIC code to uppercase (spaces stripped); | | `swift_validate` | Validate a SWIFT/BIC code via structural checks (length 8/11 + | | `title_case` | Title-case the value. | | `to_integer` | Parse string to integer, truncating any decimal part. | | `truncate` | Truncate string to the first n characters. | | `unit_normalize` | Normalize unit/apartment/suite designations. | | `uppercase` | Uppercase the value. | | `url_canonical` | Composite dedup key: ensure scheme, lowercase scheme+host, strip [www](http://www)., | | `url_extract_domain` | Extract domain from a URL. | | `url_normalize` | Normalize URLs: ensure scheme, lowercase domain, strip trailing slash. | | `url_strip_tracking` | Remove tracking query params (utm\_\*, gclid, fbclid, ...), preserving the | | `url_strip_www` | Strip a leading `www.` label from the host, preserving scheme, path, | | `vat_format` | Normalize a valid EU VAT number to its compact uppercase form (prefix | | `vat_validate` | Validate an EU VAT number: structural check (country prefix + length + | | `zip_normalize` | Normalize a US ZIP to 5-digit form (strip +4, zero-pad all-digit, preserve | ### Domain packs *`_DOMAINS` -- `learn_config(domain=...)`.* | Value | Meaning | | ------------- | --------------------------------- | | `carceral` | Corrections / justice field pack. | | `ecommerce` | E-commerce / retail field pack. | | `finance` | Financial-services field pack. | | `healthcare` | Healthcare field pack. | | `people_hr` | People / HR field pack. | | `real_estate` | Real-estate field pack. | ### Canonicalize kinds *`CanonicalizeKind` -- `canonicalize` op.* | Value | Meaning | | -------- | ------------------------------ | | `email` | Canonical email form. | | `name` | Canonical person-name form. | | `phone` | Canonical phone form. | | `postal` | Canonical postal-address form. | ## Environment variables (index) 5 `GOLDENFLOW_*` runtime knob(s) read by the package, scanned from source so this is complete. Grouped by area: * **ENGINE** (1): `GOLDENFLOW_ENGINE` * **FUSED** (1): `GOLDENFLOW_FUSED_APPLY` * **LLM** (1): `GOLDENFLOW_LLM` * **NATIVE** (2): `GOLDENFLOW_NATIVE`, `GOLDENFLOW_NATIVE_CSV_PARALLEL_MIN_BYTES` ## See also * [Overview](/docs/goldenflow/overview), [CLI](/docs/goldenflow/cli), [Performance](/docs/goldenflow/performance).