> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bensevern.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Performance
> How GoldenFlow stays fast: vectorized Polars fast paths with a per-row fallback, and an optional Arrow-native Rust kernel for phone normalization.
GoldenFlow is Polars-native, so most transforms already run in vectorized Rust.
The two transforms that historically dominated a run — date and phone
normalization — used to call a Python library (`dateutil` / `phonenumbers`) once
per row. They are now resolved with a vectorized fast path and a per-row
fallback, with an optional compiled kernel for the phone tail.
## The three-tier resolver
Date and phone transforms resolve each value with the cheapest tier that can,
falling through only for the rows it can't:
1. **Vectorized Polars fast path** — resolves the well-formed common case in
Rust (multi-format `str.to_date` for dates; a NANP-shape regex for phones),
leaving anything it isn't certain about unresolved.
2. **Native kernel** *(optional, phone only)* — the `goldenflow-native` Rust
kernel runs on just the residual rows.
3. **Per-row reference** — the original `dateutil` / `phonenumbers` path settles
whatever the first two tiers left.
On clean data the residual is empty, so tiers 2–3 never run. The output is
**byte-identical** to applying the per-row reference to every row — each tier
only claims rows it resolves exactly the same way.
Measured on a realistic messy 1M-row frame: `date_iso8601` **76×** faster,
`phone_e164` **19×**, `phone_digits` **4.9×** — roughly **14× end-to-end** on
a mixed date/phone/text run, with no change to the cleaned values.
## Optional native kernel
The kernels live in `goldenflow-core` (a pyo3-free Rust crate); `goldenflow-native`
is a thin PyO3 shim over it, shipped as a separate compiled runtime (abi3) — the
same split as `polars` / `polars-runtime`. The pure-Python `goldenflow` wheel
works on its own; the native kernel is opt-in and accelerates:
* the **phone** residual the Polars fast path can't reach (alpha numbers like
`1-800-FLOWERS`, extensions, `+1`-prefixed forms), and
* the **checksummed / structural identifiers** — payment card (Luhn), IBAN
(mod-97), ISBN, EAN/UPC, EU VAT, SWIFT/BIC, US ABA routing, and IMEI — via owned
Rust kernels with pure-Python fallbacks proven byte-identical over a committed
corpus. In `auto` mode the native kernel also backs the email, URL, numeric,
categorical, name (incl. `name_transliterate` / `name_script`), and US-address
families — anywhere a component kernel exists.
```bash pip theme={null}
pip install "goldenflow[native]"
```
It is **parity-safe and on by default once installed**. Rust is the reference
implementation and pure-Python is the lossy fallback: the native path runs
wherever a component's kernel is proven output-faithful. The phone kernel is
gated to North American (NANP) numbers, deferring everything international or
ambiguous to Python; the identifier kernels are region-free. You never get a
different cleaned value with the kernel on.
### Controlling it
The `GOLDENFLOW_NATIVE` environment variable selects the path (reference-mode,
mirroring `GOLDENMATCH_NATIVE`):
| Value | Behavior |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| unset / `auto` | Use the native kernel wherever a component's kernel symbol exists (identifiers, email, URL, numeric, categorical, name, and US-address families), except known-divergent components (e.g. `phone_validate` stays pure-Python). **Default.** |
| `0` | Force the pure-Python path everywhere. |
| `1` | Require native; raise if the kernel isn't importable. Runs every component with no NANP restriction — a benchmarking/parity lane that **can** differ from Python on international phone numbers. |
Dates are intentionally **not** a native kernel — the Polars fast path already
resolves them in vectorized Rust, so a per-row compiled parser would be
slower. `phone_national` stays pure Python, and `phone_validate` stays
pure-Python by design (its only native symbol implements `is_valid`, not the
product-chosen `is_possible` spec). `phone_digits` is pure Polars.
### Fused columnar apply
When a column has a run of two or more consecutive owned transforms of one
dtype (e.g. `strip → lowercase → collapse_whitespace → remove_punctuation`,
`name_transliterate → name_proper → strip_titles`, or the numeric
`round → clamp → abs_value`), the native path **fuses** them into a single pass
over the Arrow buffer instead of crossing the Python/Polars/Arrow boundary once
per transform. The output and the audit trail are **byte-identical** to applying
the transforms one at a time; the win is lower memory (fusion avoids
materializing one intermediate column per transform) plus a smaller wall-time
drop.
Measured at 5M rows: **\~20% lower peak RSS** and a modest wall speedup, growing
with row count. It is **on by default** when the native kernel is available
(needs `goldenflow-native >= 0.14.0` for the nullable URL/company/email ops,
`>= 0.13.0` for numeric + parameterized, `>= 0.12.0` for the no-arg string
families); older wheels fall back gracefully.
| `GOLDENFLOW_FUSED_APPLY` | Behavior |
| ------------------------ | ------------------------------------------------------------------- |
| unset | Fuse when the native fused kernel is present. **Default.** |
| `0` | Force the per-transform path (each transform applied individually). |
The fused set covers three kernel shapes: the owned, total (never-null)
**string** transforms (text + email + name-normalizer families, plus the
parameterized `truncate`/`pad`); the owned **numeric** f64 transforms
(`round`/`clamp`/`abs_value`/`fill_zero`) on a `Float64` column; and the
`Option`-returning **URL / company / email** transforms (`url_normalize`,
`company_normalize`, `email_mask`, …), where a value a kernel can't parse
becomes a null cell that passes through the rest of the run. The engine
recomputes the fusable set as a run advances, so a parser that changes a
column's dtype mid-chain (e.g. `currency_strip`: str→f64) lets the string head
and the numeric tail each fuse; a nullable run may also mix in the total string
ops. Only `*_validate` (boolean), multi-column split/merge, and residual-tier
(`phone`/date) transforms stay on the per-transform path — automatically, no
configuration needed.
## Cross-surface (WASM / TypeScript)
The identifier kernels also compile to WebAssembly (`goldenflow-wasm`). The
TypeScript package keeps pure-TS as the default and permanent fallback; an
opt-in `enableWasm()` (async, stays pure-TS on failure) routes the identifier
transforms through the WASM kernel — the TS analog of `goldenflow[native]`. One
shared oracle corpus is asserted byte-identical across native, WASM-TS, and
pure-Python.
## Why it's safe
The parity contract is enforced by tests that compare the full output against
the pure references (`dateutil` / `phonenumbers` for date/phone; the
`goldenflow-core` Rust oracle for identifiers) over large corpora, and CI lanes
build the native kernel and the WASM artifact and run those suites with each
active. Turning any kernel on or off only changes speed, never the cleaned data.