Skip to main content
GoldenFlow is Polars-native, so most transforms already run in vectorized Rust. The two transforms that historically dominated a run — date and phone normalization — used to call a Python library (dateutil / phonenumbers) once per row. They are now resolved with a vectorized fast path and a per-row fallback, with an optional compiled kernel for the phone tail.

​
The three-tier resolver

Date and phone transforms resolve each value with the cheapest tier that can, falling through only for the rows it can’t:
  1. Vectorized Polars fast path — resolves the well-formed common case in Rust (multi-format str.to_date for dates; a NANP-shape regex for phones), leaving anything it isn’t certain about unresolved.
  2. Native kernel (optional, phone only) — the goldenflow-native Rust kernel runs on just the residual rows.
  3. Per-row reference — the original dateutil / phonenumbers path settles whatever the first two tiers left.
On clean data the residual is empty, so tiers 2–3 never run. The output is byte-identical to applying the per-row reference to every row — each tier only claims rows it resolves exactly the same way.
Measured on a realistic messy 1M-row frame: date_iso8601 76× faster, phone_e164 19×, phone_digits 4.9× — roughly 14× end-to-end on a mixed date/phone/text run, with no change to the cleaned values.

​
Optional native kernel

The kernels live in goldenflow-core (a pyo3-free Rust crate); goldenflow-native is a thin PyO3 shim over it, shipped as a separate compiled runtime (abi3) — the same split as polars / polars-runtime. The pure-Python goldenflow wheel works on its own; the native kernel is opt-in and accelerates:
  • the phone residual the Polars fast path can’t reach (alpha numbers like 1-800-FLOWERS, extensions, +1-prefixed forms), and
  • the checksummed / structural identifiers — payment card (Luhn), IBAN (mod-97), ISBN, EAN/UPC, EU VAT, SWIFT/BIC, US ABA routing, and IMEI — via owned Rust kernels with pure-Python fallbacks proven byte-identical over a committed corpus. In auto mode the native kernel also backs the email, URL, numeric, categorical, name (incl. name_transliterate / name_script), and US-address families — anywhere a component kernel exists.
It is parity-safe and on by default once installed. Rust is the reference implementation and pure-Python is the lossy fallback: the native path runs wherever a component’s kernel is proven output-faithful. The phone kernel is gated to North American (NANP) numbers, deferring everything international or ambiguous to Python; the identifier kernels are region-free. You never get a different cleaned value with the kernel on.

​
Controlling it

The GOLDENFLOW_NATIVE environment variable selects the path (reference-mode, mirroring GOLDENMATCH_NATIVE):
Dates are intentionally not a native kernel — the Polars fast path already resolves them in vectorized Rust, so a per-row compiled parser would be slower. phone_national stays pure Python, and phone_validate stays pure-Python by design (its only native symbol implements is_valid, not the product-chosen is_possible spec). phone_digits is pure Polars.

​
Fused columnar apply

When a column has a run of two or more consecutive owned transforms of one dtype (e.g. strip → lowercase → collapse_whitespace → remove_punctuation, name_transliterate → name_proper → strip_titles, or the numeric round → clamp → abs_value), the native path fuses them into a single pass over the Arrow buffer instead of crossing the Python/Polars/Arrow boundary once per transform. The output and the audit trail are byte-identical to applying the transforms one at a time; the win is lower memory (fusion avoids materializing one intermediate column per transform) plus a smaller wall-time drop. Measured at 5M rows: ~20% lower peak RSS and a modest wall speedup, growing with row count. It is on by default when the native kernel is available (needs goldenflow-native >= 0.14.0 for the nullable URL/company/email ops, >= 0.13.0 for numeric + parameterized, >= 0.12.0 for the no-arg string families); older wheels fall back gracefully.
The fused set covers three kernel shapes: the owned, total (never-null) string transforms (text + email + name-normalizer families, plus the parameterized truncate/pad); the owned numeric f64 transforms (round/clamp/abs_value/fill_zero) on a Float64 column; and the Option-returning URL / company / email transforms (url_normalize, company_normalize, email_mask, …), where a value a kernel can’t parse becomes a null cell that passes through the rest of the run. The engine recomputes the fusable set as a run advances, so a parser that changes a column’s dtype mid-chain (e.g. currency_strip: str→f64) lets the string head and the numeric tail each fuse; a nullable run may also mix in the total string ops. Only *_validate (boolean), multi-column split/merge, and residual-tier (phone/date) transforms stay on the per-transform path — automatically, no configuration needed.

​
Cross-surface (WASM / TypeScript)

The identifier kernels also compile to WebAssembly (goldenflow-wasm). The TypeScript package keeps pure-TS as the default and permanent fallback; an opt-in enableWasm() (async, stays pure-TS on failure) routes the identifier transforms through the WASM kernel — the TS analog of goldenflow[native]. One shared oracle corpus is asserted byte-identical across native, WASM-TS, and pure-Python.

​
Why it’s safe

The parity contract is enforced by tests that compare the full output against the pure references (dateutil / phonenumbers for date/phone; the goldenflow-core Rust oracle for identifiers) over large corpora, and CI lanes build the native kernel and the WASM artifact and run those suites with each active. Turning any kernel on or off only changes speed, never the cleaned data.