JSON Tools RS

A high-performance Rust library for advanced JSON manipulation with SIMD-accelerated parsing, Rayon-based parallelism, and native Python bindings with DataFrame/Series support.

Why JSON Tools RS?

JSON Tools RS is designed for developers who need to:

• Transform nested JSON into flat structures for databases, CSV exports, or analytics
• Clean and normalize JSON data from external APIs or user input
• Process large batches of JSON documents efficiently
• Maintain type safety with perfect roundtrip support (flatten -> unflatten -> original)
• Work with both Rust and Python using the same consistent API

Key Features

• Unified API -- Single JSONTools entry point for flattening, unflattening, or pass-through transforms
• Builder Pattern -- Fluent, chainable API for configuration
• High Performance -- SIMD-accelerated parsing, FxHashMap, SmallVec stack allocation, tiered caching (~2,000+ ops/ms)
• Parallel Processing -- Rayon's persistent work-stealing thread pool for 3-5x speedup on batch operations
• Complete Roundtrip -- Flatten and unflatten with perfect fidelity
• Comprehensive Filtering -- Remove empty strings, nulls, empty objects, empty arrays
• Advanced Replacements -- Literal (default) or regex (via r'...') key/value replacements
• Collision Handling -- Collect colliding values into arrays
• Automatic Type Conversion -- Strings to numbers, booleans, dates, and nulls
• Date Normalization -- ISO-8601 detection and UTC normalization
• Batch Processing -- Single or batch JSON, dicts, lists, DataFrames, and Series
• Python Bindings -- Full Python support with perfect type preservation
• DataFrame/Series Support -- Pandas, Polars, PyArrow, and PySpark
• Modular Architecture -- 10 focused modules for maintainability with zero-overhead abstraction

Quick Example

Rust:

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonOutput};

let result = JSONTools::new()
    .flatten()
    .execute(r#"{"user": {"name": "John", "age": 30}}"#)?;
// {"user.name":"John","user.age":30}
}

Python:

import json_tools_rs as jt

result = jt.JSONTools().flatten().execute({"user": {"name": "John", "age": 30}})
# {'user.name': 'John', 'user.age': 30}

Installation

Rust

Add to your Cargo.toml:

cargo add json-tools-rs

Or manually:

[dependencies]
json-tools-rs = "0.9"

Python

Install from PyPI:

pip install json-tools-rs

This pulls in orjson as a regular dependency, which the bindings use automatically for the Python-side dict ↔ JSON-string conversion that dict, list[dict], and DataFrame inputs go through (roughly 1.4-1.6x faster end-to-end calls for those input shapes than the standard library's json module). Documents containing integers beyond 64-bit range are always routed through the standard library instead, to preserve exact integer precision.

Pre-built wheels are available for:

Python 3.9+ is supported.

Verify Installation

Rust:

use json_tools_rs::JSONTools;

fn main() {
    let result = JSONTools::new()
        .flatten()
        .execute(r#"{"hello": "world"}"#)
        .unwrap();
    println!("{:?}", result);
}

Python:

import json_tools_rs as jt

result = jt.JSONTools().flatten().execute({"hello": "world"})
print(result)  # {'hello': 'world'}

Quick Start (Rust)

The JSONTools struct provides a unified builder pattern API. Call .flatten() or .unflatten() to set the mode, chain configuration methods, then call .execute().

Basic Flattening

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonOutput};

let json = r#"{"user": {"name": "John", "profile": {"age": 30, "city": "NYC"}}}"#;
let result = JSONTools::new()
    .flatten()
    .execute(json)?;

if let JsonOutput::Single(flattened) = result {
    println!("{}", flattened);
}
// {"user.name":"John","user.profile.age":30,"user.profile.city":"NYC"}
}

Basic Unflattening

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonOutput};

let json = r#"{"user.name": "John", "user.profile.age": 30}"#;
let result = JSONTools::new()
    .unflatten()
    .execute(json)?;

if let JsonOutput::Single(nested) = result {
    println!("{}", nested);
}
// {"user":{"name":"John","profile":{"age":30}}}
}

Advanced Configuration

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonOutput};

let json = r#"{"user": {"name": "John", "details": {"age": null, "city": ""}}}"#;
let result = JSONTools::new()
    .flatten()
    .separator("::")
    .lowercase_keys(true)
    .remove_empty_strings(true)
    .remove_nulls(true)
    .execute(json)?;

if let JsonOutput::Single(flattened) = result {
    println!("{}", flattened);
}
// {"user::name":"John"}
}

Batch Processing

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonOutput};

let batch = vec![
    r#"{"user": {"name": "Alice"}}"#,
    r#"{"user": {"name": "Bob"}}"#,
    r#"{"user": {"name": "Charlie"}}"#,
];

let result = JSONTools::new()
    .flatten()
    .separator("_")
    .execute(batch.as_slice())?;

if let JsonOutput::Multiple(results) = result {
    for r in &results {
        println!("{}", r);
    }
}
// {"user_name":"Alice"}
// {"user_name":"Bob"}
// {"user_name":"Charlie"}
}

Error Handling

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonToolsError};

match JSONTools::new().flatten().execute("invalid json") {
    Ok(result) => println!("{:?}", result),
    Err(e) => {
        eprintln!("Error [{}]: {}", e.error_code(), e);
        // Error [E001]: [E001] JSON parsing failed: Invalid JSON value at line 1 column 1 ...
    }
}
}

Quick Start (Python)

The Python bindings provide the same JSONTools API with perfect type matching: input type equals output type.

Type Preservation

Basic Flattening

import json_tools_rs as jt

# Dict input -> dict output
result = jt.JSONTools().flatten().execute({"user": {"name": "John", "age": 30}})
print(result)  # {'user.name': 'John', 'user.age': 30}

# String input -> string output
result = jt.JSONTools().flatten().execute('{"user": {"name": "John"}}')
print(result)  # '{"user.name":"John"}'

Basic Unflattening

import json_tools_rs as jt

result = jt.JSONTools().unflatten().execute({"user.name": "John", "user.age": 30})
print(result)  # {'user': {'name': 'John', 'age': 30}}

Advanced Configuration

import json_tools_rs as jt

tools = (jt.JSONTools()
    .flatten()
    .separator("::")
    .lowercase_keys(True)
    .remove_empty_strings(True)
    .remove_nulls(True)
    .key_replacement("r'^user_'", "")
    .auto_convert_types(True)
)

data = {"User_Name": "Alice", "User_Age": "30", "User_Status": None}
result = tools.execute(data)
print(result)  # {'name': 'Alice', 'age': 30}

Batch Processing

import json_tools_rs as jt

tools = jt.JSONTools().flatten()

# List of dicts -> list of dicts
results = tools.execute([
    {"user": {"name": "Alice"}},
    {"user": {"name": "Bob"}},
])
print(results)  # [{'user.name': 'Alice'}, {'user.name': 'Bob'}]

# List of strings -> list of strings
results = tools.execute(['{"a": {"b": 1}}', '{"c": {"d": 2}}'])
print(results)  # ['{"a.b":1}', '{"c.d":2}']

DataFrame Support

import json_tools_rs as jt
import pandas as pd

df = pd.DataFrame([
    {"user": {"name": "Alice", "age": 30}},
    {"user": {"name": "Bob", "age": 25}},
])

result = jt.JSONTools().flatten().execute(df)
print(type(result))  # <class 'pandas.core.frame.DataFrame'>
# Also works with Polars, PyArrow Tables, and PySpark DataFrames

Normalise: Always Get a DataFrame

Sometimes you don't have a DataFrame to start with -- just a dict, a string, or a list of records -- but you still want a wide table back. normalise=True does that regardless of input shape, with target picking the library (or auto-resolving if omitted):

import json_tools_rs as jt

tools = jt.JSONTools().flatten()

# A bare dict -> a 1-row DataFrame, no DataFrame input required
df = tools.execute({"user": {"name": "Alice", "age": 30}}, normalise=True)
print(df)
#   user.name  user.age
# 0     Alice        30

# A list of differently-shaped records -> unioned, null-filled columns
data = [{"a": 1, "b": {"x": "hi"}}, {"a": 2, "c": True}]
df = tools.execute(data, normalise=True, target="polars")

See DataFrame & Series Support for the full picture, including composing with the rest of the builder pipeline and genuine PySpark DataFrame output via target="pyspark".

Error Handling

import json_tools_rs as jt

try:
    result = jt.JSONTools().flatten().execute("invalid json")
except jt.JsonToolsError as e:
    print(f"Error: {e}")

Flattening & Unflattening

Flattening

Flattening converts nested JSON into a flat key-value structure using dot-separated (or custom) keys.

// Input
{"user": {"name": "John", "address": {"city": "NYC", "zip": "10001"}}}

// Output (flattened)
{"user.name": "John", "user.address.city": "NYC", "user.address.zip": "10001"}

Arrays

Arrays are flattened with numeric indices:

// Input
{"users": [{"name": "Alice"}, {"name": "Bob"}]}

// Output
{"users.0.name": "Alice", "users.1.name": "Bob"}

Custom Separators

Use .separator() to change the key delimiter:

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .flatten()
    .separator("::")
    .execute(json)?;
// {"user::name": "John", "user::address::city": "NYC"}
}

result = jt.JSONTools().flatten().separator("::").execute(data)

Unflattening

Unflattening reverses the process, reconstructing nested structures from flat keys.

// Input
{"user.name": "John", "user.address.city": "NYC"}

// Output (unflattened)
{"user": {"name": "John", "address": {"city": "NYC"}}}

Numeric keys reconstruct arrays:

// Input
{"users.0.name": "Alice", "users.1.name": "Bob"}

// Output
{"users": [{"name": "Alice"}, {"name": "Bob"}]}

Roundtrip

Flattening and unflattening are perfect inverses. You can flatten data, apply transformations, then unflatten to recover the original structure:

#![allow(unused)]
fn main() {
let original = r#"{"user": {"name": "John", "scores": [10, 20, 30]}}"#;

// Flatten
let flat = JSONTools::new().flatten().execute(original)?;

// Unflatten back
let restored = JSONTools::new().unflatten().execute(
    &flat.try_into_single()?
)?;
// Matches original structure
}

All configuration options (filtering, replacements, collision handling, type conversion) work with both .flatten() and .unflatten() modes.

Examples

Easy: flatten a nested object

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .flatten()
    .execute(r#"{"user": {"name": "Alice", "age": 30}}"#)?;
// {"user.name": "Alice", "user.age": 30}
}

result = jt.JSONTools().flatten().execute({"user": {"name": "Alice", "age": 30}})
# {'user.name': 'Alice', 'user.age': 30}

Medium: arrays of objects, custom separator, round-trip

import json_tools_rs as jt

data = {"users": [{"name": "Alice", "roles": ["admin", "editor"]}, {"name": "Bob", "roles": []}]}

flat = jt.JSONTools().flatten().separator("::").execute(data)
# {'users::0::name': 'Alice', 'users::0::roles::0': 'admin', 'users::0::roles::1': 'editor',
#  'users::1::name': 'Bob', 'users::1::roles': []}
# Note: Bob's empty "roles" array is kept as a literal [] value under its own key --
# only a *non-empty* container gets recursively flattened into per-element keys.

restored = jt.JSONTools().unflatten().separator("::").execute(flat)
# {'users': [{'name': 'Alice', 'roles': ['admin', 'editor']}, {'name': 'Bob', 'roles': []}]}
# Exact round trip, including Bob's empty array.

Hard: flatten -> transform -> unflatten pipeline

Filtering, key/value transforms, and type conversion all run during .flatten(); .unflatten() only reconstructs structure from whatever flat keys survive, so a single flatten call can prepare data that a later unflatten call reconstructs with fewer keys and already-converted types:

import json_tools_rs as jt

data = {
    "Order_ID": "ORD-1001",
    "Customer": {"Name": "Jane Doe", "Email": "jane@old-domain.com"},
    "Items": [
        {"sku": "A1", "qty": "2", "price": "19.99"},
        {"sku": "B2", "qty": "1", "price": "9.5"},
    ],
    "Notes": None,
}

flat = (
    jt.JSONTools()
    .flatten()
    .separator("::")
    .lowercase_keys(True)
    .auto_convert_types(True)
    .remove_nulls(True)
    .execute(data)
)
# {'order_id': 'ORD-1001', 'customer::name': 'Jane Doe',
#  'customer::email': 'jane@old-domain.com', 'items::0::sku': 'A1',
#  'items::0::qty': 2, 'items::0::price': 19.99, 'items::1::sku': 'B2',
#  'items::1::qty': 1, 'items::1::price': 9.5}
# "Notes" is gone entirely -- remove_nulls ran before the flat map was built, so
# there's no "notes" key left for unflatten to see.

restored = jt.JSONTools().unflatten().separator("::").execute(flat)
# {'order_id': 'ORD-1001', 'customer': {'name': 'Jane Doe', 'email': 'jane@old-domain.com'},
#  'items': [{'sku': 'A1', 'qty': 2, 'price': 19.99}, {'sku': 'B2', 'qty': 1, 'price': 9.5}]}

The nested shape is fully restored, but qty/price come back as numbers (not the original strings) and notes never reappears -- unflatten can only rebuild structure from the keys it's given, it has no memory of what flatten discarded or converted.

Filtering

Remove unwanted values during flattening, unflattening, or .normal() processing. Filtering is applied recursively at every level of nesting.

Available Filters

Example

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonOutput};

let json = r#"{
    "name": "John",
    "bio": "",
    "age": null,
    "tags": [],
    "metadata": {},
    "city": "NYC"
}"#;

let result = JSONTools::new()
    .flatten()
    .remove_empty_strings(true)
    .remove_nulls(true)
    .remove_empty_arrays(true)
    .remove_empty_objects(true)
    .execute(json)?;

// Result: {"name": "John", "city": "NYC"}
}

import json_tools_rs as jt

data = {
    "name": "John",
    "bio": "",
    "age": None,
    "tags": [],
    "metadata": {},
    "city": "NYC",
}

result = (jt.JSONTools()
    .flatten()
    .remove_empty_strings(True)
    .remove_nulls(True)
    .remove_empty_arrays(True)
    .remove_empty_objects(True)
    .execute(data)
)
# {'name': 'John', 'city': 'NYC'}

Filtering with Unflatten

Filters also work during unflattening, applied after the nested structure is reconstructed:

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .unflatten()
    .remove_nulls(true)
    .remove_empty_strings(true)
    .execute(flat_json)?;
}

Combining Filters

All filters can be combined freely. They are applied after the flatten/unflatten operation completes.

Examples

Easy: drop nulls

import json_tools_rs as jt

data = {"name": "Alice", "middle_name": None}
result = jt.JSONTools().flatten().remove_nulls(True).execute(data)
# {'name': 'Alice'}

Medium: all four filters together

data = {"name": "John", "bio": "", "age": None, "tags": [], "metadata": {}, "city": "NYC"}

result = (jt.JSONTools()
    .flatten()
    .remove_empty_strings(True)
    .remove_nulls(True)
    .remove_empty_arrays(True)
    .remove_empty_objects(True)
    .execute(data)
)
# {'name': 'John', 'city': 'NYC'}

Hard: cascading removal in .normal() mode

Filtering runs bottom-up: a nested object's own children are filtered first, and if that leaves the object empty, remove_empty_objects removes it too -- even if it wasn't {} in the original input. This cascades all the way to the root in a single pass, so an object can vanish for a reason nowhere near itself, once every leaf inside it has been filtered away:

data = {
    "user": {"name": "Alice", "middle_name": "", "nickname": None},
    "session": {"token": "", "meta": {}},
    "tags": [],
}

result = (jt.JSONTools()
    .normal()
    .remove_empty_strings(True)
    .remove_nulls(True)
    .remove_empty_objects(True)
    .remove_empty_arrays(True)
    .execute(data)
)
# {'user': {'name': 'Alice'}}

session was never empty in the input -- but once token ("") and meta ({}) are both filtered out of it, session itself becomes {} and is removed on the same pass, one level up. tags (already []) is removed directly. user survives because name is non-empty, even though its two siblings were filtered away.

Key & Value Replacements

Replace patterns in keys and/or values using literal strings or regular expressions.

Key Replacements

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .flatten()
    .key_replacement("user_profile_", "")  // Literal
    .key_replacement("r'(User|Admin)_'", "")  // Regex
    .execute(json)?;
}

result = (jt.JSONTools()
    .flatten()
    .key_replacement("user_profile_", "")
    .key_replacement("r'(User|Admin)_'", "")
    .execute(data)
)

Value Replacements

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .flatten()
    .value_replacement("@example.com", "@company.org")  // Literal
    .value_replacement("r'^super$'", "administrator")  // Regex
    .execute(json)?;
}

Key Exclusion

Unlike key_replacement (which renames matched text within a key), exclude_key drops the entire key -- and its whole value/subtree -- from the output. Matching a container key removes everything under it, without those nested keys needing to match themselves:

#![allow(unused)]
fn main() {
let json = r#"{"user": {"name": "John", "crypto_wallet": {"coin": "BTC", "balance": 100}}}"#;
let result = JSONTools::new()
    .flatten()
    .exclude_key("crypto")  // Literal
    .exclude_key("r'^secret_'")  // Regex
    .execute(json)?;
// Output: {"user.name": "John"}
}

result = (jt.JSONTools()
    .flatten()
    .exclude_key("crypto")
    .execute(data)
)

Works identically in .flatten(), .unflatten(), and .normal() mode: checked against the full dot-path in flatten/unflatten mode, and per key at each nesting level in normal mode. Additive -- call it once per keyword to exclude multiple. Array elements are never matched, since they have no key name to check.

Value Exclusion

exclude_value is exclude_key's counterpart: it drops a key-value pair based on the value's content instead of the key's name.

#![allow(unused)]
fn main() {
let json = r#"{"user": {"name": "John", "status": "banned"}}"#;
let result = JSONTools::new()
    .flatten()
    .exclude_value("banned")  // Literal
    .exclude_value("r'^flag_'")  // Regex
    .execute(json)?;
// Output: {"user.name": "John"}
}

result = (jt.JSONTools()
    .flatten()
    .exclude_value("banned")
    .execute(data)
)

Unlike exclude_key, this only ever applies to scalar leaf values (strings, numbers, booleans, null) -- containers have no single value to check, so an object or array is never itself excluded; only its individual scalar leaves can be. The check runs after any configured value_replacement/auto_convert_types have run, so a value that only matches after being replaced or converted is still caught. It's a no-op at the document root, since there's no parent key to drop the value from.

Unflatten-specific note: string values are matched against their JSON-serialized form, including the surrounding quotes -- not the unescaped logical text. A literal pattern is unaffected by this (quotes don't change substring matching), but a regex with anchors needs to account for them: use r'^"admin"$', not r'^admin$', to match a value that's exactly "admin" in .unflatten() mode.

Regex Syntax

Wrap a pattern in r'...' (e.g. r'^prefix_') to use it as a regular expression. Any pattern not wrapped this way is matched as a literal, exact substring -- including patterns that contain characters that would otherwise be regex metacharacters (., $, (, etc.). The regex engine uses standard Rust regex syntax.

A malformed r'...' pattern (invalid regex syntax) is silently treated as "no match" for that pattern rather than raising an error -- test your patterns to confirm they compile as intended.

Multiple Replacements

You can chain multiple key and value replacements. They are applied in order:

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .flatten()
    .key_replacement("prefix_", "")
    .key_replacement("_suffix", "")
    .key_replacement("_", ".")
    .value_replacement("@old.com", "@new.com")
    .value_replacement("r'^admin$'", "administrator")
    .execute(json)?;
}

Examples

Easy: a single value replacement

import json_tools_rs as jt

data = {"user": {"email": "john@old-domain.com"}}
result = (jt.JSONTools()
    .flatten()
    .value_replacement("@old-domain.com", "@new-domain.com")
    .execute(data)
)
# {'user.email': 'john@new-domain.com'}

Medium: key + value replacement with regex

data = {"User_Name": "Alice", "User_Status": "super"}
result = (jt.JSONTools()
    .flatten()
    .key_replacement("r'^User_'", "")
    .value_replacement("r'^super$'", "administrator")
    .execute(data)
)
# {'Name': 'Alice', 'Status': 'administrator'}

Hard: normalizing an API response

Combines key replacement, value replacement, key exclusion, and value exclusion in one pipeline -- dropping an internal-only key, a sensitive subtree, and a banned-status record, while cleaning up the surviving keys and values:

data = {
    "api_response": {
        "user_id": "1001",
        "user_email": "john@old-domain.com",
        "user_status": "banned",
        "internal_debug_token": "xyz123",
        "crypto_wallet": {"coin": "BTC", "balance": 100},
    }
}

result = (jt.JSONTools()
    .flatten()
    .separator("::")
    .lowercase_keys(True)
    .key_replacement("r'^api_response::'", "")
    .key_replacement("_", ".")
    .value_replacement("@old-domain.com", "@new-domain.com")
    .exclude_key("internal")
    .exclude_key("crypto")
    .exclude_value("banned")
    .execute(data)
)
# {'user.id': '1001', 'user.email': 'john@new-domain.com'}

user_status is dropped because its value is "banned"; internal_debug_token and crypto_wallet.* are dropped by key. Note that exclude_key/exclude_value patterns are checked against keys after key_replacement has already run -- exclude_key uses "internal" (not "internal_") here because by the time the check runs, key_replacement("_", ".") has already turned internal_debug_token into internal.debug.token, so a pattern anchored on the underscore would no longer match.

Key Collision Handling

When key replacements or transformations cause multiple keys to map to the same output key, collision handling determines what happens.

Enabling Collision Handling

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .flatten()
    .key_replacement("r'(User|Admin)_'", "")
    .handle_key_collision(true)
    .execute(json)?;
}

result = (jt.JSONTools()
    .flatten()
    .key_replacement("r'(User|Admin)_'", "")
    .handle_key_collision(True)
    .execute(data)
)

How It Works

With .handle_key_collision(true), when two keys collide after transformation, their values are collected into an array:

// Input
{"User_name": "John", "Admin_name": "Jane"}

// With key_replacement("r'(User|Admin)_'", "") + handle_key_collision(true)
// Output
{"name": ["John", "Jane"]}

Without collision handling, the last value wins (overwrites previous values).

Collision with Filtering

Collision handling respects filters. If a colliding value would be filtered out (e.g., empty string with .remove_empty_strings(true)), it is excluded from the collected array:

// Input
{"User_name": "John", "Admin_name": "", "Guest_name": "Bob"}

// With key_replacement("r'(User|Admin|Guest)_'", "") + remove_empty_strings(true) + handle_key_collision(true)
// Output
{"name": ["John", "Bob"]}

Admin_name's empty-string value is dropped by the filter before collision resolution ever sees it, so only John and Bob end up in the array.

Works with All Modes

Collision handling works during .flatten(), .unflatten(), and .normal() operations.

Collected array order is not guaranteed to match input order -- collision resolution is backed by a hash map internally, so treat the array as an unordered bag of the colliding values, not a positional record of which key contributed which value.

Consistent Shape Across Documents: always_array_keys

A key's scalar-vs-array shape from .handle_key_collision(true) depends on whether a collision actually happened in that specific document: a key that collides in some rows of a batch (e.g. because of .key_replacement()) but not others ends up a plain value in some results and an array in others. That's awkward for anything expecting a stable schema -- including building a DataFrame column from the results, or normalise()'s own column typing, which needs every row's value to already be array-shaped to resolve a column to List<T>.

.always_array_keys([...]) names flattened key names that must always render as an array, even when only one value is present in a given document -- independent of .handle_key_collision():

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .flatten()
    .key_replacement("r'(User|Admin)_'", "")
    .always_array_keys(["name"])
    .execute(json)?;
}

result = (jt.JSONTools()
    .flatten()
    .key_replacement("r'(User|Admin)_'", "")
    .always_array_keys(["name"])
    .execute(data)
)

// Document with a collision -- same as handle_key_collision(true) alone
{"User_name": "John", "Admin_name": "Jane"}  ->  {"name": ["John", "Jane"]}

// Document with NO collision -- would be a bare string without always_array_keys;
// wrapped into a one-element array instead
{"User_name": "John"}  ->  {"name": ["John"]}

A document that doesn't have the key at all is unaffected -- the key stays absent, it is never injected. Keys not named in always_array_keys keep their ordinary behavior (governed by .handle_key_collision()).

Matched against the final flattened key name -- the same name .handle_key_collision() resolves collisions on -- and works for all operations (.flatten(), .unflatten(), .normal()) and for normalise()/DataFrame input, since those build on top of .flatten().

This is also the way to guarantee normalise() resolves a column to List<T> even when a particular batch happens to have zero collisions for it -- without always_array_keys, normalise() only promotes a column to List<T> once at least one row in that batch actually collides.

Examples

Easy: two colliding keys

import json_tools_rs as jt

data = {"User_name": "John", "Admin_name": "Jane"}
result = (jt.JSONTools()
    .flatten()
    .key_replacement("r'(User|Admin)_'", "")
    .handle_key_collision(True)
    .execute(data)
)
# {'name': ['John', 'Jane']}  (order not guaranteed)

Medium: collision plus filtering

data = {"User_name": "John", "Admin_name": "", "Guest_name": "Bob"}
result = (jt.JSONTools()
    .flatten()
    .key_replacement("r'(User|Admin|Guest)_'", "")
    .remove_empty_strings(True)
    .handle_key_collision(True)
    .execute(data)
)
# {'name': ['John', 'Bob']}  ("" from Admin_name never reaches the array)

Hard: collision after type conversion, in .normal() mode

Type conversion and filtering both run on each value before collision resolution collects it, so the array ends up holding already-converted values, not the original strings:

data = {"config": {"User_Score": "10", "Admin_Score": "20", "Guest_Score": "30"}}

result = (jt.JSONTools()
    .normal()
    .key_replacement("r'(User|Admin|Guest)_'", "")
    .handle_key_collision(True)
    .auto_convert_types(True)
    .execute(data)
)
# {'config': {'Score': [10, 20, 30]}}

All three sibling keys under config collapse to Score, and each "10"/"20"/"30" string has already become a JSON number by the time it lands in the array.

Automatic Type Conversion

When .auto_convert_types(true) is enabled, string values are automatically converted to their appropriate types.

Enabling

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .flatten()
    .auto_convert_types(true)
    .execute(json)?;
}

result = jt.JSONTools().flatten().auto_convert_types(True).execute(data)

try (JsonToolsHandle tools = JsonTools.builder().flatten().autoConvertTypes(true).build()) {
    String result = tools.execute(json);
}

Fine-Grained Control

auto_convert_types(true) turns on all four categories below (dates, nulls, booleans, numbers) with their default behavior. For independent control -- enabling only some categories, or customizing how a category matches -- use the per-category methods instead. Each accepts a plain bool for the common case, plus an optional customized config for the less common case.

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, DateConversionConfig, NullConversionConfig};

// Only convert numbers; leave dates/nulls/booleans as plain strings.
let result = JSONTools::new()
    .flatten()
    .convert_numbers(true)
    .execute(json)?;

// Customize a category: don't assume UTC for timezone-less datetimes, and
// recognize an extra null token.
let result = JSONTools::new()
    .flatten()
    .convert_dates_config(DateConversionConfig::new().enabled(true).assume_utc_for_naive(false))
    .convert_nulls_config(NullConversionConfig::new().enabled(true).add_extra_token("missing"))
    .execute(json)?;
}

# Only convert booleans.
result = jt.JSONTools().flatten().convert_booleans(True).execute(data)

# Customize with kwargs -- unset kwargs preserve whatever a previous call set.
result = (
    jt.JSONTools()
    .flatten()
    .convert_dates(True, assume_utc_for_naive=False)
    .convert_nulls(True, extra_tokens=["missing"])
    .execute(data)
)

try (JsonToolsHandle tools = JsonTools.builder()
        .flatten()
        .convertDates(true)
        .dateAssumeUtcForNaive(false)
        .convertNulls(true)
        .nullExtraToken("missing")
        .build()) {
    String result = tools.execute(json);
}

Calling auto_convert_types(bool) only ever flips each category's own on/off switch -- it never resets a category's customization back to its own defaults. So .convert_dates_config(...).auto_convert_types(true) keeps the customization while turning every category on, regardless of call order.

Per-Category Reference

Dates (DateConversionConfig):

• normalize_to_utc (default true) -- when false, a recognized date/datetime is left byte-for-byte unchanged (still protected from being misread as a number).
• assume_utc_for_naive (default true) -- when false, a timezone-less datetime (e.g. "2024-01-15T10:30:00") is left unchanged instead of getting a Z appended.

Nulls (NullConversionConfig) / Booleans (BooleanConversionConfig):

• extra_tokens / extra_true_tokens / extra_false_tokens -- additional strings recognized beyond the built-in list. Additive only: the built-in list stays active regardless, this only extends it. Matched exactly (case-sensitive) against the trimmed value -- consistent with every other category and the built-in lists (e.g. " 123 " already converts to 123), so a token like "si" also matches "si " (trailing whitespace), not only a byte-for-byte match against the raw string. In Rust, add one token per call (.add_extra_token("missing"), matching key_replacement()'s idiom); in Python, extra_tokens=[...] is bulk-replace (a later call's list replaces, not merges with, an earlier one); in Java, add one token per call (.nullExtraToken("missing"), additive like Rust).

Numbers (NumberConversionConfig) -- plain integers/decimals, scientific notation, and thousands-separator cleanup are always applied when the category is enabled (no one asked to disable unambiguous number parsing); the remaining sub-formats can each be disabled independently since they're more "opinionated" (each can reinterpret a string that wasn't meant to be a number):

• currency (default true) -- currency symbol/code/credit-debit-suffix stripping.
• percent (default true) -- %/permille/per-ten-thousand suffix parsing.
• basis_points (default true) -- text basis-point suffixes ("25bps").
• suffixes (default true) -- K/M/B/T magnitude suffixes.
• fractions (default true) -- fractions ("1/2").
• radix (default true) -- hex/binary/octal literals ("0x1A").

Conversion Rules

Conversions are applied in priority order: dates -> nulls -> booleans -> numbers.

Dates (ISO-8601)

Date strings are detected and normalized to UTC:

Nulls

Booleans

Note: "1" and "0" are treated as numbers, not booleans.

Numbers

Note: a 3-letter currency code (USD, EUR, GBP, etc.) is only stripped when followed by a space -- "EUR 999" converts to 999, but "EUR999" (no space) does not match and is left as the original string, to avoid misinterpreting things like alphanumeric product codes.

Note: 64-bit integer strings (e.g. Snowflake/Discord/database bigint IDs, commonly 17-19 digits) convert losslessly -- "999999999999999999" becomes the JSON integer 999999999999999999, not a precision-corrupted f64 approximation.

Non-Convertible Strings

Strings that don't match any pattern are left as-is:

{"name": "Alice", "code": "ABC"} -> {"name": "Alice", "code": "ABC"}

Full Example

#![allow(unused)]
fn main() {
let json = r#"{
    "id": "123",
    "price": "$1,234.56",
    "discount": "15%",
    "active": "yes",
    "created": "2024-01-15T10:30:00+05:00",
    "status": "N/A",
    "name": "Product"
}"#;

let result = JSONTools::new()
    .flatten()
    .auto_convert_types(true)
    .execute(json)?;

// {
//   "id": 123,
//   "price": 1234.56,
//   "discount": 15,
//   "active": true,
//   "created": "2024-01-15T05:30:00Z",
//   "status": null,
//   "name": "Product"
// }
}

Examples

Easy: turn it on

import json_tools_rs as jt

data = {"count": "42", "active": "true", "rate": "3.14"}
result = jt.JSONTools().normal().auto_convert_types(True).execute(data)
# {'count': 42, 'active': True, 'rate': 3.14}

Medium: one category, customized

Disable currency stripping specifically, while keeping percent/basis-point/suffix/ fraction/radix parsing and every other category untouched:

data = {"price": "$1,234.56", "big_id": "999999999999999999", "pct": "12.5%"}
result = jt.JSONTools().flatten().convert_numbers(True, currency=False).execute(data)
# {'price': '$1,234.56', 'big_id': 999999999999999999, 'pct': 12.5}

price is left as the original string (currency parsing is off), pct still converts (percent parsing is unaffected), and big_id -- a 19-digit ID -- converts losslessly to a JSON integer instead of a precision-corrupted f64.

Hard: mixed categories with per-category config

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, DateConversionConfig, NullConversionConfig};

let json = r#"{
    "created": "2024-01-15T10:30:00",
    "status": "missing",
    "score": "1,234.50",
    "code": "EUR999"
}"#;

let result = JSONTools::new()
    .flatten()
    .convert_dates_config(DateConversionConfig::new().enabled(true).assume_utc_for_naive(false))
    .convert_nulls_config(NullConversionConfig::new().enabled(true).add_extra_token("missing"))
    .convert_numbers(true)
    .execute(json)?;

// {
//   "created": "2024-01-15T10:30:00",  // naive datetime left untouched (assume_utc_for_naive: false)
//   "status": null,                    // "missing" recognized via the extra token
//   "score": 1234.5,                   // US thousands separator parsed
//   "code": "EUR999"                   // no space before the 3-letter code -- left as a string
// }
}

Booleans are never enabled here (auto_convert_types was not called, and convert_booleans was never turned on), so a value like "yes" elsewhere in the same document would be left as a plain string -- categories are independent switches, not an all-or-nothing package once you're using the per-category methods.

Normal Mode

Normal mode applies transformations (filtering, replacements, type conversion) without flattening or unflattening the JSON structure.

Usage

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .normal()
    .lowercase_keys(true)
    .remove_nulls(true)
    .remove_empty_strings(true)
    .auto_convert_types(true)
    .execute(json)?;
}

result = (jt.JSONTools()
    .normal()
    .lowercase_keys(True)
    .remove_nulls(True)
    .remove_empty_strings(True)
    .auto_convert_types(True)
    .execute(data)
)

When to Use Normal Mode

Use .normal() when you want to:

• Clean data without changing its structure
• Apply key transformations (lowercase, replacements), filtering, and type conversion recursively at every level of nesting, not just the top level
• Filter out unwanted values while preserving nesting
• Convert string types without flattening

Key replacement runs before lowercase_keys in normal mode (the opposite order from .flatten(), where lowercasing happens first). A pattern like r'^user_' is matched against the original-case key, so it won't match "User_Name" -- use r'^User_' (matching the actual input case) or a case-insensitive pattern like r'(?i)^user_' instead. This ordering difference is easy to trip over when porting a key_replacement pattern between .flatten() and .normal().

Example

import json_tools_rs as jt

data = {
    "User_Name": "alice@example.com",
    "User_Age": "",
    "User_Active": "true",
    "User_Score": None,
}

result = (jt.JSONTools()
    .normal()
    .lowercase_keys(True)
    .key_replacement("r'^User_'", "")
    .value_replacement("@example.com", "@company.org")
    .remove_empty_strings(True)
    .remove_nulls(True)
    .execute(data)
)
# {'name': 'alice@company.org', 'active': 'true'}

All features available in .flatten() and .unflatten() modes also work in .normal() mode, except the actual flattening/unflattening operation itself.

Examples

Easy: lowercase keys, structure untouched

import json_tools_rs as jt

data = {"User": {"Name": "Alice"}}
result = jt.JSONTools().normal().lowercase_keys(True).execute(data)
# {'user': {'name': 'Alice'}}

Medium: lowercase + replace + filter (see the example above)

The example above combines lowercase_keys, key_replacement, value_replacement, and two filters on a flat one-level object.

Hard: cascading filters on deeply nested data

Filters recurse into every level, and an object that becomes empty after its own children are filtered is itself removed on the same pass -- see Filtering for the full mechanics. In .normal() mode this applies at arbitrary depth, not just one level:

data = {
    "org": {
        "team": {
            "lead": {"name": "Priya", "notes": ""},
            "intern": {"name": "", "notes": None},
        }
    }
}

result = (jt.JSONTools()
    .normal()
    .remove_empty_strings(True)
    .remove_nulls(True)
    .remove_empty_objects(True)
    .execute(data)
)
# {'org': {'team': {'lead': {'name': 'Priya'}}}}

intern has no surviving fields (name is "", notes is null), so it collapses to {} and is removed -- which is exactly the same check that keeps lead around, just applied one level deeper.

Parallel Processing

JSON Tools RS uses Rayon-based parallelism to automatically speed up batch operations and large nested structures.

Automatic Parallelism

Batch processing (100+ items by default) automatically uses parallel execution:

#![allow(unused)]
fn main() {
let batch: Vec<&str> = large_json_collection;
let result = JSONTools::new()
    .flatten()
    .execute(batch.as_slice())?;
// Automatically parallelized
}

batch = [{"data": i} for i in range(2000)]
results = jt.JSONTools().flatten().execute(batch)
# Automatically parallelized

Configuration

Batch Threshold

Control the minimum batch size before parallelism kicks in:

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .flatten()
    .parallel_threshold(50)  // Only parallelize batches of 50+ items
    .execute(batch.as_slice())?;
}

Thread Count

Limit the number of threads used:

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .flatten()
    .num_threads(Some(4))  // Use 4 threads (default: CPU count)
    .execute(batch.as_slice())?;
}

Nested Parallelism

A single large JSON document being flattened can also be parallelized, based on how many direct children the root object/array has (nested containers deeper inside the document don't independently trigger this -- only the root's own fan-out is counted). This currently applies only to .flatten() -- .unflatten() and .normal() have no nested-parallel path and are unaffected by nested_parallel_threshold, though all three modes share the batch-level parallelism above.

#![allow(unused)]
fn main() {
let result = JSONTools::new()
    .flatten()
    .nested_parallel_threshold(200)  // Parallelize when the root has MORE than 200 direct children
    .execute(large_json)?;
}

Python Configuration

tools = (jt.JSONTools()
    .flatten()
    .parallel_threshold(50)
    .num_threads(4)
    .nested_parallel_threshold(200)
)

results = tools.execute(large_batch)

How It Works

• Batch parallelism: Input is split into chunks processed via Rayon's par_chunks. By default this runs on Rayon's persistent, process-wide work-stealing pool (no per-call thread spawn cost); setting .num_threads(Some(n)) instead builds and installs a dedicated pool sized to n for that call. Results preserve input order. Applies to .flatten(), .unflatten(), and .normal() alike.
• Nested parallelism: A .flatten() call on a single document whose root object/array has more than nested_parallel_threshold direct children splits those children across threads for parallel flattening, then merges the results.
• Thread safety: Rayon's work-stealing model requires no 'static bounds and guarantees no data races.

Environment Variables

All parallelism settings can be overridden via environment variables (applied at construction time):

export JSON_TOOLS_PARALLEL_THRESHOLD=50
export JSON_TOOLS_NESTED_PARALLEL_THRESHOLD=200
export JSON_TOOLS_NUM_THREADS=4
export JSON_TOOLS_MAX_ARRAY_INDEX=500000

Environment variables are read once per process -- at the first JSONTools::new() call anywhere in the program, not on every call -- and the resulting defaults are cached for the rest of the process's lifetime. Set them before your program starts; changing them at runtime (e.g. via std::env::set_var) after the first JSONTools has already been constructed has no effect. Builder method calls (e.g., .parallel_threshold(n)) always override the compiled-in default on a per-instance basis, regardless of when the environment variable was read.

Examples

Easy: let the default threshold decide

import json_tools_rs as jt

batch = [{"user": {"id": i}} for i in range(250)]
results = jt.JSONTools().flatten().execute(batch)
# 250 items, well over the default threshold of 100 -- parallelized automatically,
# no configuration needed. Order matches the input order.

Medium: tune the threshold and thread count together

batch = [{"user": {"id": i, "score": str(i * 1.5)}} for i in range(500)]

tools = (jt.JSONTools()
    .flatten()
    .parallel_threshold(50)   # parallelize batches of 50+ (lower than the default 100)
    .num_threads(4)           # cap at 4 threads for this call
    .auto_convert_types(True)
)
results = tools.execute(batch)

.num_threads(Some(n)) builds a dedicated Rayon pool sized to n just for this call; leave it unset to reuse the process-wide work-stealing pool (no per-call spawn cost).

Hard: batch + nested parallelism + env-configured limits together

A single very wide document (many direct children at the root) and a large batch of documents can both be parallelized at once -- they're independent mechanisms that stack:

#![allow(unused)]
fn main() {
use json_tools_rs::JSONTools;

// Each document in the batch has 300 top-level keys; the batch itself has 1000 items.
let batch: Vec<String> = generate_wide_documents(1000, 300);
let batch_refs: Vec<&str> = batch.iter().map(String::as_str).collect();

let result = JSONTools::new()
    .flatten()
    .parallel_threshold(100)          // batch-level: split 1000 items across threads
    .nested_parallel_threshold(200)   // per-document: each doc's 300 root keys (> 200) also fan out
    .num_threads(Some(8))
    .execute(batch_refs.as_slice())?;
}

Set JSON_TOOLS_MAX_ARRAY_INDEX lower than its 100,000 default in an environment that processes untrusted input, to bound how large an array .unflatten() will build from a numeric key like "items.99999.name" before treating it as suspicious rather than allocating a 100k-element array per document.

DataFrame & Series Support

The Python bindings natively support DataFrame and Series objects from popular data libraries, with perfect type preservation.

Note: the .execute(df) convenience shown here collects the DataFrame to the driver, processes it through the Rust engine, and reconstructs a new DataFrame -- fine for smaller data, but not how you want to process a large distributed Spark dataset. For genuinely distributed, per-partition processing (including from inside a Databricks Lakeflow Declarative Pipeline, where this is the only supported approach -- see Setting Up on Databricks), wrap the Python bindings in a pandas_udf instead, so each executor runs its own share of the work.

Supported Libraries

PySpark DataFrame reconstruction. A PySpark DataFrame is accepted as input (it's converted via .toPandas() internally, processed, then converted back) and .execute(df) reconstructs a genuine PySpark DataFrame (#31; earlier versions returned a plain Python list of dicts here, since there was no SparkSession reachable at reconstruction time -- an active session is now auto-discovered via SparkSession.getActiveSession(), the same mechanism normalise(target="pyspark") already used). This is still a reminder that .execute(df) on a PySpark DataFrame collects the whole thing to the driver first (see the note above) and only the final reconstruction step is genuinely distributed (via Spark's own Arrow-optimized SparkSession.createDataFrame(pandas.DataFrame, schema) bridge) -- the flatten/processing computation itself is not distributed. For that, use the pandas_udf pattern further down this page.

normalise=True (see below) uses this exact same reconstruction mechanism -- the two paths are now consistent with each other, not just individually documented.

Usage

Pandas DataFrame

import json_tools_rs as jt
import pandas as pd

df = pd.DataFrame([
    {"user": {"name": "Alice", "age": 30}},
    {"user": {"name": "Bob", "age": 25}},
])

result = jt.JSONTools().flatten().execute(df)
print(type(result))  # <class 'pandas.core.frame.DataFrame'>
print(result.columns.tolist())  # ['name', 'age']

Polars DataFrame

import json_tools_rs as jt
import polars as pl

df = pl.DataFrame([
    {"user": {"name": "Alice", "age": 30}},
    {"user": {"name": "Bob", "age": 25}},
])

result = jt.JSONTools().flatten().execute(df)
print(type(result))  # <class 'polars.dataframe.frame.DataFrame'>
print(result.columns)  # ['name', 'age']

A column holding pre-serialized JSON strings (e.g. pl.DataFrame({"data": ['{"a": 1}', ...]})) also flattens correctly in .flatten() mode -- execute() auto-detects columns holding JSON strings and expands them the same way a struct-typed column already does, so data becomes a here too (the column's own name, data, is never kept as a prefix). See Auto-Expanding JSON-String Columns below for the detection rules and caveats.

Pandas Series

import json_tools_rs as jt
import pandas as pd

series = pd.Series(['{"a": {"b": 1}}', '{"c": {"d": 2}}'])
result = jt.JSONTools().flatten().execute(series)
print(type(result))  # <class 'pandas.core.series.Series'>

Auto-Expanding JSON-String Columns

A DataFrame column that's already a dict/struct expands into flattened columns automatically -- that's just .flatten() finding real nested JSON in the row. A column holding pre-serialized JSON strings (common with data loaded from a JSON/JSONL file, a database TEXT/JSON column, or an upstream system that already serialized a payload) used to stay an opaque string instead, since a string value isn't something .flatten() re-parses -- that's not its contract. execute() on a DataFrame in .flatten() mode now detects columns holding JSON strings and expands them the same way, so it "just works" without a manual pre-parsing step (see issue #30):

import json_tools_rs as jt
import pandas as pd

df = pd.DataFrame({
    "id": [1, 2],
    "payload": ['{"user": {"name": "Alice"}}', '{"user": {"name": "Bob"}}'],
})

result = jt.JSONTools().flatten().execute(df)
print(result)
#    id user.name
# 0   1     Alice
# 1   2       Bob

This closes the gap the previous version of this page documented for the polars write_ndjson case above, and applies uniformly to pandas, polars, pyarrow, and PySpark (PySpark DataFrames convert to pandas internally first, so they get this for free too).

The source column's own name is never kept as a prefix. Every top-level key in a DataFrame row is a column name by construction, so an object-valued column (dict/struct-typed, or a JSON-string column decoded per the rules below) expands using only its own inner keys -- payload above contributes nothing to the output column names; user does, because that's genuine nesting within payload's own content, one level deeper. This applies uniformly whether the column arrived as a native dict/struct or as a JSON string, and to both plain execute(df) and execute(df, normalise=True).

If two different columns' contents (or a column's content and another top-level column) share a key name, that's a genuine collision -- resolved by whatever .handle_key_collision() is already set to, exactly as it would be for any other duplicate key this engine encounters: collected into an array when True, last value wins when False (the default).

A JSON-string-encoded array is the one exception -- it stays nested under its original column name (tags.0, tags.1, ...) rather than un-nesting, since a bare 0/1/... column name wouldn't be meaningful and would risk colliding across every array-valued column in the DataFrame.

Detection rules

• Runs only in .flatten() mode -- .unflatten() and .normal() DataFrame processing are unaffected; a JSON-string column stays exactly as-is in those modes.

• A column is a candidate only if its values, when parsed, are JSON objects or arrays -- not any scalar. A column of plain strings that happen to parse as a bare number/bool/null is never touched, and neither is a column of ordinary text:

  df = pd.DataFrame({"id": [1], "notes": ["just some text"]})
  result = jt.JSONTools().flatten().execute(df)
  print(result)
  #    id           notes
  # 0   1  just some text
  

• Detection samples the first 20 rows: a column must parse successfully as JSON in every sampled row where it holds a string value, or the whole column is left untouched (conservative -- no partial/mixed expansion). A column that's None in every one of the first 20 rows won't be detected even if later rows hold real JSON -- a known limitation of sample-based detection, not a crash.

• A JSON-string-encoded array expands into indexed sub-columns (col.0, col.1, ...) the same way an already-list-typed column does today -- including for a large array (e.g. a stringified embedding vector with hundreds of elements). There's currently no cap on this (unlike .unflatten()'s max_array_index, which guards against reconstructing a huge sparse array from a numeric key, not against a real array this large) -- a very wide array column will produce that many DataFrame columns.

• A row that fails to re-parse despite its column being detected (malformed JSON in just that one row, past the sample) keeps its original string value for that row only, and a Python warning is emitted naming the column and how many rows were affected -- so this stays visible instead of silently leaving that row's data in a different shape than the rest of the column.

Polars/PyArrow: zero-copy detection and extraction. For these two libraries specifically, detecting and reading a JSON-string column happens via pyo3-arrow's direct Arrow buffer access rather than through the DataFrame's own JSON writer -- ~41-48% faster execute() end-to-end for a table with a large embedded payload, since the writer no longer has to escape that content only for it to be immediately unescaped again. Behavior is identical either way (same detection rules, same column ordering, same fallback/warning for a row that doesn't actually parse); this only changes how the data gets read. Plain pandas isn't Arrow-backed by default and PySpark bridges through pandas already, so both keep using the text-based path described above.

Performance: The Flat-DataFrame Fast Path

execute(df) on a pandas, Polars, or PyArrow DataFrame automatically takes a faster path when the DataFrame has no nested columns to flatten -- a common case: calling .flatten() defensively in a generic pipeline, or using it purely for its key-transform (.lowercase_keys(), .key_replacement()) or .auto_convert_types() features on data that's already tabular. Normally, execute(df) serializes the whole DataFrame to JSON text, parses and flattens each row, deserializes the result back into Python objects, and reconstructs a DataFrame from those -- real work even when there's nothing nested. When every column is scalar (no struct/list/embedded-JSON-string columns), the fast path instead reads column values directly, applies the exact same per-cell transform logic in Rust, and writes results back into columns natively -- skipping the JSON round trip entirely.

import polars as pl
import json_tools_rs as jt

df = pl.DataFrame({
    "UserId": [1, 2, 3],
    "UserName": ["Alice", "Bob", "Carol"],
    "Amount": ["$19.99", "$45.00", "$8.50"],
})

# Already flat -- lowercase_keys()/auto_convert_types() apply directly to
# columns, no JSON round trip.
result = (jt.JSONTools()
    .flatten()
    .lowercase_keys(True)
    .auto_convert_types(True)
    .execute(df))

This is purely an internal optimization -- output is identical to the existing pipeline's, and it's entirely automatic (no flag to set). It falls back to the normal pipeline whenever any column has real nested structure to flatten, an embedded JSON-string column, .remove_nulls()/.exclude_value() configured, or a key-transform-induced column name collision -- in every one of those cases execute(df) behaves exactly as it always has.

The fast path's computation also releases Python's GIL while it runs, same as every other execute() call -- so it doesn't stall other Python threads in your process (a multi-threaded web server, a ThreadPoolExecutor) for the duration of a large DataFrame call. This doesn't change how long a single execute(df) call takes; it changes whether other threads can make progress while it runs.

Normalise: Always Get a Wide DataFrame

.execute() normally mirrors the input's own type (str→str, dict→dict, DataFrame→ DataFrame). execute(data, normalise=True) instead always returns a genuine wide DataFrame -- one column per flattened key -- no matter what shape data is: a bare JSON string or dict becomes a 1-row DataFrame, a list becomes an N-row one, and an existing DataFrame/Series gets re-normalised the same way. Requires .flatten() mode (a JsonToolsError explains why if it's not set -- unflattened/nested JSON can't produce clean scalar columns).

Arrow-native reconstruction. normalise's reconstruction builds one real Apache Arrow table directly in Rust -- with genuinely typed columns, including real List<T> columns for a handle_key_collision(True) result (not a stringified fallback) and real Date32/Timestamp columns for recognized dates/datetimes -- then derives whichever target was requested from it. This has four consequences worth knowing about:

• pandas output uses Arrow-backed dtypes (int64[pyarrow], string[pyarrow], ...) instead of the classic numpy-backed dtypes, e.g. df["id"].dtype is now int64[pyarrow], not int64. This is a deliberate choice -- it's genuinely zero-copy -- but is a real, breaking change to .dtype for code written against the old output.
• target="pandas" and target="pyspark" now require pyarrow installed, even though neither target returns a pyarrow object -- there's no pyarrow-free route to get genuinely Arrow-built data into a pandas DataFrame (verified directly). target="polars" is unaffected and stays usable without pyarrow, same as before.
• A column mixing genuinely different scalar kinds across rows (e.g. an int in one row, a string in another, with no list involved) still falls back to a stringified column -- Arrow's Union type is the only real alternative and was confirmed unusable across pandas/polars (both reject it outright), not a shortcut taken here.
• A column only resolves to List<T> once at least one row in that batch actually collides -- a batch where a key never happens to collide gets a plain scalar column, even if that same key collides in some other batch. If you need a column to always be List<T> regardless of any particular batch's luck, use .always_array_keys([...]) -- it forces that key's shape at the .flatten() level, before normalise()'s column typing ever runs.
• A recognized date/datetime column gets a real Date32/Timestamp type only when .convert_dates(True)/.auto_convert_types(True) is enabled -- this engine never independently pattern-matches an ordinary string into a date; it only promotes what the core engine's own opt-in date recognition already normalized. A bare date ("2024-01-15") becomes Date32; a datetime becomes a UTC Timestamp (any input timezone offset is converted, not just relabeled); a column mixing dates and datetimes promotes to Timestamp (the date becomes midnight UTC), the same "promote the narrower kind" rule int→float already uses. Date detection applies to top-level columns only, not to elements inside a handle_key_collision list.

A single record → a 1-row DataFrame

No DataFrame library or wrapping needed on the input side at all -- useful for turning a single API response or log line straight into a table row:

import json_tools_rs as jt

tools = jt.JSONTools().flatten()

df = tools.execute({"user": {"name": "Alice", "age": 30}}, normalise=True)
print(df)
#   user.name  user.age
# 0     Alice        30

Heterogeneous records → union + null-fill

A list of records that don't all share the same keys gets unioned into one consistent set of columns, in first-seen order, with None/null filling any row that's missing a given key -- the same union/null-fill behavior the "Medium" example above shows for an existing DataFrame, just starting from plain dicts instead:

import json_tools_rs as jt

tools = jt.JSONTools().flatten()
data = [
    {"a": 1, "b": {"x": "hi"}},
    {"a": 2, "c": True},
]
df = tools.execute(data, normalise=True)
print(df)
#    a   b.x     c
# 0  1    hi  None
# 1  2  None  True

Choosing the target library

Pass target to pick the library explicitly, or omit it to auto-resolve: an input that's already a live DataFrame/Series keeps that backend; otherwise pandas → polars → pyarrow is tried in order (first installed wins). target="pyspark" is never chosen automatically for bare JSON input -- see below.

import json_tools_rs as jt

tools = jt.JSONTools().flatten()
data = [{"a": 1, "b": 2}, {"a": 3, "b": 4}]

pandas_df = tools.execute(data, normalise=True, target="pandas")
polars_df = tools.execute(data, normalise=True, target="polars")
arrow_table = tools.execute(data, normalise=True, target="pyarrow")

print(type(pandas_df), type(polars_df), type(arrow_table))
# <class 'pandas.core.frame.DataFrame'> <class 'polars.dataframe.frame.DataFrame'> <class 'pyarrow.lib.Table'>

target=None's auto-resolution also applies when the input is already a live DataFrame/Series -- useful for re-normalising into a different backend than the one you started with, or for cleaning up something that isn't wide yet:

import json_tools_rs as jt
import pandas as pd

tools = jt.JSONTools().flatten()
pandas_df = pd.DataFrame([{"user": {"name": "Alice"}}, {"user": {"name": "Bob"}}])

# target=None here would keep pandas (input's own backend); pass target= to convert
polars_df = tools.execute(pandas_df, normalise=True, target="polars")
print(type(polars_df))  # <class 'polars.dataframe.frame.DataFrame'>

Composing with the rest of the builder pipeline

normalise is just the reconstruction step -- every other builder feature still runs first, exactly as it would for plain .execute():

import json_tools_rs as jt

tools = (
    jt.JSONTools()
    .flatten()
    .separator("::")
    .remove_nulls(True)
    .key_replacement("r'^admin_'", "")
    .auto_convert_types(True)
)

data = [
    {"admin_name": "Jane", "admin_status": None, "count": "42"},
    {"admin_name": "Bob", "count": "7"},
]
df = tools.execute(data, normalise=True, target="pandas")
print(df)
#    name  count
# 0  Jane     42
# 1   Bob      7

PySpark: a real distributed DataFrame, not a list

target="pyspark" requires an active SparkSession (auto-discovered via SparkSession.getActiveSession()) and is never chosen automatically for bare JSON input -- only via an explicit target="pyspark", or when the input itself was already a live PySpark object:

import json_tools_rs as jt
from pyspark.sql import SparkSession

SparkSession.builder.getOrCreate()  # normalise auto-discovers this

tools = jt.JSONTools().flatten()
data = [{"user": {"name": "Alice", "age": 30}}, {"user": {"name": "Bob", "age": 25}}]

spark_df = tools.execute(data, normalise=True, target="pyspark")
from pyspark.sql import DataFrame as SparkDataFrame
print(isinstance(spark_df, SparkDataFrame))  # True -- a real, distributed DataFrame
spark_df.show()
# +---------+--------+
# |user.name|user.age|
# +---------+--------+
# |    Alice|      30|
# |      Bob|      25|
# +---------+--------+

Under the hood, the pyspark target reuses the exact same pandas reconstruction as target="pandas", then hands that DataFrame -- plus an explicit StructType schema computed from the data -- to Spark's own Arrow-optimized SparkSession.createDataFrame(pandas.DataFrame, schema) bridge, rather than letting Spark infer the schema itself. This isn't just style: schema inference from a pandas DataFrame is unreliable specifically on the non-Arrow fallback path Spark silently takes when pyarrow isn't installed (pyspark does not depend on pyarrow) -- an explicit schema sidesteps that entirely. See the note earlier on this page for what this bridge does and doesn't distribute (the reconstruction, not the flatten computation itself).

Mixed-type columns with auto_convert_types. auto_convert_types(True) converts each value independently based on its own content, so the same flattened key can hold a clean numeric string in one row ("123" -> int 123) and ordinary text in another ("Smith" -> stays str). A column like that is detected and stringified as a whole (falling back to a string type for every value in it) rather than producing an inconsistent or broken result -- true for every target now (pyspark's Arrow bridge was always strict about this and would raise PySparkTypeError; pandas/polars/pyarrow now share the exact same real-Arrow-type reconstruction, so they get the same protection, not just a Python-level object-dtype column as before). Columns mixing only int and float are unaffected -- they promote to a numeric Float64 column. The same protection applies one level down for .key_replacement() / .handle_key_collision(True) list columns: a collision list is built from each colliding key's own independently- converted value, so a single row's collision can itself hold mixed element types (e.g. [100, "abc"]) -- those elements fall back to strings too, while a uniformly-typed collision column (e.g. every element an int) gets a real, correctly-typed List<Int64> Arrow column instead of a stringified one.

What happens without .flatten() mode

normalise=True needs .flatten() mode specifically -- .unflatten(), .normal(), or no mode set all raise a clear error rather than silently producing columns full of nested objects:

import json_tools_rs as jt

tools = jt.JSONTools().unflatten()
tools.execute({"a.b": 1}, normalise=True)
# json_tools_rs.JsonToolsError: normalise=True requires .flatten() mode -- unflattened/nested
# JSON can't produce clean scalar columns for a wide DataFrame

target is only meaningful alongside normalise=True -- setting it without also setting normalise=True is rejected too, rather than silently ignored:

tools = jt.JSONTools().flatten()
tools.execute({"a": 1}, target="pandas")  # normalise=True missing
# json_tools_rs.JsonToolsError: target is only valid when normalise=True

How It Works

1. Detection: The library uses duck typing to detect DataFrame/Series objects (checks for .to_dict(), .to_list(), etc.)
2. Extraction: Rows are extracted as JSON strings or dicts
3. JSON-string-column expansion (.flatten() mode only): columns holding JSON strings are detected and spliced into genuine nested JSON in each row -- see Auto-Expanding JSON-String Columns above
4. Processing: Each row is processed through the Rust engine (with automatic parallelism for large DataFrames)
5. Reconstruction: Results are reconstructed into the original DataFrame/Series type -- O(1) constructor calls for pandas/polars/pyarrow, or a schema-driven SparkSession.createDataFrame(...) call for PySpark (see the note above)

All Features Apply

DataFrames and Series support all the same features as regular input:

tools = (jt.JSONTools()
    .flatten()
    .separator("::")
    .lowercase_keys(True)
    .remove_nulls(True)
    .auto_convert_types(True)
    .parallel_threshold(50)
)

result = tools.execute(large_dataframe)

Examples

Easy: flatten a Pandas DataFrame

import json_tools_rs as jt
import pandas as pd

df = pd.DataFrame([{"user": {"name": "Alice", "age": 30}}, {"user": {"name": "Bob", "age": 25}}])
result = jt.JSONTools().flatten().execute(df)
# DataFrame with columns ['name', 'age'] -- "user" is the column name, not kept

Medium: Polars struct column with filtering

import polars as pl

df = pl.DataFrame([
    {"user": {"name": "Alice", "age": 30, "bio": ""}},
    {"user": {"name": "Bob", "age": None, "bio": "hi"}},
])

result = (jt.JSONTools()
    .flatten()
    .remove_empty_strings(True)
    .remove_nulls(True)
    .execute(df)
)
# shape: (2, 3)
# ┌───────┬──────┬──────┐
# │ name  ┆ age  ┆ bio  │
# ╞═══════╪══════╪══════╡
# │ Alice ┆ 30   ┆ null │
# │ Bob   ┆ null ┆ hi   │
# └───────┴──────┴──────┘

Filtering is per-row, but a DataFrame's columns are shared across all rows. Row 0's bio ("") was filtered out of that row, and row 1's age (null) was filtered out of that row -- but since each column still exists (some other row still has a value there), the filtered-out cell shows up as null in the reconstructed DataFrame rather than making the column disappear or shifting columns per row.

Hard: distributed processing with PySpark via pandas_udf

.execute(df) collects a DataFrame to the driver first -- fine for the two examples above, but not for a large distributed Spark dataset. For genuinely distributed, per-partition processing, wrap the bindings in a pandas_udf instead, so each executor processes its own share of the data with one native call per Arrow-vectorized batch (not per row):

import json_tools_rs as jt
import pandas as pd
from pyspark.sql.functions import pandas_udf
from pyspark.sql.types import StringType

_tools = (
    jt.JSONTools()
    .flatten()
    .separator("::")
    .remove_nulls(True)
    .key_replacement("r'^admin_'", "")
)

@pandas_udf(StringType())
def flatten_json(payload: pd.Series) -> pd.Series:
    return pd.Series(_tools.execute(payload.tolist()))

spark_df.withColumn("flattened", flatten_json(spark_df["payload"]))

Build the JSONTools instance once at module scope (it's reusable across calls), not inside the UDF function body. See Setting Up on Databricks for the full walkthrough, including why this is the only supported approach inside a Lakeflow Declarative Pipeline.

Setting Up on Databricks

Short version: to run json-tools-rs inside a Lakeflow Declarative Pipeline (formerly Delta Live Tables), a regular notebook, or a Databricks Job, use the Python bindings as a pandas_udf -- validated and shown below.

Using the Python bindings inside a Lakeflow Declarative Pipeline

Python packages -- including ones backed by a compiled native extension, like this one -- are a fully supported pipeline dependency. Wrapping the Python bindings in a pandas_udf gives you one native call per Arrow-vectorized batch instead of per row, while running as a genuinely distributed Python UDF across executors, not something collected to the driver. This was validated directly (not assumed) against a real Spark session running the pattern below before writing it here.

1. Add the dependency

From the pipeline editor: Settings → Pipeline environment → Edit environment → Add dependency, then enter json-tools-rs (already published to PyPI -- see Installation). Alternatively, build a wheel locally (maturin build --release --features python) and install it from a Unity Catalog Volume path instead, the same way pipeline dependencies support installing a wheel from a volume.

2. Define the UDF

Build the JSONTools instance once at module scope, not inside the UDF function body -- it's reusable across calls (the same instance can call .execute() repeatedly), and the underlying regex/pattern cache is process-wide, so there's no benefit to reconstructing it per batch:

import json_tools_rs as jt
import pandas as pd
from pyspark.sql.functions import pandas_udf
from pyspark.sql.types import StringType

_flatten_tools = (
    jt.JSONTools()
    .flatten()
    .separator("::")
    .remove_nulls(True)
    .key_replacement("r'^admin_'", "")
)


@pandas_udf(StringType())
def flatten_json(payload: pd.Series) -> pd.Series:
    return pd.Series(_flatten_tools.execute(payload.tolist()))

3. Use it in a pipeline table

import dlt
from pyspark.sql.functions import col

@dlt.table
def flattened_events():
    return (
        dlt.read_stream("raw_events")
        .withColumn("flattened_payload", flatten_json(col("payload")))
    )

That's it -- no jar, no cluster library configuration, no spark._jvm escape hatch. It works identically whether the pipeline runs on serverless or classic compute, since it's an ordinary Python dependency as far as Databricks is concerned.

Malformed input raises json_tools_rs.JsonToolsError inside the UDF, which fails the task the same way any Python UDF exception does -- wrap the .execute() call in a try/except inside the UDF function if you'd rather emit None for bad rows than fail the pipeline update, or use a Lakeflow expectation to quarantine rows that fail a validity check upstream of the UDF.

Outside a Lakeflow pipeline -- a plain notebook cell, or a notebook/Python task in a Databricks Job, running on a classic all-purpose or job cluster -- the same Python bindings and pandas_udf pattern work directly; nothing above is pipeline-specific.

Rust API Reference

Full API documentation is available on docs.rs.

JSONTools

The main builder struct for all JSON operations. Uses the owned-self builder pattern -- all configuration methods consume and return Self for chaining.

Construction

#![allow(unused)]
fn main() {
use json_tools_rs::JSONTools;

let tools = JSONTools::new();
}

JSONTools implements Default, Debug, and Clone.

Operation Modes

Exactly one mode must be set before calling .execute().

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonOutput};

// Flatten
let result = JSONTools::new()
    .flatten()
    .execute(r#"{"a": {"b": 1}}"#)?;

// Unflatten
let result = JSONTools::new()
    .unflatten()
    .execute(r#"{"a.b": 1}"#)?;

// Normal mode -- transformations only
let result = JSONTools::new()
    .normal()
    .lowercase_keys(true)
    .auto_convert_types(true)
    .execute(r#"{"Name": "John", "Age": "30"}"#)?;
}

Configuration Methods

All methods consume self and return Self for chaining. Marked #[must_use].

Note: .separator() itself never fails -- an empty separator is only rejected later, at .execute() time, with a ConfigurationError (E005), not a panic. Defaults for parallel_threshold, nested_parallel_threshold, num_threads, and max_array_index can be overridden via environment variables (see Performance Tuning). See Automatic Type Conversion for the DateConversionConfig/NullConversionConfig/BooleanConversionConfig/NumberConversionConfig field reference and customization examples; .auto_convert_types(flag) only ever flips each category's enabled bit and preserves prior customization set via the _config methods.

Execution

#![allow(unused)]
fn main() {
pub fn execute<'a, T>(&self, json_input: T) -> Result<JsonOutput, JsonToolsError>
where
    T: Into<JsonInput<'a>>,
}

Accepts any type that implements Into<JsonInput>:

Errors: Returns Err(JsonToolsError) if no mode is set, JSON is invalid, or processing fails.

Full Example

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonOutput};

let tools = JSONTools::new()
    .flatten()
    .separator("::")
    .lowercase_keys(true)
    .remove_nulls(true)
    .remove_empty_strings(true)
    .key_replacement("r'^user_'", "")
    .auto_convert_types(true)
    .parallel_threshold(50)
    .num_threads(Some(4));

// Single document
let result = tools.execute(r#"{"User_Name": "Alice", "User_Age": "30"}"#)?;
match result {
    JsonOutput::Single(s) => println!("{}", s),
    JsonOutput::Multiple(_) => unreachable!(),
}

// Batch processing
let batch: Vec<String> = (0..1000)
    .map(|i| format!(r#"{{"id": "{}"}}"#, i))
    .collect();
let results = tools.execute(batch)?;
match results {
    JsonOutput::Multiple(v) => println!("Processed {} items", v.len()),
    JsonOutput::Single(_) => unreachable!(),
}
}

JsonInput

Input enum for execute(). You rarely construct this directly -- the From implementations handle conversion automatically.

#![allow(unused)]
fn main() {
pub enum JsonInput<'a> {
    /// Single JSON string (zero-copy via Cow)
    Single(Cow<'a, str>),
    /// Multiple JSON strings (borrowing)
    Multiple(&'a [&'a str]),
    /// Multiple JSON strings (owned or mixed)
    MultipleOwned(Vec<Cow<'a, str>>),
}
}

From Implementations

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonOutput};

let tools = JSONTools::new().flatten();

// All of these work transparently:
let _ = tools.execute(r#"{"a": 1}"#);                      // &str
let s = String::from(r#"{"a": 1}"#);
let _ = tools.execute(&s);                                  // &String
let batch = vec![r#"{"a": 1}"#, r#"{"b": 2}"#];
let _ = tools.execute(batch);                               // Vec<&str>
let owned: Vec<String> = vec![r#"{"a": 1}"#.into()];
let _ = tools.execute(owned);                               // Vec<String>
}

JsonOutput

Output enum from execute().

#![allow(unused)]
fn main() {
pub enum JsonOutput {
    /// Single JSON result string
    Single(String),
    /// Multiple JSON result strings (batch)
    Multiple(Vec<String>),
}
}

Methods

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonOutput};

let result = JSONTools::new().flatten().execute(r#"{"a": {"b": 1}}"#)?;

// Pattern matching (recommended)
match result {
    JsonOutput::Single(s) => println!("Single: {}", s),
    JsonOutput::Multiple(v) => println!("Batch of {}", v.len()),
}

// Direct extraction (panics on wrong variant)
let s = JSONTools::new().flatten().execute(r#"{"a": 1}"#)?.into_single();

// Safe extraction (returns Result)
let s = JSONTools::new().flatten().execute(r#"{"a": 1}"#)?.try_into_single()?;

// Always-vec (useful for uniform handling)
let v = JSONTools::new().flatten().execute(r#"{"a": 1}"#)?.into_vec();
assert_eq!(v.len(), 1);
}

JsonToolsError

Comprehensive error enum with machine-readable error codes (E001-E008), human-readable messages, and actionable suggestions.

#![allow(unused)]
fn main() {
#[derive(Debug)]
#[non_exhaustive]
pub enum JsonToolsError {
    JsonParseError { .. },           // E001
    RegexError { .. },               // E002
    InvalidReplacementPattern { .. }, // E003
    InvalidJsonStructure { .. },     // E004
    ConfigurationError { .. },       // E005
    BatchProcessingError { .. },     // E006
    InputValidationError { .. },     // E007
    SerializationError { .. },       // E008
}
}

Display and std::error::Error are implemented by hand (not via the thiserror crate, which is not a dependency of this crate) -- Display produces the [E00x] ... 💡 Suggestion: ... text shown below.

Methods

Error Handling Example

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonToolsError};

let result = JSONTools::new().flatten().execute("invalid json");

match result {
    Ok(output) => { /* success */ }
    Err(e) => {
        // Machine-readable error code
        match e.error_code() {
            "E001" => eprintln!("JSON parsing error: {}", e),
            "E005" => eprintln!("Configuration error: {}", e),
            "E006" => eprintln!("Batch error: {}", e),
            code => eprintln!("[{}] {}", code, e),
        }

        // Pattern matching for specific handling
        match &e {
            JsonToolsError::JsonParseError { message, suggestion, .. } => {
                eprintln!("Parse failed: {}", message);
                eprintln!("Try: {}", suggestion);
            }
            JsonToolsError::BatchProcessingError { index, source, .. } => {
                eprintln!("Item {} failed: {}", index, source);
            }
            _ => eprintln!("{}", e),
        }
    }
}
}

Auto-Conversions

JsonToolsError implements From for common error types:

#![allow(unused)]
fn main() {
// These conversions happen automatically in ? chains:
impl From<json_parser::JsonError> for JsonToolsError { .. }  // -> E001
impl From<regex::Error> for JsonToolsError { .. }            // -> E002
}

See Error Codes for the full error reference.

ProcessingConfig

Low-level configuration struct used internally by JSONTools. You can construct it directly for advanced use cases, but the JSONTools builder is the recommended interface.

ProcessingConfig and its sub-config structs (FilteringConfig, CollisionConfig, ReplacementConfig, TypeConversionConfig, and the four per-category type-conversion configs) are all #[non_exhaustive] -- construct them via ::new() and the fluent setter methods, not a bare struct literal, so new fields can be added in a future release without breaking existing code.

#![allow(unused)]
fn main() {
pub struct ProcessingConfig {
    pub separator: String,
    pub lowercase_keys: bool,
    pub filtering: FilteringConfig,
    pub collision: CollisionConfig,
    pub replacements: ReplacementConfig,
    pub type_conversion: TypeConversionConfig,
    pub parallel_threshold: usize,
    pub num_threads: Option<usize>,
    pub nested_parallel_threshold: usize,
    pub max_array_index: usize,
    // some fields omitted (non_exhaustive)
}
}

Builder Methods

#![allow(unused)]
fn main() {
use json_tools_rs::{
    ProcessingConfig, FilteringConfig, CollisionConfig, ReplacementConfig,
    TypeConversionConfig, NumberConversionConfig,
};

let config = ProcessingConfig::new()
    .separator("::")
    .lowercase_keys(true)
    .filtering(FilteringConfig::new().remove_nulls(true))
    .collision(CollisionConfig::new().handle_collisions(true))
    .replacements(
        ReplacementConfig::new()
            .add_key_replacement("r'^old_'", "new_")
    )
    .type_conversion(
        TypeConversionConfig::new()
            .numbers(NumberConversionConfig::new().enabled(true).currency(false))
    );
}

FilteringConfig

Configuration for value filtering. All fields are pub and can be read directly (e.g. filtering.remove_nulls); the builder methods below exist for fluent construction.

#![allow(unused)]
fn main() {
pub struct FilteringConfig {
    pub remove_empty_strings: bool,
    pub remove_nulls: bool,
    pub remove_empty_objects: bool,
    pub remove_empty_arrays: bool,
}
}

Builder Methods

All methods consume and return Self.

Query Methods

#![allow(unused)]
fn main() {
use json_tools_rs::FilteringConfig;

let filtering = FilteringConfig::new()
    .remove_nulls(true)
    .remove_empty_strings(true);

assert!(filtering.has_any_filter());
assert!(filtering.remove_nulls); // public field, not a getter method
assert!(!filtering.remove_empty_objects);
}

CollisionConfig

Configuration for key collision handling.

#![allow(unused)]
fn main() {
pub struct CollisionConfig {
    pub handle_collisions: bool,
}
}

Builder Methods

Query Methods

#![allow(unused)]
fn main() {
use json_tools_rs::CollisionConfig;

let collision = CollisionConfig::new().handle_collisions(true);
assert!(collision.has_collision_handling());
}

ReplacementConfig

Configuration for key/value replacement and exclusion patterns. Uses SmallVec<[(String, String); 2]>/SmallVec<[String; 2]> internally to avoid heap allocation for the common case of 0-2 patterns.

#![allow(unused)]
fn main() {
pub struct ReplacementConfig {
    pub key_replacements: SmallVec<[(String, String); 2]>,
    pub value_replacements: SmallVec<[(String, String); 2]>,
    pub key_exclusions: SmallVec<[String; 2]>,
    pub value_exclusions: SmallVec<[String; 2]>,
}
}

Builder Methods

Query Methods

#![allow(unused)]
fn main() {
use json_tools_rs::ReplacementConfig;

let replacements = ReplacementConfig::new()
    .add_key_replacement("r'^user_'", "")
    .add_value_replacement("@old.com", "@new.com")
    .add_key_exclusion("crypto")
    .add_value_exclusion("banned");

assert!(replacements.has_key_replacements());
assert!(replacements.has_value_replacements());
assert!(replacements.has_key_exclusions());
assert!(replacements.has_value_exclusions());
}

Python API Reference

import json_tools_rs

JSONTools

The main builder class for all JSON operations. All configuration methods return self for chaining; only .execute() and .execute_to_output() trigger processing.

Construction

tools = json_tools_rs.JSONTools()

Creates a new JSONTools instance with all default settings. The instance is reusable -- you can call .execute() multiple times with different inputs.

Operation Modes

Exactly one mode must be set before calling .execute(). Calling a mode method replaces any previously set mode.

.flatten()

tools.flatten() -> JSONTools

Set the operation to flatten nested JSON into dot-separated (or custom separator) keys.

import json_tools_rs as jt

result = jt.JSONTools().flatten().execute({"a": {"b": {"c": 1}}})
# {"a.b.c": 1}

.unflatten()

tools.unflatten() -> JSONTools

Set the operation to reconstruct nested JSON from flat, separator-delimited keys.

result = jt.JSONTools().unflatten().execute({"a.b.c": 1})
# {"a": {"b": {"c": 1}}}

.normal()

tools.normal() -> JSONTools

Set the operation to apply transformations (filtering, replacements, type conversion) without changing the nesting structure.

result = jt.JSONTools().normal().lowercase_keys(True).execute({"Name": "Alice"})
# {"name": "Alice"}

Configuration Methods

All configuration methods return self for chaining.

.separator(sep)

tools.separator(sep: str) -> JSONTools

Set the key separator for flatten/unflatten operations.

Raises: ValueError if sep is an empty string.

result = jt.JSONTools().flatten().separator("::").execute({"a": {"b": 1}})
# {"a::b": 1}

.lowercase_keys(flag)

tools.lowercase_keys(flag: bool) -> JSONTools

Convert all keys to lowercase after processing.

result = jt.JSONTools().flatten().lowercase_keys(True).execute({"User": {"Name": "Alice"}})
# {"user.name": "Alice"}

.remove_empty_strings(flag)

tools.remove_empty_strings(flag: bool) -> JSONTools

Remove key-value pairs where the value is an empty string "".

result = jt.JSONTools().flatten().remove_empty_strings(True).execute({"a": "", "b": "hello"})
# {"b": "hello"}

.remove_nulls(flag)

tools.remove_nulls(flag: bool) -> JSONTools

Remove key-value pairs where the value is None / null.

result = jt.JSONTools().flatten().remove_nulls(True).execute({"a": None, "b": 1})
# {"b": 1}

.remove_empty_objects(flag)

tools.remove_empty_objects(flag: bool) -> JSONTools

Remove key-value pairs where the value is an empty object {}.

.remove_empty_arrays(flag)

tools.remove_empty_arrays(flag: bool) -> JSONTools

Remove key-value pairs where the value is an empty array [].

.key_replacement(find, replace)

tools.key_replacement(find: str, replace: str) -> JSONTools

Add a key replacement pattern. Patterns are literal (exact substring match) by default; wrap a pattern in r'...' (e.g. "r'^user_'") to use standard regex syntax instead. A malformed r'...' pattern is silently treated as "no match" rather than raising an error. Multiple replacements can be chained.

result = (jt.JSONTools()
    .flatten()
    .key_replacement("r'^user_'", "")
    .key_replacement("r'_name$'", "_id")
    .execute({"user_name": "Alice"}))
# {"id": "Alice"}

.value_replacement(find, replace)

tools.value_replacement(find: str, replace: str) -> JSONTools

Add a value replacement pattern. Works the same as key replacements (literal by default, r'...' for regex) but applies to string values.

result = (jt.JSONTools()
    .flatten()
    .value_replacement("@example.com", "@company.org")
    .execute({"email": "user@example.com"}))
# {"email": "user@company.org"}

.exclude_key(pattern)

tools.exclude_key(pattern: str) -> JSONTools

Drop any key -- and its entire value/subtree -- whose name contains pattern. Literal (exact substring match) by default; wrap in r'...' for regex, matching key_replacement's convention. Additive -- call once per keyword to exclude multiple. Checked against the full dot-path in flatten/unflatten mode, and per key at each nesting level in normal mode; matching a container key drops its entire subtree without walking it. Array elements are never matched (no key name to check).

result = (jt.JSONTools()
    .flatten()
    .exclude_key("crypto")
    .execute({"user": {"name": "John", "crypto_wallet": {"coin": "BTC"}}}))
# {"user.name": "John"}

.exclude_value(pattern)

tools.exclude_value(pattern: str) -> JSONTools

Drop a key-value pair whose value contains pattern. Same literal/r'...' convention as exclude_key. Additive. Only ever applies to scalar leaf values (strings/numbers/booleans/null) -- containers have no single value to check. Checked against the final value after any configured value_replacement/ auto_convert_types have run. A no-op at the document root.

Unflatten-specific note: string values are matched against their JSON-serialized form (including surrounding quotes), not the unescaped logical text. Literal patterns are unaffected; a regex with anchors needs r'^"admin"$' rather than r'^admin$' to match a value that's exactly "admin".

result = (jt.JSONTools()
    .flatten()
    .exclude_value("banned")
    .execute({"user": {"name": "John", "status": "banned"}}))
# {"user.name": "John"}

.handle_key_collision(flag)

tools.handle_key_collision(flag: bool) -> JSONTools

When enabled, keys that would collide after transformations (e.g., after lowercasing) are collected into arrays instead of overwriting each other.

result = (jt.JSONTools()
    .flatten()
    .lowercase_keys(True)
    .handle_key_collision(True)
    .execute({"Name": "Alice", "name": "Bob"}))
# {"name": ["Alice", "Bob"]}

.always_array_keys(keys)

tools.always_array_keys(keys: Sequence[str]) -> JSONTools

Flattened key names that must always render as a JSON array, even when only one value is present in a given document -- keeps a key's scalar-vs-array shape consistent across every document/row of a batch, not just documents where a collision happened to occur. Independent of .handle_key_collision(): a key named here always gets full array treatment. See Key Collision Handling for the full explanation, including why this matters for normalise()'s List<T> column typing.

result = (jt.JSONTools()
    .flatten()
    .key_replacement("r'(User|Admin)_'", "")
    .always_array_keys(["name"])
    .execute({"User_name": "John"}))
# {"name": ["John"]}  -- wrapped even though nothing collided here

.auto_convert_types(flag)

tools.auto_convert_types(flag: bool) -> JSONTools

Automatically convert string values to their native types:

• Numbers: "123" -> 123, "1,234.56" -> 1234.56, "$99.99" -> 99.99, "1e5" -> 100000
• Booleans: "true" / "TRUE" / "True" -> true, "false" / "FALSE" / "False" -> false
• Nulls: "null" / "None" -> null

If conversion fails, the original string is kept. No errors are raised on conversion failure.

result = (jt.JSONTools()
    .flatten()
    .auto_convert_types(True)
    .execute({"id": "123", "price": "1,234.56", "active": "true"}))
# {"id": 123, "price": 1234.56, "active": true}

.convert_dates(enable, normalize_to_utc=None, assume_utc_for_naive=None) / .convert_nulls(enable, extra_tokens=None) / .convert_booleans(enable, extra_true_tokens=None, extra_false_tokens=None) / .convert_numbers(enable, currency=None, percent=None, basis_points=None, suffixes=None, fractions=None, radix=None)

tools.convert_dates(enable: bool, normalize_to_utc: bool | None = None, assume_utc_for_naive: bool | None = None) -> JSONTools
tools.convert_nulls(enable: bool, extra_tokens: list[str] | None = None) -> JSONTools
tools.convert_booleans(enable: bool, extra_true_tokens: list[str] | None = None, extra_false_tokens: list[str] | None = None) -> JSONTools
tools.convert_numbers(enable: bool, currency: bool | None = None, percent: bool | None = None, basis_points: bool | None = None, suffixes: bool | None = None, fractions: bool | None = None, radix: bool | None = None) -> JSONTools

Independent, per-category alternative to .auto_convert_types(): enable/customize dates, nulls, booleans, and numbers separately instead of all-or-nothing. auto_convert_types(True) only flips each category's on/off switch and preserves customization already set via these methods -- call order doesn't reset it. A kwarg left as None on a later call also preserves whatever a previous call set (it's not reset to the built-in default).

Plain integers/decimals, scientific notation, and thousands-separator cleanup are always applied when convert_numbers is enabled, regardless of the other kwargs.

result = (jt.JSONTools()
    .flatten()
    .convert_dates(True, assume_utc_for_naive=False)
    .convert_nulls(True, extra_tokens=["missing"])
    .execute({"d": "2024-01-15T10:30:00", "a": "missing"}))
# {"d": "2024-01-15T10:30:00", "a": None}

.parallel_threshold(n)

tools.parallel_threshold(n: int) -> JSONTools

Set the minimum batch size to trigger parallel processing. Batches smaller than this are processed sequentially to avoid thread-spawning overhead.

Default can be overridden with the JSON_TOOLS_PARALLEL_THRESHOLD environment variable.

tools = jt.JSONTools().flatten().parallel_threshold(50)

.num_threads(n)

tools.num_threads(n: int | None) -> JSONTools

Set the number of threads used for parallel processing. Pass None (or omit the call) to use the system default.

Default can be overridden with the JSON_TOOLS_NUM_THREADS environment variable.

tools = jt.JSONTools().flatten().num_threads(4)

.nested_parallel_threshold(n)

tools.nested_parallel_threshold(n: int) -> JSONTools

Set the minimum number of keys/items within a single JSON document to trigger nested (intra-document) parallelism. Only objects or arrays exceeding this count are parallelized internally.

Default can be overridden with the JSON_TOOLS_NESTED_PARALLEL_THRESHOLD environment variable.

tools = jt.JSONTools().flatten().nested_parallel_threshold(200)

.max_array_index(n)

tools.max_array_index(n: int) -> JSONTools

Set the maximum array index allowed during unflattening. This is a DoS protection: a malicious key like "items.999999999" would otherwise allocate a massive array.

Default can be overridden with the JSON_TOOLS_MAX_ARRAY_INDEX environment variable.

Execution Methods

.execute(input, normalise=False, target=None)

tools.execute(input) -> str | dict | list[str] | list[dict] | DataFrame | Series
tools.execute(input, normalise=True, target=None) -> DataFrame

Execute the configured operation. By default (normalise=False) the return type mirrors the input type:

Raises: JsonToolsError if no mode is set, input is invalid, processing fails, or (PySpark input) no active SparkSession is found.

# String input -> string output
result = jt.JSONTools().flatten().execute('{"a": {"b": 1}}')
assert isinstance(result, str)

# Dict input -> dict output
result = jt.JSONTools().flatten().execute({"a": {"b": 1}})
assert isinstance(result, dict)

# Batch string input -> batch string output
results = jt.JSONTools().flatten().execute(['{"a": 1}', '{"b": 2}'])
assert isinstance(results, list) and isinstance(results[0], str)

# Batch dict input -> batch dict output
results = jt.JSONTools().flatten().execute([{"a": {"b": 1}}, {"c": {"d": 2}}])
assert isinstance(results, list) and isinstance(results[0], dict)

normalise / target: always get back a wide DataFrame

normalise=True bypasses the input-mirroring table above entirely: regardless of input's shape, the result is always a wide DataFrame (one column per flattened key) -- a bare str/dict becomes a 1-row DataFrame. Requires .flatten() mode. Reconstruction builds one real Arrow table internally with genuinely typed columns (including real List<T> for handle_key_collision(True)), then derives whichever target was requested from it -- see DataFrame & Series Support's "Arrow-native reconstruction" callout for the full behavior (key union/null-fill order, target auto-resolution, the pandas dtype change, the PySpark path) and examples.

tools = jt.JSONTools().flatten()

df = tools.execute({"user": {"name": "Alice"}}, normalise=True)          # auto-resolved target
df = tools.execute([{"a": 1}, {"a": 2}], normalise=True, target="polars")

Additional raises (when normalise=True or target is set): mode is not .flatten(); target is set while normalise=False; target names an unknown or uninstalled library; target="pandas"/"pyspark" without pyarrow installed (both require it internally now -- target="polars" does not); or (target="pyspark") no active SparkSession is found.

.execute_to_output(input)

tools.execute_to_output(input) -> JsonOutput

Execute the operation but return a JsonOutput wrapper instead of native Python types. Useful when you need to inspect whether the result is single or multiple before extracting.

Note: DataFrame and Series inputs are not supported with execute_to_output(). Use .execute() for those types.

output = jt.JSONTools().flatten().execute_to_output('{"a": {"b": 1}}')
if output.is_single:
    print(output.get_single())
elif output.is_multiple:
    for item in output.get_multiple():
        print(item)

Pickling and to_config_json() / from_config_json()

JSONTools instances are picklable (pickle.dumps/pickle.loads), which means a configured instance can also be captured in a closure that crosses a real process boundary via cloudpickle -- most notably inside a PySpark UDF or mapInPandas function, without needing a workaround.

tools = jt.JSONTools().flatten().remove_nulls(True)

config = tools.to_config_json()          # -> str
restored = jt.JSONTools.from_config_json(config)  # a fresh, independent instance

to_config_json()/from_config_json() are the mechanism pickling is built on top of, and are directly useful on their own for the same reason: a mapInPandas partition function should close over the config string (not the JSONTools instance itself) and call from_config_json() once inside each partition to get a working, independent instance -- this is exactly how the pickle support works internally (__reduce__ returns (from_config_json, (config_json,))).

JsonOutput

Output wrapper returned by .execute_to_output(). Provides typed access to results.

Properties

Methods

.get_single()

output.get_single() -> str

Extract the single JSON string result.

Raises: ValueError if the result is multiple.

.get_multiple()

output.get_multiple() -> list[str]

Extract the list of JSON string results.

Raises: ValueError if the result is single.

.to_python()

output.to_python() -> str | list[str]

Convert to native Python type: returns str for single results, list[str] for multiple results.

String Representations

str(output) returns the JSON string (single) or a list representation (multiple). repr(output) returns JsonOutput.Single('...') or JsonOutput.Multiple([...]).

DataFrame and Series Support

JSON Tools RS natively supports Pandas, Polars, PyArrow, and PySpark DataFrames and Series. Detection is performed via duck typing -- no explicit imports are required.

Performance note: in .flatten() mode, a DataFrame with no nested columns to flatten automatically takes a faster internal path that skips JSON serialization entirely -- transparent, no flag to set, same output either way. See Performance: The Flat-DataFrame Fast Path.

Pandas DataFrame

Each row is serialized to a JSON object (column names become keys) and processed as a whole document -- so flattening finds nested structure in columns holding actual nested Python objects (dicts/lists) directly. In .flatten() mode, columns holding pre-serialized JSON-text strings are also detected and expanded the same way (auto-detected, not requiring the column to already be dict/list-typed) -- see Auto-Expanding JSON-String Columns for the detection rules. The source column's own name is never kept as a prefix in the output -- only nesting within the column's own content is (see that same section). .unflatten()/.normal() mode leave a JSON-string column's value untouched, as a plain string scalar.

import pandas as pd
import json_tools_rs as jt

df = pd.DataFrame({"user": [
    {"name": "Alice", "age": 30},
    {"name": "Bob", "age": 25},
]})

tools = jt.JSONTools().flatten().separator(".")

# Each row -> {"user": {"name": ..., "age": ...}} -> flattened, "user" itself
# dropped (it's the column name, not part of the payload)
result_df = tools.execute(df)
# Returns a DataFrame with flattened columns: "name", "age"

Pandas Series

series = pd.Series([
    '{"a": {"b": 1}}',
    '{"a": {"b": 2}}',
])

result_series = jt.JSONTools().flatten().execute(series)
# Returns a Series of flattened JSON strings

Polars DataFrame

Like Pandas, this flattens a column of nested Struct values -- a column of JSON-text strings round-trips unchanged, since there's no nested structure inside a string scalar for .flatten() to find.

import polars as pl

df = pl.DataFrame({
    "user": [{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}]
})

result_df = jt.JSONTools().flatten().execute(df)
# result_df columns: ["name", "age"]

Polars Series

series = pl.Series("data", [
    '{"a": {"b": 1}}',
    '{"a": {"b": 2}}',
])

result_series = jt.JSONTools().flatten().execute(series)

PyArrow Table

Same rule as Pandas/Polars: flatten a struct-typed column, not a plain string column holding JSON text.

import pyarrow as pa

table = pa.table({
    "user": pa.array([{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}])
})

result_table = jt.JSONTools().flatten().execute(table)
# result_table columns: ["name", "age"]

PySpark DataFrame

.execute(df) collects the DataFrame to the driver via toPandas(), runs it through the same row-is-a-JSON-object pipeline as Pandas (so a StructType column flattens; a plain string column does not, unless it holds JSON -- see Auto-Expanding JSON-String Columns), then reconstructs a genuine, distributed pyspark.sql.DataFrame via the active SparkSession (#31; auto-discovered via SparkSession.getActiveSession(), raising JsonToolsError if none is found).

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([
    {"name": "Alice", "age": 30},
    {"name": "Bob", "age": 25},
])

result = jt.JSONTools().flatten().execute(df)
print(type(result))  # <class 'pyspark.sql.dataframe.DataFrame'>

Nested StructType columns and .toPandas(): a top-level flat column always flattens correctly, as above. A nested struct column usually flattens correctly too, but its exact behavior through toPandas()'s non-Arrow fallback path (taken automatically when pyarrow isn't installed) can differ from the Arrow-optimized path in edge cases -- e.g. a Row-typed nested field has been observed losing its field names on that fallback path (surfacing as positional 0/1 instead of name/age), a PySpark toPandas() characteristic, not something this library controls. Installing pyarrow avoids the fallback path entirely and is recommended for any nested-struct-heavy workload.

JsonToolsError

Exception class for all errors raised by JSON Tools RS.

import json_tools_rs as jt

try:
    result = jt.JSONTools().flatten().execute("not valid json")
except jt.JsonToolsError as e:
    print(f"Error: {e}")
    # Error: Failed to process JSON string: [E001] JSON parsing failed: ...

Error messages embed a machine-readable code (E001-E008) in square brackets. Note that the Python bindings prepend their own context before the underlying Rust message (e.g. "Failed to process JSON string: ", "Failed to process Python dict: "), so the code is not always the very first characters of str(e) -- check for "[E00x]" as a substring rather than a prefix. See Error Codes for the full reference.

Error Codes Quick Reference

Handling Specific Errors

import json_tools_rs as jt

try:
    result = jt.JSONTools().execute({"a": 1})  # No mode set
except jt.JsonToolsError as e:
    msg = str(e)
    if "[E005]" in msg:
        print("Forgot to call .flatten() or .unflatten()")
    elif "[E001]" in msg:
        print("Invalid JSON input")

Complete Example

import json_tools_rs as jt

# Build once, reuse many times
tools = (jt.JSONTools()
    .flatten()
    .separator("::")
    .lowercase_keys(True)
    .remove_nulls(True)
    .remove_empty_strings(True)
    .key_replacement("r'^user_'", "")
    .auto_convert_types(True)
    .parallel_threshold(50)
    .num_threads(4)
)

# Single dict
result = tools.execute({"User_Name": "Alice", "User_Age": "30"})
# {"name": "Alice", "age": 30}

# Batch of dicts (processed in parallel if >= 50 items)
results = tools.execute([{"data": str(i)} for i in range(1000)])

# JSON string
result = tools.execute('{"User_Name": "Alice", "nested": {"User_Age": "30"}}')

# DataFrame -- each row becomes {"User_Name": ..., "User_Age": ...} (column names
# are the JSON keys); see "DataFrame and Series Support" above for why a column of
# nested dict/struct values flattens but a column of JSON-text strings does not
import pandas as pd
df = pd.DataFrame({
    "User_Name": ["Alice", "Bob"],
    "User_Age": ["30", "25"],
})
df_result = tools.execute(df)

Architecture

JSON Tools RS is organized into focused, single-responsibility modules. This modular design improves maintainability while preserving performance -- Rust modules are compile-time organization only, with zero runtime overhead.

Module Structure

src/
├── lib.rs            Facade: mod declarations + pub use re-exports
├── json_parser.rs    Conditional SIMD parser (sonic-rs / simd-json)
├── types.rs          Core types: JsonInput, JsonOutput
├── error.rs          Error types with codes E001-E008
├── config.rs         Configuration structs and operation modes
├── cache.rs          Multi-tier regex pattern cache (compile-time table, sticky,
│                     thread-local, global)
├── fxhash.rs         Custom FxHash-style Hasher/BuildHasher for FxHashMap/FxIndexMap
├── convert.rs        Type conversion: numbers, dates, booleans, nulls
├── transform.rs      Filtering, key/value replacements, collision handling
├── flatten.rs        Tape-based flattening engine (scan -> walk -> output)
├── unflatten.rs      Tape-based unflattening with SIMD separator detection
├── builder.rs        Public JSONTools builder API and execute()
├── python.rs         Python bindings via PyO3
├── tests.rs          Unit tests
└── main.rs           CLI examples

Module Descriptions

json_parser -- JSON Parsing Abstraction

Conditional compilation wrapper that selects the fastest available JSON parser:

• 64-bit platforms: sonic-rs (AVX2/SSE4.2 SIMD, 30-50% faster)
• 32-bit platforms: simd-json (fallback)

Exposes from_str(), to_string(), and parse_json() with a unified JsonError type.

types -- Core Types

Defines the public-facing input/output types:

• JsonInput<'a> -- Enum accepting &str, &[&str], Vec<String>, etc.
• JsonOutput -- Enum returning Single(String) or Multiple(Vec<String>)

error -- Error Handling

JsonToolsError enum with 8 error variants (E001-E008), each with machine-readable codes, Display/Error impls, and constructors. Includes From impls for automatic conversion from parse and regex errors.

config -- Configuration

All configuration structs used by the builder:

• ProcessingConfig -- Main config holding all options
• FilteringConfig -- Empty string/null/object/array removal
• CollisionConfig -- Key collision handling settings
• ReplacementConfig -- Key and value replacement patterns
• OperationMode -- Flatten, Unflatten, or Normal

cache -- Regex Pattern Caching

Multi-tier cache for compiled regex patterns used by key_replacement()/value_replacement() (not a general key-deduplication cache -- there is no phf-based key cache or KeyDeduplicator in the current codebase; phf is not a dependency of this crate):

1. COMMON_REGEX_PATTERNS -- a LazyLock<FxHashMap<...>> of ~60 pre-compiled common patterns (whitespace, UUIDs, dates, user_/admin_ prefixes, etc.), checked first
2. STICKY_REGEX_CACHE -- a tiny thread-local linear-scan cache (capacity 4) of the most recently used patterns, added to short-circuit the hashing/locking below for the common case of the same 1-2 patterns reused across an entire batch
3. THREAD_LOCAL_REGEX_CACHE -- a larger thread-local FxHashMap (capacity 128)
4. REGEX_CACHE -- a global RwLock<FxHashMap<Arc<str>, (Arc<Regex>, AtomicU64)>> (capacity 512)

Tiers 2-4 track per-entry "last used" ticks and evict the genuinely least-recently-used entry when full (not an arbitrary one).

fxhash -- Fast Hashing

Hand-rolled FxHasher/FxBuildHasher (the same algorithm popularized by rustc-hash, reimplemented in-tree rather than taken as a dependency) backing the FxHashMap/FxIndexMap type aliases used throughout cache, flatten, and unflatten.

convert -- Type Conversion

Automatic type conversion for string values (~1,200 lines, the largest leaf module):

• Number parsing: integers, decimals, currency, percentages, basis points, scientific notation, suffixed (K/M/B)
• Date parsing: ISO-8601 variants with UTC normalization
• Boolean/null detection via direct string matching (try_parse_bool(), is_null_string()) -- not a phf perfect hash map; phf is not a dependency of this crate
• SIMD-optimized clean_number_string() with extend_skipping_3/4 helpers

transform -- Transformations

Core transformation logic applied after flatten/unflatten:

• Key/value replacements (literal and regex, with SIMD fast-path)
• Filtering (empty strings, nulls, empty objects/arrays)
• Key collision handling (collect into arrays)
• Lowercase key conversion

flatten -- Flattening Algorithm

Tape-based engine (scan -> walk -> output), not a naive recursive serde_json::Value walk:

• scan_and_fixup() -- single-pass structural scanner producing a TapeEntry tape (merges structural scan, validation, container pairing, and string-length computation), with a byte-classification lookup table instead of multiple memchr calls
• SeparatorCache for pre-computed separator properties (single-byte fast path vs. multi-byte)
• Zero-copy ValueRef::Raw byte ranges into the original input, avoiding serde_json::Value tree allocation, with a direct-to-output fast path when no key transforms/collision handling are configured
• flatten_collecting_parallel() for Rayon-parallel flattening of large objects/arrays once a document crosses nested_parallel_threshold
• Arena-allocated (bumpalo::Bump) key storage on the slow path (key lowercasing/replacement/collision-handling), avoiding one heap allocation per dotted key path

unflatten -- Unflattening Algorithm

Reconstructs nested JSON from flat key-value pairs using the same tape scanner as flatten:

• SIMD-accelerated separator detection (find_separator()/find_separator_offsets())
• Path type analysis for array vs. object reconstruction
• Recursive set_nested_value()/set_nested_value_recursive() and set_nested_array_value()
• FxIndexMap-backed object tree (insertion-ordered, O(1) lookup) instead of a hash map + full key sort

builder -- Public API

The JSONTools struct and its ~19 public methods (3 mode setters, 14 configuration methods, plus new() and execute()). Routes execute() calls to the appropriate processing function based on operation mode (flatten, unflatten, normal).

python -- Python Bindings

PyO3-based Python bindings with:

• Perfect type preservation (input type = output type)
• Native DataFrame/Series support (Pandas, Polars, PyArrow, PySpark)
• GIL release during compute-intensive operations

Processing Pipeline

Input → Parse → Flatten/Unflatten → Transform → Filter → Convert → Serialize → Output
         │            │                  │          │         │          │
    json_parser    flatten/         transform   transform   convert   json_parser
                   unflatten

For a single JSON document, flatten/unflatten/normal all share one tape scanner (scan_and_fixup(), defined in flatten.rs, imported by unflatten.rs and transform.rs) and walk it directly to output. json_parser's conditional SIMD parser (sonic-rs/simd-json) is reserved for a narrower case: a root-level JSON primitive (e.g. a bare "hello" or 42, not an object/array), which falls back to a serde_json::Value round-trip in both flatten and unflatten.

Public API Surface

All public types are re-exported from lib.rs, preserving a flat import path:

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonInput, JsonOutput, JsonToolsError};
use json_tools_rs::{ProcessingConfig, FilteringConfig, CollisionConfig, ReplacementConfig};
}

Internal modules use pub(crate) visibility for cross-module access without exposing internals.

Error Codes

All errors include a machine-readable code accessible via .error_code() (Rust) or in the error message (Python).

Rust Error Handling

#![allow(unused)]
fn main() {
use json_tools_rs::{JSONTools, JsonToolsError};

match JSONTools::new().flatten().execute("not valid json") {
    Ok(result) => { /* success */ }
    Err(e) => {
        // e.error_code() -> "E001" (bare code, for match arms / logging fields)
        // format!("{e}")  -> "[E001] JSON parsing failed: Invalid literal (`true`,
        //                    `false`, or a `null`) while parsing at line 1 column 4
        //                    ...
        //                    💡 Suggestion: Verify your JSON syntax using a JSON
        //                    validator. ..." (Display already includes the bracketed code)
        eprintln!("{e}");
    }
}
}

Python Error Handling

import json_tools_rs as jt

try:
    result = jt.JSONTools().flatten().execute("not valid json")
except jt.JsonToolsError as e:
    print(f"Error: {e}")
    # Error: Failed to process JSON string: [E001] JSON parsing failed: ...

The Python bindings prepend their own context (e.g. "Failed to process JSON string: ") before the underlying Rust message, so the [E00x] code is embedded in str(e) but not necessarily its first characters -- match it as a substring ("[E001]" in str(e)), not a prefix.

Common Errors

E005: No mode set

# Wrong: no mode set
tools = jt.JSONTools().execute(data)  # Raises E005

# Correct: set a mode first
tools = jt.JSONTools().flatten().execute(data)

E001: Invalid JSON

# Wrong: not valid JSON
tools = jt.JSONTools().flatten().execute("hello world")  # Raises E001

# Correct: valid JSON string
tools = jt.JSONTools().flatten().execute('{"key": "value"}')

Performance & Benchmarks

JSON Tools RS achieves ~2,000+ ops/ms through multiple optimization layers.

Optimization Techniques

Benchmark Results

Measured on Apple Silicon (M4) via cargo bench --bench stress_benchmarks -- --quick against the current source tree -- a quick/low-sample Criterion run, so treat these as indicative rather than lab-precise; re-run the suite yourself (see Running Benchmarks below) for reproducible numbers on your own hardware.

Stress Benchmarks

Throughput Targets (v0.9.0)

Performance Tuning

Three threshold parameters control when parallelism activates. Tuning them for your workload can significantly affect throughput.

parallel_threshold (default: 100)

Controls when batch processing (multiple JSON documents) switches from sequential to parallel execution.

When to lower (e.g., 20-50):

• Each document is large or complex (deep nesting, many keys)
• CPU cores are available and not contended
• You are processing 50-100 items and want parallel speedup

When to raise (e.g., 200-500):

• Each document is small (a few keys, shallow nesting)
• Thread-spawning overhead dominates processing time
• Running inside a container with limited CPU

# For large documents, parallel even at small batch sizes
tools = jt.JSONTools().flatten().parallel_threshold(20)

# For tiny documents, avoid parallelism overhead
tools = jt.JSONTools().flatten().parallel_threshold(500)

#![allow(unused)]
fn main() {
let tools = JSONTools::new()
    .flatten()
    .parallel_threshold(50);
}

nested_parallel_threshold (default: 100)

Controls when a single JSON document's top-level keys/array items are processed in parallel (intra-document parallelism). This is independent of batch parallelism.

When to lower (e.g., 50):

• Individual documents have very wide objects (500+ keys) with deep sub-trees
• Processing includes expensive transformations (regex replacements, type conversion)

When to raise (e.g., 500-1000) or effectively disable:

• Documents are moderately sized (under 100 keys)
• Sub-trees are shallow (1-2 levels), so per-key work is minimal
• You want deterministic (sequential) output ordering

# Large documents with heavy per-key work
tools = jt.JSONTools().flatten().nested_parallel_threshold(50)

# Disable nested parallelism entirely
tools = jt.JSONTools().flatten().nested_parallel_threshold(999_999)

num_threads (default: CPU count)

Controls the number of worker threads for parallel processing.

When to set explicitly:

• Running alongside other CPU-intensive workloads -- limit threads to avoid contention
• In a container or VM with a CPU quota -- match thread count to available cores
• Benchmarking -- fix thread count for reproducible results

tools = jt.JSONTools().flatten().num_threads(4)

#![allow(unused)]
fn main() {
let tools = JSONTools::new()
    .flatten()
    .num_threads(Some(4));
}

Environment Variable Overrides

All threshold defaults can be overridden without code changes via environment variables. These are read once at process startup (via LazyLock).

# Example: tune for a workload of many small documents
export JSON_TOOLS_PARALLEL_THRESHOLD=200
export JSON_TOOLS_NUM_THREADS=8

python my_pipeline.py

Environment variable values are parsed as usize. Invalid values (non-numeric, negative) silently fall back to the default.

Running Benchmarks

# All benchmarks
cargo bench

# Specific suite
cargo bench --bench isolation_benchmarks
cargo bench --bench comprehensive_benchmark
cargo bench --bench stress_benchmarks
cargo bench --bench realworld_benchmarks
cargo bench --bench combination_benchmarks

Benchmark Suites

Profiling

On macOS, use samply for profiling:

# Build with profiling symbols
cargo bench --profile profiling --bench stress_benchmarks --no-run

# Profile with samply
samply record --save-only -o /tmp/profile.json -- \
    ./target/profiling/deps/stress_benchmarks-* --bench

# View results
samply load /tmp/profile.json

Architecture

The codebase is organized into focused, single-responsibility modules (see Architecture for the full breakdown):

src/
├── lib.rs            Facade: mod declarations + pub use re-exports
├── json_parser.rs    Conditional SIMD parser (sonic-rs / simd-json) -- used for the
│                     root-primitive fallback, not the main tape-based paths
├── types.rs          Core types: JsonInput, JsonOutput
├── error.rs          Error types with codes E001-E008
├── config.rs         Configuration structs and operation modes
├── cache.rs          Multi-tier regex pattern cache (common-pattern table, sticky,
│                     thread-local, global RwLock)
├── fxhash.rs         Custom FxHash-style Hasher for FxHashMap/FxIndexMap
├── convert.rs        Type conversion: numbers, dates, booleans, nulls
├── transform.rs      Filtering, key/value replacements, collision handling
├── flatten.rs        Tape-based flattening engine (scan -> walk -> output)
├── unflatten.rs      Tape-based unflattening with SIMD separator detection
├── builder.rs        Public JSONTools builder API and execute()
├── python.rs         Python bindings via PyO3
├── tests.rs          Unit tests
└── main.rs           CLI examples

The processing pipeline:

1. Parse -- single-pass tape scan (scan_and_fixup(), shared by flatten/unflatten/transform); json_parser's SIMD parser only handles the root-primitive edge case
2. Flatten/Unflatten -- tape walk with CompactString-inlined keys (and an arena allocator for the slow path involving key transforms) (flatten/unflatten)
3. Transform -- Lowercase, replacements (cached regex), collision handling (transform)
4. Filter -- Remove empty strings, nulls, empty objects/arrays (transform)
5. Convert -- Type conversion with first-byte discriminators (convert)
6. Serialize -- Output to JSON string or native Python types

Troubleshooting

This guide covers common errors, their causes, and how to resolve them.

Error Code Reference

All errors embed a machine-readable code (E001-E008) in square brackets in the error message. Use these codes for programmatic error handling. In Rust, Display on JsonToolsError always starts with the bracketed code (e.g. [E001] ...). In Python, the bindings prepend their own context (e.g. "Failed to process JSON string: ") before the underlying message, so the code is embedded in str(e) but is not always its first characters -- match "[E00x]" as a substring, not a prefix.

E001: JsonParseError

Message: [E001] JSON parsing failed: ...

Cause: The input string is not valid JSON.

Common triggers:

• Missing quotes around keys or values
• Trailing commas after the last element
• Single quotes instead of double quotes
• Unescaped special characters in strings
• Incomplete JSON (missing closing braces or brackets)
• Passing a file path instead of the file contents

Solution:

# Wrong
result = tools.execute("hello world")          # Not JSON
result = tools.execute("{'key': 'value'}")     # Single quotes
result = tools.execute('{"a": 1,}')            # Trailing comma

# Correct
result = tools.execute('{"key": "value"}')
result = tools.execute({"key": "value"})       # Pass a dict directly

E002: RegexError

Message: [E002] Regex pattern error: ...

Cause: Reserved for regex compilation failures. In practice this code is not currently reachable through .key_replacement() / .value_replacement(): patterns are literal by default (exact substring match, no regex engine involved at all), and a pattern explicitly wrapped in r'...' that fails to compile as regex is silently treated as "no match" rather than raised as an error -- so a broken r'...' pattern won't crash your pipeline, it just won't replace anything. E002 is kept in the error enum for API completeness/forward compatibility.

Common mistake this code used to cover, now handled differently:

# "user.name" is now a LITERAL pattern -- the dot is not a wildcard.
# It only matches the exact substring "user.name", not "username".
tools.key_replacement("user.name", "id")

# To use regex (e.g. so the dot matches any character), wrap in r'...':
tools.key_replacement("r'user.name'", "id")

# To match the literal dot as regex, escape it inside the wrapper:
tools.key_replacement(r"r'user\.name'", "id")

See Key & Value Replacements for the full r'...' convention.

E003: InvalidReplacementPattern

Message: [E003] Invalid replacement pattern: ...

Cause: Reserved for a malformed replacement pattern configuration. Like E002, this code is not currently constructed anywhere in the codebase -- .key_replacement() / .value_replacement() always take exactly two arguments (find, replace), so there's no "wrong number of arguments in a pattern list" case to detect. It's kept in the error enum for API completeness/forward compatibility.

# Both key_replacement and value_replacement always take exactly (find, replace)
tools.key_replacement("find_pattern", "replacement")
tools.value_replacement("old_value", "new_value")

E004: InvalidJsonStructure

Message: [E004] Invalid JSON structure: ...

Cause: The set of flat keys given to .unflatten() describes a structurally inconsistent tree -- most commonly, one key is a strict prefix of another but already holds a scalar value, so the longer key can't navigate "into" it.

Two inputs that look like they should trigger this but don't: a root-level JSON array passed to .unflatten() is a dedicated early-exit case that returns "{}" instead of erroring, and a value that's itself a nested object (e.g. {"a": {"b": 1}}) is accepted as-is -- .unflatten() only requires keys to be splittable on the separator; it doesn't require values to be scalar.

Solution:

# Wrong -- "a" is set to a scalar (1), then "a.b" tries to navigate into it as an object
result = jt.JSONTools().unflatten().execute('{"a": 1, "a.b": 2}')
# Raises: [E004] Invalid JSON structure: Cannot navigate into non-object/non-array
# value at key: a

# Correct -- keys don't collide on structure
result = jt.JSONTools().unflatten().execute('{"a.b": 1, "a.c": 2}')

E005: ConfigurationError

Message: [E005] Operation mode not configured: ...

Cause: .execute() was called without first setting an operation mode.

Solution: Always call .flatten(), .unflatten(), or .normal() before .execute():

# Wrong
result = jt.JSONTools().execute(data)

# Correct
result = jt.JSONTools().flatten().execute(data)
result = jt.JSONTools().unflatten().execute(data)
result = jt.JSONTools().normal().execute(data)

This error also occurs if num_threads is set to 0 -- note the check happens at .execute() time, not when .num_threads(0) is called (the message text is the shared ConfigurationError template, "Operation mode not configured: ...", even though the actual problem is the thread count, not a missing mode):

# Wrong -- raises E005 when .execute() runs, even though a mode was set
tools = jt.JSONTools().flatten().num_threads(0)
tools.execute(data)

# Correct
tools = jt.JSONTools().flatten().num_threads(1)    # At least 1
tools = jt.JSONTools().flatten()                    # Use default (CPU count)

E006: BatchProcessingError

Message: [E006] Batch processing failed at index {N}: Failed to process item at index {N}

Cause: One or more items in a batch failed to process. The Rust core wraps the failing item's original error as the source of a BatchProcessingError -- but the printed message only ever says "Failed to process item at index N", not the specific underlying reason (e.g. the E001 parse error text). In Rust, you can recover the specific cause by pattern-matching the source field (see the Rust API error-handling example); the Python bindings don't expose source, so str(e) alone won't tell you why that item failed.

Solution: Use the reported index to isolate and re-run just that item, so its own error (not the generic batch wrapper) surfaces:

try:
    results = tools.execute(batch_of_json)
except jt.JsonToolsError as e:
    msg = str(e)
    if "[E006]" in msg:
        # The message only gives you the index, e.g. "...at index 1: Failed to
        # process item at index 1" -- re-run that single item to see its real cause.
        for i, item in enumerate(batch_of_json):
            try:
                tools.execute(item)
            except jt.JsonToolsError as item_err:
                print(f"Item {i} failed: {item_err}")

E007: InputValidationError

Message: [E007] Input validation failed: ...

Cause: Raised by the Rust core itself (not the Python binding layer) for a handful of specific conditions:

• Empty JSON input (an empty or all-whitespace string passed to .flatten() -- .unflatten() treats this as {} instead, not an error)
• Input exceeding the 4 GiB size limit
• An array index during .unflatten() that exceeds max_array_index() (e.g. a flattened key like "items.999999999")

Solution:

# Wrong -- empty input to flatten
result = jt.JSONTools().flatten().execute("")  # Raises E007

# Wrong -- array index beyond max_array_index (default 100,000)
result = jt.JSONTools().unflatten().execute('{"items.999999999": 1}')  # Raises E007

# Correct -- either supply valid data or raise the limit
result = jt.JSONTools().unflatten().max_array_index(10_000_000).execute('{"items.999999999": 1}')

Not the same as an unsupported Python input type. Passing a type the Python bindings don't recognize at all (an int/float/bool directly, a list containing something other than strings/dicts, or a DataFrame/Series to execute_to_output()) raises a plain Python ValueError at the binding layer, not jt.JsonToolsError -- jt.JsonToolsError does not subclass ValueError, so except jt.JsonToolsError will not catch it:

# Wrong -- these all raise plain ValueError, not jt.JsonToolsError
result = tools.execute(42)
result = tools.execute([1, 2, 3])
output = tools.execute_to_output(some_dataframe)

# Correct
result = tools.execute('{"value": 42}')
result = tools.execute({"value": 42})
result = tools.execute(['{"a": 1}', '{"b": 2}'])
result = tools.execute(some_dataframe)  # use execute(), not execute_to_output(), for DataFrames

E008: SerializationError

Message: [E008] JSON serialization failed: ...

Cause: The processed result could not be serialized back to JSON. This is typically an internal error.

Solution: If you encounter this error, please report it as a bug. As a workaround, check that your input does not contain unusual Unicode sequences or extremely large numbers that may not round-trip through JSON.

Common Issues

Empty Separator

The separator must be a non-empty string. Using an empty separator is always a logic error -- it would make keys ambiguous.

# This raises an error
tools = jt.JSONTools().flatten().separator("")

# Use any non-empty string
tools = jt.JSONTools().flatten().separator(".")
tools = jt.JSONTools().flatten().separator("::")
tools = jt.JSONTools().flatten().separator("/")

In Rust, .separator("") itself doesn't fail -- the check happens at .execute() time, which returns Err(JsonToolsError::ConfigurationError) (E005), not a panic. In Python, .separator("") raises a ValueError immediately, at builder-call time.

Missing Operation Mode

The most common mistake is forgetting to set a mode:

# This always raises E005
tools = jt.JSONTools()
tools.execute(data)  # Error!

# Set a mode first
tools = jt.JSONTools().flatten()
tools.execute(data)  # OK

Dict vs String Input

Both str and dict inputs are accepted, but the output type mirrors the input type:

# String in -> string out
result = tools.execute('{"a": {"b": 1}}')
assert isinstance(result, str)
# result == '{"a.b":1}'

# Dict in -> dict out
result = tools.execute({"a": {"b": 1}})
assert isinstance(result, dict)
# result == {"a.b": 1}

If you need the raw JSON string output from a dict input, use .execute_to_output():

output = tools.execute_to_output({"a": {"b": 1}})
json_str = output.get_single()  # Returns a JSON string

Literal vs. Regex Patterns in Replacements

Replacement patterns are literal (exact substring match) by default. Wrap a pattern in r'...' to use standard regex syntax instead:

# Literal: matches the exact substring "user_" anywhere in the key
tools.key_replacement("user_", "")

# Regex: anchors, character classes, etc. only work inside r'...'
tools.key_replacement("r'^user_'", "")       # Only at start of key
tools.key_replacement("r'_suffix$'", "")     # Only at end of key
tools.key_replacement("r'user.name'", "id")  # Dot matches any character

# A bare pattern with regex metacharacters is still literal --
# this looks for the exact substring "user.name", not "username"
tools.key_replacement("user.name", "id")

A malformed pattern inside r'...' is silently ignored (no match, no error) rather than raising E002 -- see above.

Performance Tuning

When Parallelism Helps

Parallel processing adds overhead for thread spawning and synchronization. It helps when:

• Batch size is large (100+ items by default) -- amortizes spawning cost
• Individual documents are complex -- deep nesting, many keys, expensive transformations
• CPU cores are available -- parallelism on a single-core machine adds only overhead

When Parallelism Hurts

Reduce or disable parallelism when:

• Documents are tiny (a few flat keys) -- thread overhead dominates
• Batch sizes are small (<50 items) -- raise parallel_threshold
• Memory is constrained -- each thread needs its own stack and working set
• Running inside a GIL-heavy Python workload -- the GIL is released during Rust processing, but other Python threads may contend

# Disable parallelism for small workloads
tools = jt.JSONTools().flatten().parallel_threshold(999_999)

# Or limit threads
tools = jt.JSONTools().flatten().num_threads(1)

Profiling Tips

Use the built-in benchmark suites to profile your specific workload pattern:

# Profile stress scenarios
cargo bench --profile profiling --bench stress_benchmarks --no-run
samply record --save-only -o /tmp/profile.json -- \
    ./target/profiling/deps/stress_benchmarks-* --bench

For Python profiling, measure wall-clock time since CPU profilers may not capture time spent in Rust:

import time
start = time.perf_counter()
result = tools.execute(data)
elapsed = time.perf_counter() - start
print(f"Processing took {elapsed:.3f}s")

Platform Notes

mimalloc (Rust-only)

The mimalloc global allocator is an optional feature that provides a measured ~14-28% performance improvement on allocation-heavy paths (normal-mode transforms, unflatten, batch-parallel flatten -- macOS aarch64). Enable it with features = ["mimalloc"] in your Cargo.toml. It is not included in Python builds -- PyO3 manages memory through Python's allocator, and an earlier attempt to bundle mimalloc into published wheels hit real cross-compilation breakage on several platforms and was reverted; this feature is for cargo add/source consumers who opt in themselves.

sonic-rs (64-bit only)

The main flatten/unflatten/normal-mode paths use an in-tree, tape-based scanner (scan_and_fixup(), shared across those three operations) rather than a general-purpose serde_json-style parser. sonic-rs (SIMD, 64-bit platforms) / simd-json (32-bit fallback) is used for a narrower case: a root-level JSON primitive (e.g. a bare "hello" or 42, not an object/array). This is transparent either way -- the public API is identical regardless of which parser handles a given input.

macOS Profiling

On macOS, flamegraph requires full Xcode (not just Command Line Tools). Use samply instead:

cargo install samply
samply record --save-only -o profile.json -- ./target/profiling/deps/BENCH_BINARY --bench
samply load profile.json  # Opens Firefox Profiler

Valgrind does not work on modern macOS. Use Instruments (if Xcode is installed) or samply for profiling.

Changelog

Unreleased

v0.9.30 (2026-08-09)

Performance

Round 16: continuing the algorithmic audit into territory prior rounds hadn't fully covered (builder.rs, every rayon/parallel-dispatch call site), found that whenever .num_threads(Some(n)) is set explicitly and a batch of documents is processed where individual documents are also wide enough to trigger nested parallelism, every worker thread of the batch-level thread pool independently built another fresh n-thread pool per qualifying document instead of reusing the pool it was already running inside -- up to O(batch_size) pool constructions instead of one. flatten_collecting_parallel now detects when the ambient pool already matches the requested thread count and reuses it instead of rebuilding. Confirmed 1.77x-2.09x faster (interleaved A/B, batches of 10/50/100 150-key documents, num_threads(Some(4))). Also cached std::thread::available_parallelism() (previously re-queried on every document qualifying for nested parallelism with no explicit num_threads override) -- a small, real, but modest win.

v0.9.29 (2026-08-08)

Performance

Round 15: continuing the algorithmic focus from rounds 13-14 (which covered python.rs's DataFrame layer three rounds running), this round's audit turned to the core engine underneath it (flatten.rs, unflatten.rs, convert.rs, transform.rs) and found two real issues. Normal mode's (non-.flatten()/.unflatten()) key-transform/collision path no longer allocates a String for every key. Using lowercase_keys, key replacement, or handle_key_collision(true) without flattening/unflattening used to unconditionally copy every object key at every nesting depth into a fresh heap-allocated String, even when nothing about the key actually changed; key storage switched from String to Cow<'a, str>, matching the allocation-avoidance idiom used everywhere else in this codebase. Also removed a redundant hashmap lookup in collision serialization. Confirmed 1.4x-1.85x faster (interleaved A/B, ~500KB nested JSON, lowercase_keys/handle_key_collision). Unflatten's array-to-object conversion now pre-sizes the new map instead of starting from zero capacity -- the same IndexMap doubling-growth fix already applied at three other sites in the same file, just missed at this one (a narrow but genuine path: a digit-only flattened key that overflows usize while sharing a parent with legitimate array indices). Confirmed 1.54x faster (interleaved A/B, 1000-element array converted to an object).

v0.9.28 (2026-08-07)

Performance

Round 14: continuing round 13's audit into the parts it hadn't covered (pandas fast path, general splice/unnest pipeline, arrow_columnar.rs, cache.rs, config.rs) found two more real issues, both in python.rs. Pandas fast-path eligibility now samples before fully extracting a column -- deciding whether an object-dtype column secretly holds embedded JSON used to extract the entire column via .tolist() + per-cell .extract::<String>() before sampling just the first 20 non-null values; now decoding stops as soon as the sample disqualifies. Confirmed ~2.1x faster (30K-row pandas DataFrame with an embedded-JSON column). Splicing and un-nesting fused into one parse+reconstruct pass instead of two -- every row with an embedded-JSON string column used to be parsed and serialized twice (once to splice, again to un-nest the freshly-spliced field). Confirmed ~2x faster (pandas) / ~2.3x faster (Polars). Caught a real regression during this change via the existing test suite: a second call site (normalise=True on DataFrame input) needed the same fix or it would have double-un-nested.

v0.9.27 (2026-08-06)

Performance

Round 13: with the core Rust engine already at its ceiling after 10 prior rounds (round 11: dependency freshness only; round 12: mimalloc-for-wheels re-confirmed already-rejected, a pure-Rust talc allocator tested and rejected -- regresses 4-5x under rayon's parallel path), an algorithmic audit focused on python.rs's DataFrame/normalise layer found two real issues. Eliminated duplicate Arrow string-column extraction on the flat-DataFrame fast-path fallback (.flatten().execute(df) with an embedded-JSON string column used to extract every string column twice); verified via code tracing, though end-to-end benchmarking showed no consistent wall-clock signal, so this is reported as a confirmed redundant-work fix, not a speed claim. normalise()'s per-batch allocation reduced from O(n_keys) separate heap allocations to one -- for a batch with mostly-disjoint keys (the same pattern that caused a real O(n^2) bug, already fixed, in unflatten.rs), confirmed ~30-35% faster median and, more strikingly, eliminates the wide run-to-run variance the many-allocations version showed. Also a small free memoization fix in unflatten.rs (a value-exclusion check recomputed per array-gap node instead of once per call).

v0.9.26 (2026-08-05)

Maintenance

Round 11 of this project's ongoing performance-optimization effort: researched and A/B tested several candidates (sonic-rs vs simd-json, simdutf8, mimalloc vs jemalloc, PyO3 free-threaded/no-GIL Python), but profiling confirmed the hot path is already at the ceiling reached by the prior 10 rounds -- nothing measurable to ship. The one concrete outcome is a dependency freshness pass: lifted two pins left artificially stale by the 2026-07-30 MSRV bump to 1.85, sonic-rs =0.5.7 -> =0.5.8 and indexmap >=2.11, <2.12 -> >=2.11, <2.15, both of which needed a newer rustc than the then-MSRV of 1.80. Verified clean on stable and MSRV 1.85; A/B benchmarked with no measurable latency change, as expected -- neither dependency sits on this crate's hot path.

v0.9.25 (2026-08-04)

Performance

The flat-DataFrame fast path (.flatten().execute(df) on pandas/Polars/PyArrow, added in 0.9.24) now releases the GIL for its computation, matching every other execution path -- it was the one path that held the GIL for its entire duration. This is a concurrency fix, not a latency optimization: a single call takes the same wall-clock time (confirmed, no regression), but other Python threads in the process can now make progress while it runs instead of stalling. Measured via a background pure-Python counting thread run concurrently with execute(df) (ratio of concurrent to solo throughput, 20K x 20 DataFrame): Polars 0.21 -> 1.00, PyArrow 0.14 -> 0.97, pandas 0.73 -> 0.76 (a smaller, real win -- pandas still builds one Python object per cell with the GIL held, unlike Arrow's zero-copy array construction). New benchmarks: bench_fastpath_gil.py, bench_fastpath_latency.py.

JSONTools::default() no longer allocates a String for the default separator, which SeparatorCache immediately special-cases into a borrowed constant anyway -- internal field is now Cow<'static, str>, no public API change.

Fixed

Pandas flat-DataFrame fast path: a column with both a genuine null and a remove_empty_strings-filtered-to-empty cell now matches the slow path's reconstruction exactly (None for the genuine null, NaN for the filtered cell, matching pandas' own missing-key behavior) -- previously both collapsed to None. A narrow, pre-existing 0.9.24 gap caught by a new differential test while implementing the GIL-release fix above. Polars/PyArrow were never affected.

v0.9.24 (2026-08-03)

Performance

execute(df) on a pandas/Polars/PyArrow DataFrame with no nested columns now skips the JSON-text round trip entirely (serialize -> parse+flatten -> deserialize -> reconstruct), reading column values directly and applying the same per-cell transform logic natively instead. Measured: the old round trip cost ~90-99ms for a 20K-row x 20-col DataFrame with no nesting, of which this crate's own flatten logic was only ~6.5ms -- the round trip itself was the cost. Strict whole-DataFrame fallback discipline (any nested column, embedded-JSON-string column, remove_nulls/value_exclusions, or a rename-induced key collision falls through to the existing pipeline unchanged) -- pure optimization, no second behavior to maintain. Scoped to plain .flatten().execute(df) (normalise=True already has its own Arrow-native path). Confirmed via interleaved A/B: Polars ~2.7-3.7x faster, PyArrow ~4.2-5.6x faster, pandas ~1.5-2.2x faster (up to ~345x for the pure column-rename case). No behavior change -- differential-tested against the existing pipeline's actual output across 10-11 cases per backend.

unflatten() no longer re-checks a container's array-vs-object classification on every visit to an already-created node -- only needed once, at creation, but was queried on every recursive call set_nested_value_recursive made (src/unflatten.rs). Found by looking at this project's own tracked CI benchmark history (benches/history.csv): unflatten baseline medium has consistently run ~4-5x slower than flatten baseline medium on the same document, across all 72 tracked commits; a live profile of that exact scenario pointed at this redundant lookup specifically. Confirmed ~9-10% faster across 3 interleaved A/B rounds. No behavior change.

v0.9.23 (2026-08-02)

Performance

Follow-up zero-copy audit of convert.rs/flatten.rs/unflatten.rs/python.rs. auto_convert_types's converted-value chain switched from Cow<str> (always heap-allocates when owned) to a new ConvertedStr type backed by CompactString, so short converted values (bools, small numbers) stay on the stack; flatten.rs's/unflatten.rs's collision-handling value storage got the same treatment. Confirmed ~11-13% faster for .flatten() with a key transform configured, ~3-9% faster for .normal() mode alone, both across 3 interleaved rounds. A third change (Arrow JSON-string-column extraction in python.rs, same String→CompactString idea) was measured the same way and showed no consistent signal (-1.8%/+0.5%/+9.2% across 3 rounds) -- kept as correct but not claimed as a win.

v0.9.22 (2026-08-02)

Added

• .always_array_keys([...]) -- flattened key names that must always render as a JSON array, even with only one value present, keeping a key's shape consistent across every document/row of a batch regardless of .handle_key_collision(). Also guarantees normalise() resolves that column to List<T> even when a particular batch has zero collisions for it. See the Key Collision Handling guide.

Performance

Audited every clone/copy site in the codebase (the core engine had zero .clone() calls already). PyJsonOutput.get_single()/get_multiple()/__str__() (the execute_to_output() API) cloned a result before PyO3's own unavoidable copy at the FFI boundary -- fixed to build the Python object directly from the borrowed value. Measured ~25-27% faster, confirmed via interleaved A/B, zero behavior change. Two further changes made on the same reasoning (an unflatten.rs allocation avoidance, a flatten.rs capacity hint) were measured the same way and honestly did not hold up as real wins -- reported plainly rather than claimed.

v0.9.21 (2026-08-01)

Performance

Profiling-driven follow-up round after v0.9.20 (samply/macOS sample against the Criterion stress suite, plus a targeted audit of the Arrow-native normalise() path). Three fixes, each verified via interleaved A/B:

• Date/datetime normalization output no longer re-parses a chrono format string per value (src/convert.rs) -- hand-rolled formatting replaces DateTime::format(). ~10.6% faster for date-heavy convert_dates(True) workloads.
• unflatten() no longer allocates a fresh path buffer per flattened key (src/unflatten.rs) -- one buffer reused across a document instead of one per entry. ~3.8% faster for wide flattened documents.
• normalise=True's Arrow-native reconstruction no longer parses a list-valued column's JSON array text twice (src/python.rs) -- exactly handle_key_collision(True)'s own headline scenario. ~5-6% faster for a collision-heavy scenario.

Second round, focused on batch processing and DataFrame conversion. Batch processing's core parallel dispatch was profiled directly and found already optimal (no fix needed); batching Python-side JSON parse calls was tested and found to make no difference (correctly abandoned before shipping). The real cost, found via cProfile: DataFrame extraction re-allocated an owned String per JSON object key on every row. unnest_object_valued_columns and splice_row now try a zero-copy key parse first, falling back to owned keys only when a key needs unescaping. ~6-9% faster per call for un-nesting (a quarter of a realistic execute(df) call), ~8.9% faster end to end for embedded-JSON-string-column DataFrames.

Third round, same focus: found the same "owned key per row" pattern one level up in build_normalise_table, with a second cost stacked on top -- key_order.insert(key.clone()) ran for every key of every row, even ones already seen (at issue #31's scale, 754 rows x 4,042 columns, ~3 million wasted clones). Now a contains check first means a key only allocates once, the first time it's actually new. ~13.0% faster for the issue #31-scale scenario, the largest single win of these three rounds. The same key-parse fix applied to splice_zerocopy_columns for consistency, measured as noise-level (not claimed as a win) since that path already minimizes what it re-parses per row by design.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.20 (2026-07-31)

Changed (BREAKING)

• DataFrame column expansion no longer prefixes with the source column's name -- a column named payload holding {"user": {"name": "Alice"}} now expands to user.name, not payload.user.name (dict/struct-typed columns and JSON-string columns alike, in both execute(df) and execute(df, normalise=True)). Genuine nesting within a column's content still prefixes normally; array-valued columns are unaffected (tags.0, tags.1, ...). A key colliding across two columns resolves via the existing .handle_key_collision() setting. See DataFrame & Series Support.
• normalise=True/target=... reconstruction is now Arrow-native (issue #35) -- one real Arrow RecordBatch built directly in Rust, no new methods. handle_key_collision(True) list columns now build as real, correctly-typed List<T> instead of being stringified. Recognized date/datetime columns build as real Date32/Timestamp columns, gated on .convert_dates(True)/.auto_convert_types(True) (never independently guessed). target="pandas" output uses Arrow-backed dtypes (int64[pyarrow], ...) -- genuinely zero-copy, but a breaking dtype change. target="pandas"/"pyspark" now require pyarrow installed (target="polars" does not). See DataFrame & Series Support.

Performance

• normalise=True/target=... reconstruction: honest, modest ~1-4% end-to-end win -- core flattening/input serialization dominate total time for typical data, not reconstruction; the real win this round is column-typing correctness, not speed.
• .convert_dates(True)'s own detection cost: ~0.1-4.5% end-to-end, measured directly including the worst case (conversion on, no actual dates present) -- not charged when date conversion is off.

Removed (BREAKING)

• The JVM/Java/Scala binding has been removed entirely -- jvm/, src/jvm.rs, the jni dependency/jvm Cargo feature, and the JVM CI workflow are all gone, and the io.github.amaye15:json-tools-rs-spark Maven Central artifact will not receive new versions (0.9.19 remains available there). The Rust core and Python bindings are unaffected. Databricks/Spark users should switch to the Python bindings wrapped in a pandas_udf -- see Setting Up on Databricks.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.19 (2026-07-30)

Changed

• MSRV raised from 1.80 to 1.85 -- required for current pyo3-arrow releases (see Performance below). Affects every source (cargo add) consumer.

Performance

• execute() on a Polars DataFrame/PyArrow Table with an embedded JSON-string column is ~41-48% faster end-to-end. Detection/extraction of such columns now uses pyo3-arrow's zero-copy Arrow buffer access instead of round-tripping through the DataFrame's native JSON writer (escape, then immediately unescape). Column ordering is preserved exactly as before. Scoped to Polars/PyArrow -- plain pandas and PySpark are unaffected, still using the existing path.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.18 (2026-07-30)

Performance

• execute() on a PyArrow Table/RecordBatch is ~2x faster -- extraction now bridges through pandas's native JSON writer (using types_mapper=pd.ArrowDtype to avoid a real integer-with-nulls-to-float corruption bug caught while building this fix) instead of to_pylist() + per-item conversion. splice_row's per-key escaping now reuses the crate's existing zero-allocation key writer instead of serde_json::to_string; a real cleanup, though measured end-to-end impact was within noise at realistic scale.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.17 (2026-07-30)

Performance

• Core flatten()/unflatten() collision-handling paths ~5-8% faster for documents that trigger key transforms/collision detection -- eliminated a redundant second hashmap lookup per unique key in both flatten.rs and unflatten.rs. The remaining non-normalise DataFrame/Series reconstruction functions now use the same PyOnceLock import caching added in 0.9.16. mimalloc's doc comment updated to real measured numbers (~14-28%, previously an unverified "~5-10%") plus new CI coverage; still not used for published wheels/jars.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.16 (2026-07-30)

Performance

• execute(..., normalise=True, target=...) and PySpark execute(spark_df) are 13-18% faster for large/wide results (e.g. 754 rows x 4,042 columns). union_and_columnarize rewritten from an O(rows x columns) PyDict hash-lookup pattern to a single forward pass, plus PyOnceLock-cached pandas/polars/pyarrow/pyspark module imports across the reconstruction path. Verified via interleaved A/B against the real Python API.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.15 (2026-07-29)

Fixed

• execute(spark_df) no longer crashes when a .key_replacement()/.handle_key_collision(True) list column holds genuinely mixed element types (#33). The list-flavored twin of the 0.9.14 fix: a collision list is built from each colliding key's own independently-converted value, so a single row's collision could already mix kinds (e.g. [100, "abc"]); such columns now fall back to string elements, while uniformly-typed list columns (e.g. all int) correctly get a typed array instead of unnecessary stringification.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.14 (2026-07-29)

Fixed

• execute(spark_df) with auto_convert_types(True) no longer crashes on columns with genuinely mixed types (#32). A column that ends up holding e.g. both str and int values across rows (a natural consequence of per-value auto-conversion) previously broke Spark's Arrow bridge with PySparkTypeError; such columns now fall back to a uniform string column instead, while int/float-only mixes still promote correctly to double. Also hardens normalise(target=...) for all four DataFrame backends, which share the same column-unioning step.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.13 (2026-07-29)

Fixed

• execute(df) on a PySpark DataFrame now returns a real, distributed pyspark.sql.DataFrame, not a plain list[dict] (#31). Behavior change: code relying on the old list fallback needs updating.

Performance

• JSON-string-column auto-expansion (0.9.12) is ~40-43% faster for large embedded payloads, rewritten around serde_json::value::RawValue to avoid a redundant full-tree parse/reserialize per row, while keeping the same graceful per-row fallback behavior. See CHANGELOG.md for the full root-cause writeup.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.12 (2026-07-29)

Fixed

• execute(df) in .flatten() mode now auto-expands DataFrame columns holding JSON strings, not just columns already typed as dicts/structs (#30) -- see Auto-Expanding JSON-String Columns. Behavior change: a DataFrame with a JSON-string column now produces more/differently-shaped output columns in flatten mode than before; .unflatten()/.normal() mode are unaffected.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.11 (2026-07-28)

Fixed

• normalise(target="pyspark") could silently corrupt an all-None column on Spark's non-Arrow fallback path (taken when pyarrow isn't installed) -- a missing value could serialize as the literal string "<NA>" instead of a real null. Fixed by passing an explicit schema to createDataFrame instead of relying on Spark to infer it.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.10 (2026-07-28)

Fixed

• Critical: auto_convert_types panicked on multi-byte UTF-8 content in specific positions (e.g. "5€ García", a "+1Á2" timezone offset) -- two fixed-byte-offset string slices assumed the offset was always a UTF-8 character boundary. Fixed with an is_char_boundary guard at each site; no behavior change for valid inputs. (#29)

Added

• Python: JSONTools is now picklable, including across a real process boundary (e.g. captured in a PySpark UDF/mapInPandas closure via cloudpickle) -- via __reduce__ plus a new to_config_json()/from_config_json() method pair. (#29)
• Python: execute(input, normalise=True, target=None) -- always returns a wide DataFrame (one column per flattened key) regardless of input shape, working natively across pandas, polars, pyarrow, and now genuinely PySpark (a real pyspark.sql.DataFrame, closing the previous list-of-dicts fallback for this path). See DataFrame & Series Support.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.8 (2026-07-26)

Changed

• Python: orjson is now a required dependency, used automatically as the dict/DataFrame-row JSON (de)serialization backend -- pip install json-tools-rs is all that's needed, no separate opt-in. A per-call fallback to the standard library still covers inputs orjson can't handle -- see Installation.

Performance

• Python binding: exact-typed str/dict/list inputs skip DataFrame/Series detection entirely, and the JSON callables are resolved once instead of per call. Dict-input calls ~37% faster, str-input calls ~39% faster.
• JVM binding: native execute/executeBatch now cross the JNI boundary as UTF-8 byte[] instead of String (JIT-intrinsified on the Java side, avoiding JNI's UTF-16 conversion). ~22-38% faster per call, ~33% faster for batches. Public API unchanged.
• Unflatten: nested-container capacity hints tuned against this project's own benchmark corpus (was an undersized flat guess), plus a single-lookup entry() replacing a double hash probe. ~5-6% faster unflatten, ~4-5% faster roundtrip.
• Core scanner: removed a double-scan of scalar/whitespace bytes in the tape scanner. ~13-16% faster flatten across payload sizes.

See the repository's CHANGELOG.md for the full, itemized list.

v0.9.7 (2026-07-20)

Added

• .exclude_key(pattern) (Rust/Python/JVM): drop any key -- and its entire value/subtree -- whose name contains pattern (literal by default, r'...' for regex). Additive. Works identically in .flatten(), .unflatten(), and .normal() mode; matching a container key drops its entire subtree in O(1) without walking it. Array elements are never matched. See Key Exclusion.
• .exclude_value(pattern) (Rust/Python/JVM): drop a key-value pair whose value contains pattern. Same convention as .exclude_key(). Only applies to scalar leaf values; checked after .value_replacement()/.auto_convert_types() have run, matching .remove_nulls()'s ordering guarantee. A no-op at the document root. In .unflatten() mode, string values are matched against their JSON-serialized (quoted) form -- see Value Exclusion for the regex-anchor caveat this implies.

Fixed

• .remove_nulls() now runs consistently last across .flatten()/.unflatten()/.normal() mode. Previously .value_replacement() and .auto_convert_types() composed in three different orders across the three engines (flatten and unflatten each had a real ordering bug; only normal mode was already correct), so the same document/config could produce different results depending on mode, and a value that only became null after a replacement could slip past .remove_nulls(). All three engines -- including each one's root-primitive (bare-scalar-document) path -- now compose replacement-then-conversion identically.

v0.9.6 (2026-07-19)

Added

• Fine-grained, per-category control over automatic type conversion, across Rust, Python, and JVM: .convert_dates(), .convert_nulls(), .convert_booleans(), .convert_numbers() let each category be enabled/disabled independently instead of the previous all-or-nothing .auto_convert_types(bool) (unchanged, still means "all four categories, default behavior"). Each category also accepts real customization via a _config method/kwargs/dedicated fluent methods (per language idiom) -- dates: normalize_to_utc/assume_utc_for_naive; nulls/booleans: extra recognized tokens (additive); numbers: individually disable currency, percent/permille, text basis points, K/M/B/T suffixes, fractions, or hex/binary/octal parsing. See Type Conversion.
• New public types: TypeConversionConfig, DateConversionConfig, NullConversionConfig, BooleanConversionConfig, NumberConversionConfig, plus runnable examples, tests, and benchmarks for the new API across all three languages.

Changed (BREAKING)

• ProcessingConfig (and FilteringConfig/CollisionConfig/ReplacementConfig) are now #[non_exhaustive], matching JsonToolsError's existing precedent. Breaks external code constructing these via a bare struct literal instead of ::new() + the fluent builder methods.
• ProcessingConfig.auto_convert_types: bool removed, replaced by ProcessingConfig.type_conversion: TypeConversionConfig. The JSONTools builder's own .auto_convert_types(bool) method is unaffected.

Performance

• The existing, heavily-profiled try_convert_string_to_json_bytes hot path is unmodified -- it remains the code path for the common (all-default) case, selected via a mode cached once per execute() call. all_default_via_new_api benchmark confirms within ~1% of the prior auto_convert_types cost.

See the repository's CHANGELOG.md for the full, itemized list including edge-case coverage details.

v0.9.5 (2026-07-18)

Fixed

• Documentation-wide accuracy sweep: every root-level doc, the full mdBook site, and the JVM Java source's own doc comments audited against actual source code and live runtime behavior across four parallel passes, rather than trusting existing prose. Corrected fabricated/stale internals (references to a phf key cache, rustc-hash, Arc<str> key dedup, and function names that no longer exist), benchmark numbers stale by up to 14x, wrong error-handling semantics, several broken guide examples (a .normal() mode key-replacement/lowercasing ordering bug, an impossible collision-handling example, a no-op Polars example), and stale "not yet published" claims for Maven Central/PyPI (both have been live for a while). Fixed a real internal contradiction in the JVM Java source itself (FlattenUDF/BatchTransform javadoc claimed Lakeflow Pipeline support that Databricks doesn't allow). See CHANGELOG.md for the full, itemized list.

Added

• Runnable examples covering every builder feature individually, plus curated multi-feature pipelines, mirrored with matching inputs/outputs across Rust, Python, and Java.
• JVM API reference page, closing a gap where Rust and Python each had one and the JVM bindings didn't.

Changed

• Regex pattern lookup for key_replacement/value_replacement no longer re-hashes and re-walks the cache on every key/value check -- a thread-local "sticky" cache of recently-used patterns short-circuits the common case. ~9-22% faster on regex-heavy scenarios (Criterion).
• Consolidated two near-duplicate replacement-application code paths, which also fixed a missing SIMD fast-path for literal value replacement (~15-19% faster for that case).

v0.9.4 (2026-07-17)

Fixed

• auto_convert_types silently corrupted the trailing digits of large integer strings: numeric-string-to-JSON-number conversion always routed every candidate through f64 (only ~15-17 significant decimal digits of exact precision) before reformatting, so any string-encoded integer longer than that came back corrupted, e.g. "999999999999999999" → 1000000000000000000. Real-world 64-bit IDs (Snowflake/Discord/database bigint primary keys) are commonly stored as JSON strings specifically to avoid this exact class of precision loss elsewhere, and are typically 17-19 digits, so this was a live bug. Already-canonical integer strings are now reused directly instead of being parsed to f64 and reformatted, covering the entire range the previous float round-trip claimed to support (checked precisely against i64/u64 bounds, not a rough digit-count cutoff).

Changed

• Python bindings: dict/list[dict]/DataFrame/Series conversion switched from pythonize/depythonize to Python's own json module. Benchmarked against the actual built extension (not just reasoned about): depythonize's generic serde-based Python↔Rust traversal was 5-30% slower than a plain json.dumps/json.loads round-trip for nested dicts (the case .flatten()/.unflatten() exist for), and ~1.6x slower end-to-end for DataFrame rows (this library's other headline feature) — the reverse of what the code's own prior comments claimed. DataFrame input now uses each library's native line-delimited JSON export (pandas to_json, polars write_ndjson) instead of to_dict()/to_dicts() + per-row conversion. Removes the pythonize dependency entirely. Trade-off, reported honestly: flat/shallow dicts are slower under the new approach (still microsecond-scale in absolute terms) — see CHANGELOG.md for the full numbers.
• Credit/debit currency suffix stripping ("100CR"/"100DR", part of auto_convert_types) no longer chains str::trim_end_matches calls with string patterns, which forced std to construct generic substring-search machinery for a fixed 2-byte suffix check. ~13-17% faster on currency-heavy conversion (Criterion).
• Literal (non-regex) key/value replacement now locates matches with SIMD substring search (memchr::memmem) instead of str::replace's matcher. ~2.6-4.8% faster (Criterion).
• unflatten's internal object maps (root and per-branch) now start pre-sized instead of growing from empty capacity one key at a time. ~7-9% faster combined (Criterion), found via sampling profiler.
• auto_convert_types's date detection now validates via chrono's direct date constructors instead of its generic format-string parser. ~25% faster on mixed real-dates/false-positive-numeric-ID workloads.

Added

• flatten's slow path (key lowercasing/replacement/collision-handling configured) now uses an arena allocator for key storage on single-document processing, instead of allocating each dotted key path individually. Up to ~14% faster end-to-end on deep-nesting workloads; neutral on shallow/mixed data.

v0.9.3 (2026-07-16)

Fixed

• flatten produced invalid JSON for keys with escaped characters: any key containing \", \\, or a control-character escape produced syntactically invalid JSON output when no key transform (lowercase_keys/key_replacement/collision-handling) was configured -- the default, most common usage. The fast path unescaped such keys to build its internal path buffer but never re-escaped before writing that buffer directly as the output key.
• Re-escaping corrupted multi-byte UTF-8 characters: whenever a string needed escaping (an embedded quote, backslash, or control character) and also contained non-ASCII text, the slow escaping path reinterpreted each byte individually as its own Latin-1 codepoint, e.g. turning café "quoted" into café \"quoted\". Affected key escaping under lowercase_keys/key_replacement/collision-handling, value escaping under value_replacement, and unflatten's key serialization.

Changed

• JSON object keys now use CompactString instead of String, inlining keys up to 24 bytes with no heap allocation. unflatten is ~19-22% faster (Criterion, p < 0.05).
• unflatten's tree-building pass no longer re-scans each key's separators a second time.
• The regex pattern cache now evicts the genuinely least-recently-used entry when full, instead of an arbitrary one.
• unflatten's output buffer is sized from the input JSON's byte length instead of a fixed 256-byte default.

v0.9.2 (2026-07-15)

Note: v0.9.1 was tagged the day before but only completed publishing to Maven Central -- a crates.io/PyPI release pipeline bug caused those two to fail before any upload. Fixed and re-cut as v0.9.2 across all three registries; no code changes beyond the release pipeline fix itself.

Added

• JVM (Java) bindings: Apache Spark UDFs (row and batched mapPartitions tiers) via a JNI shim over the same Rust core, full feature parity with the Python bindings. See Setting Up on Databricks.
• crates.io and Maven Central publishing on tagged releases.

Changed (BREAKING)

• key_replacement/value_replacement pattern syntax: patterns are now literal (exact substring match) by default; wrap a pattern in r'...' (e.g. r'^admin_') to use it as a regex. Previously every pattern was always compiled as regex regardless of content. See Key & Value Replacements.

Fixed

• has_escape scanner bug: escape sequences not adjacent to a quote (a lone \n, \t, \r, \uXXXX) were invisible to the tape scanner, so auto_convert_types, replacements, lowercase_keys, and collision handling could silently operate on still-escaped text for affected strings.
• Parallelism reverted from Crossbeam back to Rayon: batch processing now uses Rayon's persistent work-stealing pool instead of spawning fresh std::thread::scope OS threads on every .execute() call -- measurably faster for small-to-medium batches.
• unflatten's object tree switched from a hash map + full key sort to an order-preserving map (IndexMap), removing an O(n) lookup that degraded to O(n^2) for JSON objects used as wide keyed maps.

See the repository's CHANGELOG.md for full details.

v0.9.0 (2026-03-09)

Added

• DataFrame & Series Support (Python): Native support for Pandas, Polars, PyArrow, and PySpark DataFrames and Series with perfect type preservation.
• Crossbeam Parallelism: Migrated from Rayon to Crossbeam for finer-grained parallel control with scoped threads.
• Modular Architecture: Refactored monolithic lib.rs into 10 focused modules (json_parser, types, error, config, cache, convert, transform, flatten, unflatten, builder) with zero public API changes.

Performance Improvements

Rust Core (6 optimizations):

• Eliminated per-entry HashMap allocation in parallel flatten -- single partial map per chunk
• Added early-exit first-byte discriminators for type conversion fast-path
• SIMD literal fallback for regex patterns (memchr before regex compilation)
• Thread-local regex cache half-eviction (LRU-style, capacity 64)
• Expanded SmallVec buffers (32 -> 64 bytes) and separator cache
• Vectorized clean_number_string() with SIMD skip helpers

Python Bindings (3 optimizations):

• mem::replace -> mem::take across 13 builder methods, eliminating a default JSONTools::new() construction per call
• O(N) -> O(1) DataFrame/Series reconstruction (single into_pyobject + clone_ref instead of a per-item clone)
• GIL release via py.detach() during compute-intensive operations

v0.8.0 (2026-01-01)

• Full Python Bindings Feature Parity: all Rust features now available in Python, including .auto_convert_types(), .parallel_threshold(), .num_threads(), and .nested_parallel_threshold()
• 128 comprehensive Python tests covering all features

v0.7.0 (2025-10-17)

• Parallel configuration methods (parallel_threshold, num_threads, nested_parallel_threshold)
• HashMap capacity and hashing optimizations

v0.6.0 (2025-10-13)

• Python GIL release for parallel operations (5-13% improvement)
• Inline hints on hot functions

v0.5.0 (2025-10-12)

• #[inline(always)] on hot-path functions and #[cold]/#[inline(never)] on error paths (2-5% additional improvement, 32-60% cumulative from baseline)

v0.4.0 (2025-10-11)

• FxHashMap replacing standard HashMap (15-30% faster string key operations)
• SIMD JSON parsing optimizations, reduced string clones (~50% fewer), pre-allocated collections (30-55% overall improvement)

v0.3.0 (2025-10-10)

• Automatic type conversion
• Python bindings via PyO3

v0.2.0 (2025-10-09)

• Key collision handling
• Comprehensive filtering (empty strings, nulls, objects, arrays)
• Regex-based replacements

v0.1.0 (2025-10-08)

• Initial release
• JSON flattening and unflattening
• Custom separators
• Batch processing

For the full changelog with migration guides, see CHANGELOG.md.