Python API Reference

import json_tools_rs

JSONTools

The main builder class for all JSON operations. All configuration methods return self for chaining; only .execute() and .execute_to_output() trigger processing.

Construction

tools = json_tools_rs.JSONTools()

Creates a new JSONTools instance with all default settings. The instance is reusable -- you can call .execute() multiple times with different inputs.

Operation Modes

Exactly one mode must be set before calling .execute(). Calling a mode method replaces any previously set mode.

.flatten()

tools.flatten() -> JSONTools

Set the operation to flatten nested JSON into dot-separated (or custom separator) keys.

import json_tools_rs as jt

result = jt.JSONTools().flatten().execute({"a": {"b": {"c": 1}}})
# {"a.b.c": 1}

.unflatten()

tools.unflatten() -> JSONTools

Set the operation to reconstruct nested JSON from flat, separator-delimited keys.

result = jt.JSONTools().unflatten().execute({"a.b.c": 1})
# {"a": {"b": {"c": 1}}}

.normal()

tools.normal() -> JSONTools

Set the operation to apply transformations (filtering, replacements, type conversion) without changing the nesting structure.

result = jt.JSONTools().normal().lowercase_keys(True).execute({"Name": "Alice"})
# {"name": "Alice"}

Configuration Methods

All configuration methods return self for chaining.

.separator(sep)

tools.separator(sep: str) -> JSONTools

Set the key separator for flatten/unflatten operations.

Raises: ValueError if sep is an empty string.

result = jt.JSONTools().flatten().separator("::").execute({"a": {"b": 1}})
# {"a::b": 1}

.lowercase_keys(flag)

tools.lowercase_keys(flag: bool) -> JSONTools

Convert all keys to lowercase after processing.

result = jt.JSONTools().flatten().lowercase_keys(True).execute({"User": {"Name": "Alice"}})
# {"user.name": "Alice"}

.remove_empty_strings(flag)

tools.remove_empty_strings(flag: bool) -> JSONTools

Remove key-value pairs where the value is an empty string "".

result = jt.JSONTools().flatten().remove_empty_strings(True).execute({"a": "", "b": "hello"})
# {"b": "hello"}

.remove_nulls(flag)

tools.remove_nulls(flag: bool) -> JSONTools

Remove key-value pairs where the value is None / null.

result = jt.JSONTools().flatten().remove_nulls(True).execute({"a": None, "b": 1})
# {"b": 1}

.remove_empty_objects(flag)

tools.remove_empty_objects(flag: bool) -> JSONTools

Remove key-value pairs where the value is an empty object {}.

.remove_empty_arrays(flag)

tools.remove_empty_arrays(flag: bool) -> JSONTools

Remove key-value pairs where the value is an empty array [].

.key_replacement(find, replace)

tools.key_replacement(find: str, replace: str) -> JSONTools

Add a key replacement pattern. Patterns are literal (exact substring match) by default; wrap a pattern in r'...' (e.g. "r'^user_'") to use standard regex syntax instead. A malformed r'...' pattern is silently treated as "no match" rather than raising an error. Multiple replacements can be chained.

result = (jt.JSONTools()
    .flatten()
    .key_replacement("r'^user_'", "")
    .key_replacement("r'_name$'", "_id")
    .execute({"user_name": "Alice"}))
# {"id": "Alice"}

.value_replacement(find, replace)

tools.value_replacement(find: str, replace: str) -> JSONTools

Add a value replacement pattern. Works the same as key replacements (literal by default, r'...' for regex) but applies to string values.

result = (jt.JSONTools()
    .flatten()
    .value_replacement("@example.com", "@company.org")
    .execute({"email": "user@example.com"}))
# {"email": "user@company.org"}

.exclude_key(pattern)

tools.exclude_key(pattern: str) -> JSONTools

Drop any key -- and its entire value/subtree -- whose name contains pattern. Literal (exact substring match) by default; wrap in r'...' for regex, matching key_replacement's convention. Additive -- call once per keyword to exclude multiple. Checked against the full dot-path in flatten/unflatten mode, and per key at each nesting level in normal mode; matching a container key drops its entire subtree without walking it. Array elements are never matched (no key name to check).

result = (jt.JSONTools()
    .flatten()
    .exclude_key("crypto")
    .execute({"user": {"name": "John", "crypto_wallet": {"coin": "BTC"}}}))
# {"user.name": "John"}

.exclude_value(pattern)

tools.exclude_value(pattern: str) -> JSONTools

Drop a key-value pair whose value contains pattern. Same literal/r'...' convention as exclude_key. Additive. Only ever applies to scalar leaf values (strings/numbers/booleans/null) -- containers have no single value to check. Checked against the final value after any configured value_replacement/ auto_convert_types have run. A no-op at the document root.

Unflatten-specific note: string values are matched against their JSON-serialized form (including surrounding quotes), not the unescaped logical text. Literal patterns are unaffected; a regex with anchors needs r'^"admin"$' rather than r'^admin$' to match a value that's exactly "admin".

result = (jt.JSONTools()
    .flatten()
    .exclude_value("banned")
    .execute({"user": {"name": "John", "status": "banned"}}))
# {"user.name": "John"}

.handle_key_collision(flag)

tools.handle_key_collision(flag: bool) -> JSONTools

When enabled, keys that would collide after transformations (e.g., after lowercasing) are collected into arrays instead of overwriting each other.

result = (jt.JSONTools()
    .flatten()
    .lowercase_keys(True)
    .handle_key_collision(True)
    .execute({"Name": "Alice", "name": "Bob"}))
# {"name": ["Alice", "Bob"]}

.always_array_keys(keys)

tools.always_array_keys(keys: Sequence[str]) -> JSONTools

Flattened key names that must always render as a JSON array, even when only one value is present in a given document -- keeps a key's scalar-vs-array shape consistent across every document/row of a batch, not just documents where a collision happened to occur. Independent of .handle_key_collision(): a key named here always gets full array treatment. See Key Collision Handling for the full explanation, including why this matters for normalise()'s List<T> column typing.

result = (jt.JSONTools()
    .flatten()
    .key_replacement("r'(User|Admin)_'", "")
    .always_array_keys(["name"])
    .execute({"User_name": "John"}))
# {"name": ["John"]}  -- wrapped even though nothing collided here

.auto_convert_types(flag)

tools.auto_convert_types(flag: bool) -> JSONTools

Automatically convert string values to their native types:

• Numbers: "123" -> 123, "1,234.56" -> 1234.56, "$99.99" -> 99.99, "1e5" -> 100000
• Booleans: "true" / "TRUE" / "True" -> true, "false" / "FALSE" / "False" -> false
• Nulls: "null" / "None" -> null

If conversion fails, the original string is kept. No errors are raised on conversion failure.

result = (jt.JSONTools()
    .flatten()
    .auto_convert_types(True)
    .execute({"id": "123", "price": "1,234.56", "active": "true"}))
# {"id": 123, "price": 1234.56, "active": true}

.convert_dates(enable, normalize_to_utc=None, assume_utc_for_naive=None) / .convert_nulls(enable, extra_tokens=None) / .convert_booleans(enable, extra_true_tokens=None, extra_false_tokens=None) / .convert_numbers(enable, currency=None, percent=None, basis_points=None, suffixes=None, fractions=None, radix=None)

tools.convert_dates(enable: bool, normalize_to_utc: bool | None = None, assume_utc_for_naive: bool | None = None) -> JSONTools
tools.convert_nulls(enable: bool, extra_tokens: list[str] | None = None) -> JSONTools
tools.convert_booleans(enable: bool, extra_true_tokens: list[str] | None = None, extra_false_tokens: list[str] | None = None) -> JSONTools
tools.convert_numbers(enable: bool, currency: bool | None = None, percent: bool | None = None, basis_points: bool | None = None, suffixes: bool | None = None, fractions: bool | None = None, radix: bool | None = None) -> JSONTools

Independent, per-category alternative to .auto_convert_types(): enable/customize dates, nulls, booleans, and numbers separately instead of all-or-nothing. auto_convert_types(True) only flips each category's on/off switch and preserves customization already set via these methods -- call order doesn't reset it. A kwarg left as None on a later call also preserves whatever a previous call set (it's not reset to the built-in default).

Plain integers/decimals, scientific notation, and thousands-separator cleanup are always applied when convert_numbers is enabled, regardless of the other kwargs.

result = (jt.JSONTools()
    .flatten()
    .convert_dates(True, assume_utc_for_naive=False)
    .convert_nulls(True, extra_tokens=["missing"])
    .execute({"d": "2024-01-15T10:30:00", "a": "missing"}))
# {"d": "2024-01-15T10:30:00", "a": None}

.parallel_threshold(n)

tools.parallel_threshold(n: int) -> JSONTools

Set the minimum batch size to trigger parallel processing. Batches smaller than this are processed sequentially to avoid thread-spawning overhead.

Default can be overridden with the JSON_TOOLS_PARALLEL_THRESHOLD environment variable.

tools = jt.JSONTools().flatten().parallel_threshold(50)

.num_threads(n)

tools.num_threads(n: int | None) -> JSONTools

Set the number of threads used for parallel processing. Pass None (or omit the call) to use the system default.

Default can be overridden with the JSON_TOOLS_NUM_THREADS environment variable.

tools = jt.JSONTools().flatten().num_threads(4)

.nested_parallel_threshold(n)

tools.nested_parallel_threshold(n: int) -> JSONTools

Set the minimum number of keys/items within a single JSON document to trigger nested (intra-document) parallelism. Only objects or arrays exceeding this count are parallelized internally.

Default can be overridden with the JSON_TOOLS_NESTED_PARALLEL_THRESHOLD environment variable.

tools = jt.JSONTools().flatten().nested_parallel_threshold(200)

.max_array_index(n)

tools.max_array_index(n: int) -> JSONTools

Set the maximum array index allowed during unflattening. This is a DoS protection: a malicious key like "items.999999999" would otherwise allocate a massive array.

Default can be overridden with the JSON_TOOLS_MAX_ARRAY_INDEX environment variable.

Execution Methods

.execute(input, normalise=False, target=None)

tools.execute(input) -> str | dict | list[str] | list[dict] | DataFrame | Series
tools.execute(input, normalise=True, target=None) -> DataFrame

Execute the configured operation. By default (normalise=False) the return type mirrors the input type:

Raises: JsonToolsError if no mode is set, input is invalid, processing fails, or (PySpark input) no active SparkSession is found.

# String input -> string output
result = jt.JSONTools().flatten().execute('{"a": {"b": 1}}')
assert isinstance(result, str)

# Dict input -> dict output
result = jt.JSONTools().flatten().execute({"a": {"b": 1}})
assert isinstance(result, dict)

# Batch string input -> batch string output
results = jt.JSONTools().flatten().execute(['{"a": 1}', '{"b": 2}'])
assert isinstance(results, list) and isinstance(results[0], str)

# Batch dict input -> batch dict output
results = jt.JSONTools().flatten().execute([{"a": {"b": 1}}, {"c": {"d": 2}}])
assert isinstance(results, list) and isinstance(results[0], dict)

normalise / target: always get back a wide DataFrame

normalise=True bypasses the input-mirroring table above entirely: regardless of input's shape, the result is always a wide DataFrame (one column per flattened key) -- a bare str/dict becomes a 1-row DataFrame. Requires .flatten() mode. Reconstruction builds one real Arrow table internally with genuinely typed columns (including real List<T> for handle_key_collision(True)), then derives whichever target was requested from it -- see DataFrame & Series Support's "Arrow-native reconstruction" callout for the full behavior (key union/null-fill order, target auto-resolution, the pandas dtype change, the PySpark path) and examples.

tools = jt.JSONTools().flatten()

df = tools.execute({"user": {"name": "Alice"}}, normalise=True)          # auto-resolved target
df = tools.execute([{"a": 1}, {"a": 2}], normalise=True, target="polars")

Additional raises (when normalise=True or target is set): mode is not .flatten(); target is set while normalise=False; target names an unknown or uninstalled library; target="pandas"/"pyspark" without pyarrow installed (both require it internally now -- target="polars" does not); or (target="pyspark") no active SparkSession is found.

.execute_to_output(input)

tools.execute_to_output(input) -> JsonOutput

Execute the operation but return a JsonOutput wrapper instead of native Python types. Useful when you need to inspect whether the result is single or multiple before extracting.

Note: DataFrame and Series inputs are not supported with execute_to_output(). Use .execute() for those types.

output = jt.JSONTools().flatten().execute_to_output('{"a": {"b": 1}}')
if output.is_single:
    print(output.get_single())
elif output.is_multiple:
    for item in output.get_multiple():
        print(item)

Pickling and to_config_json() / from_config_json()

JSONTools instances are picklable (pickle.dumps/pickle.loads), which means a configured instance can also be captured in a closure that crosses a real process boundary via cloudpickle -- most notably inside a PySpark UDF or mapInPandas function, without needing a workaround.

tools = jt.JSONTools().flatten().remove_nulls(True)

config = tools.to_config_json()          # -> str
restored = jt.JSONTools.from_config_json(config)  # a fresh, independent instance

to_config_json()/from_config_json() are the mechanism pickling is built on top of, and are directly useful on their own for the same reason: a mapInPandas partition function should close over the config string (not the JSONTools instance itself) and call from_config_json() once inside each partition to get a working, independent instance -- this is exactly how the pickle support works internally (__reduce__ returns (from_config_json, (config_json,))).

JsonOutput

Output wrapper returned by .execute_to_output(). Provides typed access to results.

Properties

Methods

.get_single()

output.get_single() -> str

Extract the single JSON string result.

Raises: ValueError if the result is multiple.

.get_multiple()

output.get_multiple() -> list[str]

Extract the list of JSON string results.

Raises: ValueError if the result is single.

.to_python()

output.to_python() -> str | list[str]

Convert to native Python type: returns str for single results, list[str] for multiple results.

String Representations

str(output) returns the JSON string (single) or a list representation (multiple). repr(output) returns JsonOutput.Single('...') or JsonOutput.Multiple([...]).

DataFrame and Series Support

JSON Tools RS natively supports Pandas, Polars, PyArrow, and PySpark DataFrames and Series. Detection is performed via duck typing -- no explicit imports are required.

Performance note: in .flatten() mode, a DataFrame with no nested columns to flatten automatically takes a faster internal path that skips JSON serialization entirely -- transparent, no flag to set, same output either way. See Performance: The Flat-DataFrame Fast Path.

Pandas DataFrame

Each row is serialized to a JSON object (column names become keys) and processed as a whole document -- so flattening finds nested structure in columns holding actual nested Python objects (dicts/lists) directly. In .flatten() mode, columns holding pre-serialized JSON-text strings are also detected and expanded the same way (auto-detected, not requiring the column to already be dict/list-typed) -- see Auto-Expanding JSON-String Columns for the detection rules. The source column's own name is never kept as a prefix in the output -- only nesting within the column's own content is (see that same section). .unflatten()/.normal() mode leave a JSON-string column's value untouched, as a plain string scalar.

import pandas as pd
import json_tools_rs as jt

df = pd.DataFrame({"user": [
    {"name": "Alice", "age": 30},
    {"name": "Bob", "age": 25},
]})

tools = jt.JSONTools().flatten().separator(".")

# Each row -> {"user": {"name": ..., "age": ...}} -> flattened, "user" itself
# dropped (it's the column name, not part of the payload)
result_df = tools.execute(df)
# Returns a DataFrame with flattened columns: "name", "age"

Pandas Series

series = pd.Series([
    '{"a": {"b": 1}}',
    '{"a": {"b": 2}}',
])

result_series = jt.JSONTools().flatten().execute(series)
# Returns a Series of flattened JSON strings

Polars DataFrame

Like Pandas, this flattens a column of nested Struct values -- a column of JSON-text strings round-trips unchanged, since there's no nested structure inside a string scalar for .flatten() to find.

import polars as pl

df = pl.DataFrame({
    "user": [{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}]
})

result_df = jt.JSONTools().flatten().execute(df)
# result_df columns: ["name", "age"]

Polars Series

series = pl.Series("data", [
    '{"a": {"b": 1}}',
    '{"a": {"b": 2}}',
])

result_series = jt.JSONTools().flatten().execute(series)

PyArrow Table

Same rule as Pandas/Polars: flatten a struct-typed column, not a plain string column holding JSON text.

import pyarrow as pa

table = pa.table({
    "user": pa.array([{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}])
})

result_table = jt.JSONTools().flatten().execute(table)
# result_table columns: ["name", "age"]

PySpark DataFrame

.execute(df) collects the DataFrame to the driver via toPandas(), runs it through the same row-is-a-JSON-object pipeline as Pandas (so a StructType column flattens; a plain string column does not, unless it holds JSON -- see Auto-Expanding JSON-String Columns), then reconstructs a genuine, distributed pyspark.sql.DataFrame via the active SparkSession (#31; auto-discovered via SparkSession.getActiveSession(), raising JsonToolsError if none is found).

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([
    {"name": "Alice", "age": 30},
    {"name": "Bob", "age": 25},
])

result = jt.JSONTools().flatten().execute(df)
print(type(result))  # <class 'pyspark.sql.dataframe.DataFrame'>

Nested StructType columns and .toPandas(): a top-level flat column always flattens correctly, as above. A nested struct column usually flattens correctly too, but its exact behavior through toPandas()'s non-Arrow fallback path (taken automatically when pyarrow isn't installed) can differ from the Arrow-optimized path in edge cases -- e.g. a Row-typed nested field has been observed losing its field names on that fallback path (surfacing as positional 0/1 instead of name/age), a PySpark toPandas() characteristic, not something this library controls. Installing pyarrow avoids the fallback path entirely and is recommended for any nested-struct-heavy workload.

JsonToolsError

Exception class for all errors raised by JSON Tools RS.

import json_tools_rs as jt

try:
    result = jt.JSONTools().flatten().execute("not valid json")
except jt.JsonToolsError as e:
    print(f"Error: {e}")
    # Error: Failed to process JSON string: [E001] JSON parsing failed: ...

Error messages embed a machine-readable code (E001-E008) in square brackets. Note that the Python bindings prepend their own context before the underlying Rust message (e.g. "Failed to process JSON string: ", "Failed to process Python dict: "), so the code is not always the very first characters of str(e) -- check for "[E00x]" as a substring rather than a prefix. See Error Codes for the full reference.

Error Codes Quick Reference

Handling Specific Errors

import json_tools_rs as jt

try:
    result = jt.JSONTools().execute({"a": 1})  # No mode set
except jt.JsonToolsError as e:
    msg = str(e)
    if "[E005]" in msg:
        print("Forgot to call .flatten() or .unflatten()")
    elif "[E001]" in msg:
        print("Invalid JSON input")

Complete Example

import json_tools_rs as jt

# Build once, reuse many times
tools = (jt.JSONTools()
    .flatten()
    .separator("::")
    .lowercase_keys(True)
    .remove_nulls(True)
    .remove_empty_strings(True)
    .key_replacement("r'^user_'", "")
    .auto_convert_types(True)
    .parallel_threshold(50)
    .num_threads(4)
)

# Single dict
result = tools.execute({"User_Name": "Alice", "User_Age": "30"})
# {"name": "Alice", "age": 30}

# Batch of dicts (processed in parallel if >= 50 items)
results = tools.execute([{"data": str(i)} for i in range(1000)])

# JSON string
result = tools.execute('{"User_Name": "Alice", "nested": {"User_Age": "30"}}')

# DataFrame -- each row becomes {"User_Name": ..., "User_Age": ...} (column names
# are the JSON keys); see "DataFrame and Series Support" above for why a column of
# nested dict/struct values flattens but a column of JSON-text strings does not
import pandas as pd
df = pd.DataFrame({
    "User_Name": ["Alice", "Bob"],
    "User_Age": ["30", "25"],
})
df_result = tools.execute(df)