Technology

condense-json 1.0: A Token-Saver for Repetitive JSON Payloads

A small Python library compresses JSON for LLM prompts with user-defined replacement rules, then restores the original strings on the way out. Free and stable.

condense-json 1.0 is a small, dependency-free Python library that compresses JSON payloads for LLM prompts by replacing user-configured string patterns with compact markers and reversing the operation on the way back. The library supports two output shapes for whole-string and partial-string replacements, hits 1.0 with a stable API, and is available on PyPI and as a Debian package. It is a domain-specific tool aimed at reducing token costs in pipelines that pass structured JSON with repeated boilerplate to language models.

What it does

condense-json is a Python library that condenses JSON objects by replacing user-specified string patterns with compact token-efficient markers, and reverses the process on the way back out. The package hit version 1.0 recently and is available on PyPI under a permissive license, with a Debian package (python3-condense-json) tracking the release.

The core functions are condense_json and uncondense_json. The first walks a JSON-like object (nested dicts, lists, and strings) and replaces any occurrences of the configured replacement strings with a compact representation. The second reverses the operation.

The two output shapes

Two encodings are used. If a string consists entirely of a single replacement string, it is replaced with {"$": "replacement_id"}. If a string contains one or more replacement strings mixed with other content, it is replaced with {"$r": [segments...]} where the segments list contains the original string fragments interleaved with the replacement IDs.

The result is a JSON object that looks structurally identical to the input but has shorter strings wherever the configured replacements fired. A payload with 10 mentions of a long system prompt might drop a meaningful fraction of its serialized size, which translates directly into fewer tokens when the JSON is sent as context to a language model.

Why it is worth knowing about

The library is small, dependency-free, and PSF-sponsored. It is the kind of tool that pays for itself in any LLM pipeline that passes structured data with repeated boilerplate. Common targets include system prompts embedded in tool definitions, schema descriptions repeated across records, and shared preamble text in agent trajectories.

The 1.0 release signals that the API is considered stable, which is a meaningful marker for anyone considering it for production pipelines. The library is not a general-purpose JSON compressor, and it is not a replacement for gzip or brotli at the transport layer. It is a domain-specific tool for one specific job: making structured JSON cheaper to feed to a language model.

Where it fits in the broader tooling landscape

The library is not the only tool in this space. JSON minification tools remove whitespace and produce up to 80% size reduction. Transport-layer compression with gzip or brotli adds another 70-90% on top of that. condense-json sits in a different layer, addressing the case where the JSON contains repeated semantic content that the model needs to interpret but the application does not need to transmit verbatim.

The right mental model is that condense-json is one of several tools you might combine in a token-efficient LLM pipeline. It is not a replacement for compression at the transport layer. It is an additional step that pays off when the same long strings appear many times in a single payload, which is the common case for agent trajectories and tool definitions.

The 1.0 release with a stable API makes it a reasonable choice for production pipelines that need predictable behavior over time. The library is small enough to read in an afternoon, and the API surface is small enough that any change is easy to track. That combination is rare in LLM-adjacent tooling, and it is part of why the 1.0 release is worth paying attention to.