headroomlabs-ai/headroom · error

doc_json must be JSON: {e}

Error message

doc_json must be JSON: {e}

What it means

Python-binding panic from compact_document_json: the doc_json argument is not parseable by serde_json. Same contract-violation pattern as items_json — the Rust closure panics (surfacing as a Python RuntimeError) rather than returning a fallback, per the no-silent-fallback project rule.

Source

Thrown at crates/headroom-py/src/lib.rs:868

    /// return the compacted document as JSON.
    ///
    /// The walker recursively descends through objects, arrays, and
    /// strings; tabular sub-arrays become rendered CSV+schema strings,
    /// long opaque blobs become `<<ccr:HASH,KIND,SIZE>>` markers (with
    /// originals stashed in this crusher's CCR store, so `ccr_get`
    /// resolves them).
    ///
    /// Distinct from `crush_array_json`: this is the lossless walker
    /// pass without per-array lossy crushing — useful when the caller
    /// wants document-shape compaction (forms, configs, mixed records)
    /// rather than statistical row drop.
    fn compact_document_json(&self, py: Python<'_>, doc_json: &str) -> String {
        // Heavy: JSON parse + recursive walker + tabular compaction +
        // re-serialize. None of it touches Python; release the GIL.
        let doc_json = doc_json.to_string();
        py.detach(|| {
            let parsed: serde_json::Value = serde_json::from_str(&doc_json)
                .unwrap_or_else(|e| panic!("doc_json must be JSON: {e}"));
            let mut dc = DocumentCompactor::new().with_config(CompactConfig {
                classify: ClassifyConfig {
                    emit_opaque_markers: self.inner.config.opaque_markers_enabled(),
                    ..ClassifyConfig::default()
                },
                ..CompactConfig::default()
            });
            if let Some(store) = self.inner.ccr_store() {
                dc = dc.with_ccr_store(store.clone());
            }
            let out = dc.compact(parsed);
            serde_json::to_string(&out).expect("serialize compacted document")
        })
    }

    /// Look up an original payload by CCR hash.
    ///
    /// When the lossy path drops rows, it stashes the **full original**

View on GitHub (pinned to 322425c43b)

Solutions

1. Serialize with json.dumps(doc) before calling.
2. Validate provenance: json.loads(doc_json) on the Python side first, or fix the upstream writer that produced truncated JSON.
3. Handle the empty case explicitly (pass '{}' or skip the call) instead of ''.
4. Catch RuntimeError around the call if malformed input is possible in production paths, and log the offending string length/prefix for diagnosis.

Example fix

# before
out = hr.compact_document_json(str(doc))

# after
import json
out = hr.compact_document_json(json.dumps(doc))

# Validate before calling
import json
def safe_compact(hr, doc_json: str) -> str:
    try:
        json.loads(doc_json)
    except json.JSONDecodeError as e:
        raise ValueError(f"doc_json is not valid JSON: {e}") from e
    return hr.compact_document_json(doc_json)

def is_json_string(s: str) -> bool:
    try:
        json.loads(s)
        return True
    except (json.JSONDecodeError, TypeError):
        return False

try:
    out = hr.compact_document_json(json.dumps(doc))
except RuntimeError as e:
    log.error("compact_document_json failed: %s", e)
    raise

When it happens

Trigger: Calling compact_document_json(doc_json) with str()/repr() of a Python dict, a truncated file read, empty string, or otherwise malformed JSON text.

Common situations: Forgetting json.dumps on the dict; slurping a truncated/partially-written JSON file; passing text with a BOM or encoding damage; passing an empty string for 'no document'.

Related errors

• items_json must be JSON: {e}
• items_json must be a JSON array, got {}
• invalid pipeline config TOML: {0}
• recommendations file not found: {0}
• target_savings must be between 0 and 1

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/1af6a2e0d1eb60c5. Report an issue: GitHub.