quickwit-oss/tantivy · error

doc encoding failed. This is a bug

Error message

doc encoding failed. This is a bug

What it means

This panic occurs inside `Document::to_json`, which serializes the document's named fields via `serde_json::to_string`. The API documents that encoding a document cannot fail, so a `serde_json` error is treated as an internal bug and surfaced with `expect("doc encoding failed. This is a bug")`. In practice it means serialization produced an error despite the document model being JSON-safe.

Source

Thrown at src/schema/document/mod.rs:269

    fn to_named_doc(&self, schema: &Schema) -> NamedFieldDocument {
        let mut field_map = BTreeMap::new();
        for (field, field_values) in self.get_sorted_field_values() {
            let field_name = schema.get_field_name(field);
            let values: Vec<OwnedValue> = field_values
                .into_iter()
                .map(|val| OwnedValue::from(val.as_value()))
                .collect();
            field_map.insert(field_name.to_string(), values);
        }
        NamedFieldDocument(field_map)
    }

    /// Encode the doc in JSON.
    ///
    /// Encoding a document cannot fail.
    fn to_json(&self, schema: &Schema) -> String {
        serde_json::to_string(&self.to_named_doc(schema))
            .expect("doc encoding failed. This is a bug")
    }
}

pub(crate) mod type_codes {
    pub const TEXT_CODE: u8 = 0;
    pub const U64_CODE: u8 = 1;
    pub const I64_CODE: u8 = 2;
    pub const HIERARCHICAL_FACET_CODE: u8 = 3;
    pub const BYTES_CODE: u8 = 4;
    pub const DATE_CODE: u8 = 5;
    pub const F64_CODE: u8 = 6;
    pub const EXT_CODE: u8 = 7;

    #[deprecated]
    pub const JSON_OBJ_CODE: u8 = 8; // Replaced by the `OBJECT_CODE`.
    pub const BOOL_CODE: u8 = 9;
    pub const IP_CODE: u8 = 10;
    pub const NULL_CODE: u8 = 11;

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Audit any custom `Document` or `Value` implementation to ensure `to_named_doc` only returns `serde_json::Value`-compatible data (string keys, JSON scalar/array/object values).
2. Test document serialization in isolation with `serde_json::to_string(&doc.to_named_doc(schema))` and handle/display the actual error to identify the offending value.
3. Upgrade/align tantivy and serde_json versions if using a patched or mismatched dependency tree.
4. If it reproduces with stock tantivy values, file a bug with a minimal reproducing document — the library treats this path as infallible by contract.

Example fix

// before: custom value with non-string map key
NamedDoc::from(map_with_integer_keys)

// after: stringify keys before handing to serde
NamedDoc::from(map.into_iter().map(|(k, v)| (k.to_string(), v)).collect())

fn check_doc_serializable(doc: &Document, schema: &Schema) -> Result<(), serde_json::Error> {
    serde_json::to_string(&doc.to_named_doc(schema))
        .map(|_| ())
        .map_err(|e| e)
}

fn is_json_safe(v: &serde_json::Value) -> bool {
    match v {
        serde_json::Value::Object(m) => m.iter().all(|(k, v)| !k.is_empty() && is_json_safe(v)),
        serde_json::Value::Array(a) => a.iter().all(is_json_safe),
        _ => true,
    }
}

let json = std::panic::catch_unwind(|| doc.to_json(schema))
    .map_err(|_| anyhow::anyhow!("doc serialization failed — inspect custom Value impl"))?;

When it happens

Trigger: Calling `to_json(&schema)` (directly or indirectly through document printing/debugging) when `serde_json::to_string` fails — realistically only from a poisoned/broken custom `Value` implementation that yields non-serializable data, or a memory/allocation failure inside serde_json.

Common situations: Custom `Document`/`Value` trait implementations that produce values serde cannot serialize (e.g. maps with non-string keys when not using serde's map-key escaping correctly), or using a forked/patched serde_json. Extremely rare for standard users.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• JSON serialization for IndexMeta should never fail.
• Invalid type code in JSON term
• expected json type in term
• No multivalued index is allowed when stacking column index
• Internal Error: Called get_val of empty column.

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/525e6e2e815067c9. Report an issue: GitHub.