Pre-tokenized string support in dynamic fields is not yet im

Error message

Pre-tokenized string support in dynamic fields is not yet implemented

What it means

This panic fires when a PreTokStr (pre-tokenized text) leaf value is passed to the JSON dynamic field columnar writer. Pre-tokenized strings are not supported in dynamic fields' fast-field path, so the arm is marked unimplemented. It is a deliberate library limitation.

Source

Thrown at src/fastfield/writer.rs:349

            ReferenceValueLeaf::Bool(val) => {
                columnar_writer.record_bool(doc, json_path_writer.as_str(), val);
            }
            ReferenceValueLeaf::Date(val) => {
                columnar_writer.record_datetime(doc, json_path_writer.as_str(), val);
            }
            ReferenceValueLeaf::Facet(_) => {
                unimplemented!("Facet support in dynamic fields is not yet implemented")
            }
            ReferenceValueLeaf::Bytes(_) => {
                // TODO: This can be re added once it is added to the JSON Utils section as well.
                // columnar_writer.record_bytes(doc, json_path_writer.as_str(), val);
                unimplemented!("Bytes support in dynamic fields is not yet implemented")
            }
            ReferenceValueLeaf::IpAddr(_) => {
                unimplemented!("IP address support in dynamic fields is not yet implemented")
            }
            ReferenceValueLeaf::PreTokStr(_) => {
                unimplemented!(
                    "Pre-tokenized string support in dynamic fields is not yet implemented"
                )
            }
            ReferenceValueLeaf::Custom(_) => {
                unimplemented!("the JSON field does not support custom field types")
            }
        },
        ReferenceValue::Array(elements) => {
            for el in elements {
                record_json_value_to_columnar_writer(
                    doc,
                    el,
                    remaining_depth_limit,
                    json_path_writer,
                    columnar_writer,
                    tokenizer,
                );
            }

View on GitHub (pinned to b5d8deb80c)

Solutions

Store pre-tokenized strings in a dedicated text field defined in the schema, not inside a JSON dynamic field.
Convert the pre-tokenized string back to a plain string before placing it in the JSON object.
Remove pre-tokenized values from JSON documents at ingestion.
Request/await upstream support for PreTokStr in dynamic fields.

Example fix

// before
json_field.add_json_object(doc, json!({"title": PreTokStr::from(...) })); // panic
// after
let title_pre = PreTokStr::from(tokens);
// use a dedicated schema text field for the pre-tokenized value;
// keep only plain JSON values inside the JSON field

Defensive patterns

Strategy: validation

Validate before calling

fn json_has_pretok_values(obj: &serde_json::Value) -> bool {
    match obj {
        serde_json::Value::Object(m) => m.values().any(json_has_pretok_values),
        serde_json::Value::Array(a) => a.iter().any(json_has_pretok_values),
        _ => false, // PreTokStr arrives as a ReferenceValue, check before wrapping
    }
}
// for ReferenceValue input:
fn contains_pretok(v: &ReferenceValue) -> bool {
    match v {
        ReferenceValue::Leaf(ReferenceValueLeaf::PreTokStr(_)) => true,
        ReferenceValue::Array(els) => els.iter().any(contains_pretok),
        ReferenceValue::Object(_) => false,
        _ => false,
    }
}

Type guard

fn is_pretok_leaf(v: &ReferenceValueLeaf) -> bool {
    matches!(v, ReferenceValueLeaf::PreTokStr(_))
}

Prevention

Never place pre-tokenized strings inside JSON dynamic fields; use schema text fields.
Convert PreTokStr to plain strings before JSON ingestion.
Cover document-construction helpers with tests over all leaf types you emit.

When it happens

Trigger: Adding a document where a JSON/dynamic field value is a PreTokStr (e.g. from Value::from(PreTokStr) or schema JSON fields fed pre-tokenized values) via add_json_object/add_named_doc or recursive Array traversal.

Common situations: Using pre-tokenized text (from an external tokenizer or keyword extraction) inside a catch-all JSON field; programmatically constructing documents mixing pre-tokenized strings into json field raw writers.

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/0bb2a7fcdcabb42d. Report an issue: GitHub.