Pre-tokenized string support in dynamic fields is not yet im

Error message

Pre-tokenized string support in dynamic fields is not yet implemented

What it means

index_json_value walks a JSON value's in-memory representation and panics via unimplemented!() when a leaf is a pre-tokenized string (PreTokStr). Dynamic (JSON) fields do not support storing pre-tokenized text; only explicitly schema-typed fields can hold it. The panic is intentional: the feature is not yet implemented.

Source

Thrown at src/core/json_utils.rs:217

                    term_buffer,
                    ctx.path_to_unordered_id
                        .get_or_allocate_unordered_id(json_path_writer.as_str()),
                );
                term_buffer.append_type_and_fast_value(val);
                postings_writer.subscribe(doc, 0u32, term_buffer, ctx);
            }
            ReferenceValueLeaf::Date(val) => {
                set_path_id(
                    term_buffer,
                    ctx.path_to_unordered_id
                        .get_or_allocate_unordered_id(json_path_writer.as_str()),
                );
                let val = val.truncate(DATE_TIME_PRECISION_INDEXED);
                term_buffer.append_type_and_fast_value(val);
                postings_writer.subscribe(doc, 0u32, term_buffer, ctx);
            }
            ReferenceValueLeaf::PreTokStr(_) => {
                unimplemented!(
                    "Pre-tokenized string support in dynamic fields is not yet implemented"
                )
            }
            ReferenceValueLeaf::Bytes(_) => {
                unimplemented!("Bytes support in dynamic fields is not yet implemented")
            }
            ReferenceValueLeaf::Facet(_) => {
                unimplemented!("Facet support in dynamic fields is not yet implemented")
            }
            ReferenceValueLeaf::IpAddr(_) => {
                unimplemented!("IP address support in dynamic fields is not yet implemented")
            }
            ReferenceValueLeaf::Custom(_) => {
                unimplemented!("the JSON field does not support custom field types")
            }
        },
        ReferenceValue::Array(elements) => {
            for val in elements {

View on GitHub (pinned to b5d8deb80c)

Solutions

Declare the field in the schema with a proper text field type that supports pre-tokenized input and index the document against that field instead
Convert the value to a plain string before adding it to the JSON field
Remove the pre-tokenized value from JSON payloads

Example fix

// before
json_obj.insert("body", Value::PreTokStr(pretokenized));
// after
schema_builder.add_text_field("body", TEXT); // index pretokenized text as a typed field

Defensive patterns

Strategy: validation

Validate before calling

fn ensure_json_safe(v: &serde_json::Value) -> bool {
    // plain JSON scalars/objects/arrays are always indexable; PreTokStr only arises
    // from tantivy Value::PreTokStr — reject non-standard payloads
    !matches!(v, serde_json::Value::Null) // plus app-level checks for typed payloads
}
assert!(ensure_json_safe(&doc_value));

Type guard

fn is_pretok_str(v: &tantivy::schema::Value) -> bool {
    matches!(v.as_value(), tantivy::schema::ReferenceValueLeaf::PreTokStr(_))
}

Try / catch

let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| index_writer.add_document(doc)));
if result.is_err() { eprintln!("doc had unsupported dynamic field type (PreTokStr)"); }

Prevention

Only insert plain JSON values (string/number/bool/array/object) into dynamic JSON fields
Index pre-tokenized text via schema-declared text fields
Add document-level validation before index_writer.add_document
Check tantivy docs/release notes before relying on dynamic-field extras

When it happens

Trigger: Indexing a document where a dynamically-typed JSON field's value is a tantivy PreTokStr object (instead of a plain string/number/bool) — via index_document on a JSON field.

Common situations: Passing structured objects from another tantivy field type into a JSON field; programmatically building values with pretokenized text and accidentally routing them to dynamic JSON indexing.

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/54c79f55c5d19628. Report an issue: GitHub.