{"record":{"id":"54c79f55c5d19628","repo":"quickwit-oss/tantivy","slug":"pre-tokenized-string-support-in-dynamic-fields-is","errorCode":null,"errorMessage":"Pre-tokenized string support in dynamic fields is not yet implemented","messagePattern":"Pre-tokenized string support in dynamic fields is not yet implemented","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/core/json_utils.rs","lineNumber":217,"sourceCode":"                    term_buffer,\n                    ctx.path_to_unordered_id\n                        .get_or_allocate_unordered_id(json_path_writer.as_str()),\n                );\n                term_buffer.append_type_and_fast_value(val);\n                postings_writer.subscribe(doc, 0u32, term_buffer, ctx);\n            }\n            ReferenceValueLeaf::Date(val) => {\n                set_path_id(\n                    term_buffer,\n                    ctx.path_to_unordered_id\n                        .get_or_allocate_unordered_id(json_path_writer.as_str()),\n                );\n                let val = val.truncate(DATE_TIME_PRECISION_INDEXED);\n                term_buffer.append_type_and_fast_value(val);\n                postings_writer.subscribe(doc, 0u32, term_buffer, ctx);\n            }\n            ReferenceValueLeaf::PreTokStr(_) => {\n                unimplemented!(\n                    \"Pre-tokenized string support in dynamic fields is not yet implemented\"\n                )\n            }\n            ReferenceValueLeaf::Bytes(_) => {\n                unimplemented!(\"Bytes support in dynamic fields is not yet implemented\")\n            }\n            ReferenceValueLeaf::Facet(_) => {\n                unimplemented!(\"Facet support in dynamic fields is not yet implemented\")\n            }\n            ReferenceValueLeaf::IpAddr(_) => {\n                unimplemented!(\"IP address support in dynamic fields is not yet implemented\")\n            }\n            ReferenceValueLeaf::Custom(_) => {\n                unimplemented!(\"the JSON field does not support custom field types\")\n            }\n        },\n        ReferenceValue::Array(elements) => {\n            for val in elements {","sourceCodeStart":199,"sourceCodeEnd":235,"githubUrl":"https://github.com/quickwit-oss/tantivy/blob/b5d8deb80c26924e6b007a5b1a7630f35ca64de4/src/core/json_utils.rs#L199-L235","documentation":"index_json_value walks a JSON value's in-memory representation and panics via unimplemented!() when a leaf is a pre-tokenized string (PreTokStr). Dynamic (JSON) fields do not support storing pre-tokenized text; only explicitly schema-typed fields can hold it. The panic is intentional: the feature is not yet implemented.","triggerScenarios":"Indexing a document where a dynamically-typed JSON field's value is a tantivy PreTokStr object (instead of a plain string/number/bool) — via index_document on a JSON field.","commonSituations":"Passing structured objects from another tantivy field type into a JSON field; programmatically building values with pretokenized text and accidentally routing them to dynamic JSON indexing.","solutions":["Declare the field in the schema with a proper text field type that supports pre-tokenized input and index the document against that field instead","Convert the value to a plain string before adding it to the JSON field","Remove the pre-tokenized value from JSON payloads"],"exampleFix":"// before\njson_obj.insert(\"body\", Value::PreTokStr(pretokenized));\n// after\nschema_builder.add_text_field(\"body\", TEXT); // index pretokenized text as a typed field","handlingStrategy":"validation","validationCode":"fn ensure_json_safe(v: &serde_json::Value) -> bool {\n    // plain JSON scalars/objects/arrays are always indexable; PreTokStr only arises\n    // from tantivy Value::PreTokStr — reject non-standard payloads\n    !matches!(v, serde_json::Value::Null) // plus app-level checks for typed payloads\n}\nassert!(ensure_json_safe(&doc_value));","typeGuard":"fn is_pretok_str(v: &tantivy::schema::Value) -> bool {\n    matches!(v.as_value(), tantivy::schema::ReferenceValueLeaf::PreTokStr(_))\n}","tryCatchPattern":"let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| index_writer.add_document(doc)));\nif result.is_err() { eprintln!(\"doc had unsupported dynamic field type (PreTokStr)\"); }","preventionTips":["Only insert plain JSON values (string/number/bool/array/object) into dynamic JSON fields","Index pre-tokenized text via schema-declared text fields","Add document-level validation before index_writer.add_document","Check tantivy docs/release notes before relying on dynamic-field extras"],"tags":["rust","unimplemented","json","dynamic-fields","indexing"],"backgroundTag":"unimplemented-dynamic-field-type","analyzedSha":"b5d8deb80c26924e6b007a5b1a7630f35ca64de4","analyzedAt":"2026-09-05T13:20:51.521Z","contentChangedAt":"2026-09-05T13:20:51.521Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}