{"record":{"id":"0bb2a7fcdcabb42d","repo":"quickwit-oss/tantivy","slug":"pre-tokenized-string-support-in-dynamic-fields-is-0bb2a7","errorCode":null,"errorMessage":"Pre-tokenized string support in dynamic fields is not yet implemented","messagePattern":"Pre-tokenized string support in dynamic fields is not yet implemented","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/fastfield/writer.rs","lineNumber":349,"sourceCode":"            ReferenceValueLeaf::Bool(val) => {\n                columnar_writer.record_bool(doc, json_path_writer.as_str(), val);\n            }\n            ReferenceValueLeaf::Date(val) => {\n                columnar_writer.record_datetime(doc, json_path_writer.as_str(), val);\n            }\n            ReferenceValueLeaf::Facet(_) => {\n                unimplemented!(\"Facet support in dynamic fields is not yet implemented\")\n            }\n            ReferenceValueLeaf::Bytes(_) => {\n                // TODO: This can be re added once it is added to the JSON Utils section as well.\n                // columnar_writer.record_bytes(doc, json_path_writer.as_str(), val);\n                unimplemented!(\"Bytes support in dynamic fields is not yet implemented\")\n            }\n            ReferenceValueLeaf::IpAddr(_) => {\n                unimplemented!(\"IP address support in dynamic fields is not yet implemented\")\n            }\n            ReferenceValueLeaf::PreTokStr(_) => {\n                unimplemented!(\n                    \"Pre-tokenized string support in dynamic fields is not yet implemented\"\n                )\n            }\n            ReferenceValueLeaf::Custom(_) => {\n                unimplemented!(\"the JSON field does not support custom field types\")\n            }\n        },\n        ReferenceValue::Array(elements) => {\n            for el in elements {\n                record_json_value_to_columnar_writer(\n                    doc,\n                    el,\n                    remaining_depth_limit,\n                    json_path_writer,\n                    columnar_writer,\n                    tokenizer,\n                );\n            }","sourceCodeStart":331,"sourceCodeEnd":367,"githubUrl":"https://github.com/quickwit-oss/tantivy/blob/b5d8deb80c26924e6b007a5b1a7630f35ca64de4/src/fastfield/writer.rs#L331-L367","documentation":"This panic fires when a PreTokStr (pre-tokenized text) leaf value is passed to the JSON dynamic field columnar writer. Pre-tokenized strings are not supported in dynamic fields' fast-field path, so the arm is marked unimplemented. It is a deliberate library limitation.","triggerScenarios":"Adding a document where a JSON/dynamic field value is a PreTokStr (e.g. from Value::from(PreTokStr) or schema JSON fields fed pre-tokenized values) via add_json_object/add_named_doc or recursive Array traversal.","commonSituations":"Using pre-tokenized text (from an external tokenizer or keyword extraction) inside a catch-all JSON field; programmatically constructing documents mixing pre-tokenized strings into json field raw writers.","solutions":["Store pre-tokenized strings in a dedicated text field defined in the schema, not inside a JSON dynamic field.","Convert the pre-tokenized string back to a plain string before placing it in the JSON object.","Remove pre-tokenized values from JSON documents at ingestion.","Request/await upstream support for PreTokStr in dynamic fields."],"exampleFix":"// before\njson_field.add_json_object(doc, json!({\"title\": PreTokStr::from(...) })); // panic\n// after\nlet title_pre = PreTokStr::from(tokens);\n// use a dedicated schema text field for the pre-tokenized value;\n// keep only plain JSON values inside the JSON field","handlingStrategy":"validation","validationCode":"fn json_has_pretok_values(obj: &serde_json::Value) -> bool {\n    match obj {\n        serde_json::Value::Object(m) => m.values().any(json_has_pretok_values),\n        serde_json::Value::Array(a) => a.iter().any(json_has_pretok_values),\n        _ => false, // PreTokStr arrives as a ReferenceValue, check before wrapping\n    }\n}\n// for ReferenceValue input:\nfn contains_pretok(v: &ReferenceValue) -> bool {\n    match v {\n        ReferenceValue::Leaf(ReferenceValueLeaf::PreTokStr(_)) => true,\n        ReferenceValue::Array(els) => els.iter().any(contains_pretok),\n        ReferenceValue::Object(_) => false,\n        _ => false,\n    }\n}","typeGuard":"fn is_pretok_leaf(v: &ReferenceValueLeaf) -> bool {\n    matches!(v, ReferenceValueLeaf::PreTokStr(_))\n}","tryCatchPattern":null,"preventionTips":["Never place pre-tokenized strings inside JSON dynamic fields; use schema text fields.","Convert PreTokStr to plain strings before JSON ingestion.","Cover document-construction helpers with tests over all leaf types you emit."],"tags":["unimplemented","fastfield","json","pretokstr"],"backgroundTag":"unimplemented-fastfield-json-type","analyzedSha":"b5d8deb80c26924e6b007a5b1a7630f35ca64de4","analyzedAt":"2026-09-05T13:20:51.521Z","contentChangedAt":"2026-09-05T13:20:51.521Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}