{"record":{"id":"37823386578fd459","repo":"quickwit-oss/tantivy","slug":"term-dict-returned-non-utf-8","errorCode":null,"errorMessage":"term dict returned non-UTF-8","messagePattern":"term dict returned non-UTF-8","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/aggregation/bucket/multi_terms/mod.rs","lineNumber":1173,"sourceCode":"/// Mirrors the logic in `composite/collector.rs:resolve_term` but emits\n/// [`IntermediateKey`] instead of [`CompositeIntermediateKey`].\nfn resolve_column_value(\n    val: u64,\n    col_type: &ColumnType,\n    str_dict_column: &Option<StrColumn>,\n    col: &Column<u64>,\n) -> crate::Result<IntermediateKey> {\n    match col_type {\n        ColumnType::Str => {\n            let fallback_dict = Dictionary::empty();\n            let term_dict = str_dict_column\n                .as_ref()\n                .map(|c| c.dictionary())\n                .unwrap_or_else(|| &fallback_dict);\n            let mut buffer = Vec::new();\n            term_dict.ord_to_term(val, &mut buffer)?;\n            Ok(IntermediateKey::Str(\n                String::from_utf8(buffer).expect(\"term dict returned non-UTF-8\"),\n            ))\n        }\n        ColumnType::DateTime => {\n            let val = i64::from_u64(val);\n            let date = format_date(val)?;\n            Ok(IntermediateKey::Str(date))\n        }\n        ColumnType::Bool => Ok(IntermediateKey::Bool(bool::from_u64(val))),\n        ColumnType::IpAddr => {\n            let compact_space_accessor = col\n                .values\n                .clone()\n                .downcast_arc::<CompactSpaceU64Accessor>()\n                .map_err(|_| {\n                    TantivyError::AggregationError(\n                        crate::aggregation::AggregationError::InternalError(\n                            \"Type mismatch: Could not downcast to CompactSpaceU64Accessor\"\n                                .to_string(),","sourceCodeStart":1155,"sourceCodeEnd":1191,"githubUrl":"https://github.com/quickwit-oss/tantivy/blob/b5d8deb80c26924e6b007a5b1a7630f35ca64de4/src/aggregation/bucket/multi_terms/mod.rs#L1155-L1191","documentation":"A panic from `String::from_utf8(...).expect(\"term dict returned non-UTF-8\")` in multi-terms aggregation. The code converts a term dictionary's `ord_to_term` output bytes into a Rust String, assuming the index's term dictionary only stores valid UTF-8. A panic means the dictionary returned bytes that are not valid UTF-8, indicating index corruption or a mismatched column type.","triggerScenarios":"Running `IntermediateTermBucketEntry` finalization over a `ColumnType::Str` column whose term dictionary (`term_dict.ord_to_term`) yields bytes that fail `String::from_utf8` — typically a corrupted or externally-written/legacy index segment.","commonSituations":"Corrupted index files after a crash or manual copy; reading an index produced by a different/older writer version with non-UTF-8 encoded text columns; mounting a data dir written by another search engine.","solutions":["Validate/repair the index: run the platform's index validation or reindex the affected field from source data.","Check that the aggregating field is a UTF-8 text/keyword column, not a binary or legacy column.","Confirm the index was written by a compatible version of this engine; upgrade or re-migrate the index.","If you maintain the code, replace the expect with from_utf8_lossy or a Result-returning error."],"exampleFix":"// before\nString::from_utf8(buffer).expect(\"term dict returned non-UTF-8\")\n// after\nString::from_utf8(buffer).map_err(|_| {\n    crate::AggregationError::Internal(\"term dict returned non-UTF-8\".to_string())\n})?","handlingStrategy":"validation","validationCode":"// Validate index health / field encoding before aggregating.\nfn validate_term_buffer_is_utf8(buffer: &[u8]) -> bool {\n    std::str::from_utf8(buffer).is_ok()\n}\n// Also ensure the aggregating column is ColumnType::Str backed by valid UTF-8 data.","typeGuard":"fn is_utf8(b: &[u8]) -> bool { std::str::from_utf8(b).is_ok() }","tryCatchPattern":"// Wrap the aggregation call; panics surface as search failures.\nlet res = std::panic::catch_unwind(|| run_multi_terms_agg(...));\nmatch res {\n    Ok(r) => r,\n    Err(_) => fallback_to_composite_agg(),\n}","preventionTips":["Run index validation/repair tooling after crashes or manual file copies.","Only aggregate over UTF-8 text/keyword columns; never index binary data into keyword fields.","Keep writer and reader engine versions compatible; re-migrate old indexes."],"tags":["rust","panic","utf-8","index-corruption","aggregation"],"backgroundTag":"invalid-utf8-decode","analyzedSha":"b5d8deb80c26924e6b007a5b1a7630f35ca64de4","analyzedAt":"2026-09-05T13:20:51.521Z","contentChangedAt":"2026-09-05T13:20:51.521Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}