quickwit-oss/tantivy · error

term dict returned non-UTF-8

Error message

term dict returned non-UTF-8

What it means

A panic from `String::from_utf8(...).expect("term dict returned non-UTF-8")` in multi-terms aggregation. The code converts a term dictionary's `ord_to_term` output bytes into a Rust String, assuming the index's term dictionary only stores valid UTF-8. A panic means the dictionary returned bytes that are not valid UTF-8, indicating index corruption or a mismatched column type.

Source

Thrown at src/aggregation/bucket/multi_terms/mod.rs:1173

/// Mirrors the logic in `composite/collector.rs:resolve_term` but emits
/// [`IntermediateKey`] instead of [`CompositeIntermediateKey`].
fn resolve_column_value(
    val: u64,
    col_type: &ColumnType,
    str_dict_column: &Option<StrColumn>,
    col: &Column<u64>,
) -> crate::Result<IntermediateKey> {
    match col_type {
        ColumnType::Str => {
            let fallback_dict = Dictionary::empty();
            let term_dict = str_dict_column
                .as_ref()
                .map(|c| c.dictionary())
                .unwrap_or_else(|| &fallback_dict);
            let mut buffer = Vec::new();
            term_dict.ord_to_term(val, &mut buffer)?;
            Ok(IntermediateKey::Str(
                String::from_utf8(buffer).expect("term dict returned non-UTF-8"),
            ))
        }
        ColumnType::DateTime => {
            let val = i64::from_u64(val);
            let date = format_date(val)?;
            Ok(IntermediateKey::Str(date))
        }
        ColumnType::Bool => Ok(IntermediateKey::Bool(bool::from_u64(val))),
        ColumnType::IpAddr => {
            let compact_space_accessor = col
                .values
                .clone()
                .downcast_arc::<CompactSpaceU64Accessor>()
                .map_err(|_| {
                    TantivyError::AggregationError(
                        crate::aggregation::AggregationError::InternalError(
                            "Type mismatch: Could not downcast to CompactSpaceU64Accessor"
                                .to_string(),

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Validate/repair the index: run the platform's index validation or reindex the affected field from source data.
2. Check that the aggregating field is a UTF-8 text/keyword column, not a binary or legacy column.
3. Confirm the index was written by a compatible version of this engine; upgrade or re-migrate the index.
4. If you maintain the code, replace the expect with from_utf8_lossy or a Result-returning error.

Example fix

// before
String::from_utf8(buffer).expect("term dict returned non-UTF-8")
// after
String::from_utf8(buffer).map_err(|_| {
    crate::AggregationError::Internal("term dict returned non-UTF-8".to_string())
})?

// Validate index health / field encoding before aggregating.
fn validate_term_buffer_is_utf8(buffer: &[u8]) -> bool {
    std::str::from_utf8(buffer).is_ok()
}
// Also ensure the aggregating column is ColumnType::Str backed by valid UTF-8 data.

fn is_utf8(b: &[u8]) -> bool { std::str::from_utf8(b).is_ok() }

// Wrap the aggregation call; panics surface as search failures.
let res = std::panic::catch_unwind(|| run_multi_terms_agg(...));
match res {
    Ok(r) => r,
    Err(_) => fallback_to_composite_agg(),
}

When it happens

Trigger: Running `IntermediateTermBucketEntry` finalization over a `ColumnType::Str` column whose term dictionary (`term_dict.ord_to_term`) yields bytes that fail `String::from_utf8` — typically a corrupted or externally-written/legacy index segment.

Common situations: Corrupted index files after a crash or manual copy; reading an index produced by a different/older writer version with non-UTF-8 encoded text columns; mounting a data dir written by another search engine.

Related errors

• could not convert to String
• could not convert to String
• dictionary missing for str accessor
• Key length mismatch
• non-zero packed bits fit in u64

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/37823386578fd459. Report an issue: GitHub.