quickwit-oss/tantivy · error

could not convert to String

Error message

could not convert to String

What it means

resolve_term converts a term ordinal back to bytes and then, for Str columns, .expect()s that the bytes are valid UTF-8, panicking with "could not convert to String" if not. The term dictionary is assumed to hold well-formed UTF-8 strings; invalid bytes imply corruption of the dictionary or that the value actually belongs to a non-str column type.

Source

Thrown at src/aggregation/bucket/composite/collector.rs:452

}

fn resolve_term(
    val: u64,
    column_type: &ColumnType,
    str_dict_column: &Option<StrColumn>,
    column: &Column,
) -> crate::Result<CompositeIntermediateKey> {
    let key = if *column_type == ColumnType::Str {
        let fallback_dict = Dictionary::empty();
        let term_dict = str_dict_column
            .as_ref()
            .map(|el| el.dictionary())
            .unwrap_or_else(|| &fallback_dict);

        let mut buffer = Vec::new();
        term_dict.ord_to_term(val, &mut buffer)?;
        CompositeIntermediateKey::Str(
            String::from_utf8(buffer.to_vec()).expect("could not convert to String"),
        )
    } else if *column_type == ColumnType::DateTime {
        let val = i64::from_u64(val);
        CompositeIntermediateKey::DateTime(val)
    } else if *column_type == ColumnType::Bool {
        let val = bool::from_u64(val);
        CompositeIntermediateKey::Bool(val)
    } else if *column_type == ColumnType::IpAddr {
        let compact_space_accessor = column
            .values
            .clone()
            .downcast_arc::<CompactSpaceU64Accessor>()
            .map_err(|_| {
                TantivyError::AggregationError(crate::aggregation::AggregationError::InternalError(
                    "Type mismatch: Could not downcast to CompactSpaceU64Accessor".to_string(),
                ))
            })?;
        let val: u128 = compact_space_accessor.compact_to_u128(val as u32);

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Fix the source data/indexing so str fields contain valid UTF-8; re-index the affected segment.
2. Replace the expect with String::from_utf8_lossy (or return TantivyError::InternalError) to degrade gracefully.
3. Verify the column_type check before this branch — a Bytes column misclassified as Str would land here.
4. Validate the segment/dictionary integrity if corruption is suspected.

Example fix

// before
String::from_utf8(buffer.to_vec()).expect("could not convert to String")
// after
String::from_utf8_lossy(&buffer).into_owned()

// Validate UTF-8 before conversion
match std::str::from_utf8(&buffer) {
    Ok(s) => CompositeIntermediateKey::Str(s.to_string()),
    Err(e) => return Err(TantivyError::InternalError(format!(
        "term is not valid UTF-8: {}", e))),
}

fn valid_utf8_term(bytes: &[u8]) -> Option<&str> {
    std::str::from_utf8(bytes).ok()
}

When it happens

Trigger: Calling resolve_term (via resolve_internal_value_repr) on a Str-typed column whose ord_to_term output is not valid UTF-8: corrupted term dictionary bytes, wrong column type classification (binary data stored as str), or val pointing at an out-of-range ordinal producing garbage.

Common situations: Indexes written by third-party/older tools with non-UTF-8 term bytes; user data injected as raw bytes into a str field; segment corruption after disk issues.

Related errors

• dictionary missing for str accessor
• Key length mismatch
• term dict returned non-UTF-8
• could not convert to String
• unsupported

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/cb2c9c17737d5c99. Report an issue: GitHub.