quickwit-oss/tantivy · error

could not convert to String

Error message

could not convert to String

What it means

A panic from `String::from_utf8(term.to_vec()).expect("could not convert to String")` in the terms aggregation finalization. `sorted_ords_to_term_cb` yields raw term bytes from the term dictionary, and the code asserts they are valid UTF-8 before using them as bucket keys. The panic fires when the dictionary returns non-UTF-8 bytes, i.e. the index's text column is corrupt or was written with a non-UTF-8 codec.

Source

Thrown at src/aggregation/bucket/term_agg/mod.rs:1372

            let intermediate_entries: Vec<IntermediateTermBucketEntry> = buckets
                .into_iter()
                .map(|bucket| {
                    into_intermediate_bucket_entry(
                        bucket,
                        reborrow_opt_collector(&mut sub_agg_collector),
                        agg_data,
                    )
                })
                .collect::<crate::Result<_>>()?;

            let mut intermediate_entry_it = intermediate_entries.into_iter();

            term_dict.sorted_ords_to_term_cb(&term_ids[..], |term| {
                let intermediate_entry = intermediate_entry_it.next().unwrap();
                dict.insert(
                    IntermediateKey::Str(
                        String::from_utf8(term.to_vec()).expect("could not convert to String"),
                    ),
                    intermediate_entry,
                );
            })?;

            if term_req.req.min_doc_count == 0 {
                // TODO: Handle rev streaming for descending sorting by keys
                let mut stream = term_dict.stream()?;
                let empty_sub_aggregation =
                    IntermediateAggregationResults::empty_from_req(&term_req.sug_aggregations);
                while stream.advance() {
                    if dict.len() >= term_req.req.segment_size as usize {
                        break;
                    }

                    // Respect allowed filters if present
                    if let Some(allowed_bs) = term_req.allowed_term_ids.as_ref() {
                        if !allowed_bs.contains(stream.term_ord() as u32) {

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Reindex or repair the affected index/segment so the term dictionary contains valid UTF-8 terms.
2. Verify the field's storage type is text/keyword and that no binary bytes are being indexed.
3. Check writer/reader version compatibility for the index format being read.
4. If you maintain the code, use String::from_utf8_lossy or propagate an error instead of panicking.

Example fix

// before
String::from_utf8(term.to_vec()).expect("could not convert to String")
// after
String::from_utf8(term.to_vec())
    .map_err(|_| crate::AggregationError::Internal("non-UTF-8 term".to_string()))?

// Verify terms are valid UTF-8 before building bucket keys.
fn term_is_utf8(term: &[u8]) -> bool {
    std::str::from_utf8(term).is_ok()
}

fn is_utf8_term(b: &[u8]) -> bool { std::str::from_utf8(b).is_ok() }

let res = std::panic::catch_unwind(|| run_terms_agg(...));
if res.is_err() {
    // trigger index repair / reindex of the affected field
    schedule_reindex(field);
}

When it happens

Trigger: Finalizing a terms aggregation where `term_dict.sorted_ords_to_term_cb` invokes the callback with bytes that fail UTF-8 validation — corrupted segment, wrong column type, or index written by an incompatible writer.

Common situations: Torn or manually-copied index directories; ingesting binary data into a keyword field; reading old index formats after a version upgrade.

Related errors

• term dict returned non-UTF-8
• could not convert to String
• dictionary missing for str accessor
• Key length mismatch
• non-zero packed bits fit in u64

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/ba496844d4081eff. Report an issue: GitHub.