What causes "number of unique terms for tag field {} > {}" in quickwit-oss/quickwit?

Remove the high-cardinality field from `tag_fields` in the index config.

number of unique terms for tag field {} > {}

Error message

number of unique terms for tag field {} > {}

What it means

`try_extract_terms` builds tag term lists from the split's inverted indexes but enforces a per-field cap (`max_terms`) on the number of unique terms summed across indexes. Exceeding it would bloat split metadata, so the extraction fails.

Source

Thrown at quickwit/quickwit-indexing/src/actors/packager.rs:228

///
/// returns None if:
/// - the number of terms exceed MAX_VALUES_PER_TAG_FIELD
/// - some of the terms are not value utf8.
/// - an error occurs.
///
/// Returns None may hurt split pruning and affects performance,
/// but it does not affect Quickwit's result validity.
fn try_extract_terms(
    named_field: &NamedField,
    inv_indexes: &[Arc<InvertedIndexReader>],
    max_terms: usize,
) -> anyhow::Result<Vec<String>> {
    let num_terms = inv_indexes
        .iter()
        .map(|inv_index| inv_index.terms().num_terms())
        .sum::<usize>();
    if num_terms > max_terms {
        bail!(
            "number of unique terms for tag field {} > {}",
            named_field.name,
            max_terms
        );
    }
    let mut terms = Vec::with_capacity(num_terms);
    for inv_index in inv_indexes {
        let mut terms_streamer = inv_index.terms().stream()?;
        while let Some((term_data, _)) = terms_streamer.next() {
            let term = match named_field.field_type {
                FieldType::U64(_) => u64_from_term_data(term_data)?.to_string(),
                FieldType::I64(_) => {
                    tantivy::u64_to_i64(u64_from_term_data(term_data)?).to_string()
                }
                FieldType::F64(_) => {
                    tantivy::u64_to_f64(u64_from_term_data(term_data)?).to_string()
                }
                FieldType::Bool(_) => match u64_from_term_data(term_data)? {

View on GitHub (pinned to a39730c5cd)

Solutions

Remove the high-cardinality field from `tag_fields` in the index config.
Increase the max-terms threshold if you genuinely need many tags (within operational limits).
Pre-aggregate or bucket the field (e.g. hash_mod routing, truncated values) to lower cardinality.

Example fix

// before (index config)
tag_fields: ["request_id"]
// after
tag_fields: ["service", "level"]

Defensive patterns

Strategy: validation

Validate before calling

// before configuring tag fields, estimate cardinality
fn cardinality_within_limit(terms_per_index: &[usize], max_terms: usize) -> bool {
    terms_per_index.iter().sum::<usize>() <= max_terms
}

Try / catch

match create_packaged_split(...) {
    Ok(p) => p,
    Err(e) if e.to_string().contains("number of unique terms") => {
        // drop the offending field from tag_fields and repackage
        ...
    }
    Err(e) => return Err(e.into()),
}

Prevention

Only mark low-cardinality fields (service, level, host) as tag_fields.
Never use IDs or high-entropy fields as tag fields.
Review tag_fields configuration whenever the schema gains new fields.

When it happens

Trigger: Packaging a split whose tag field (per `tag_fields` config) has more unique indexed terms than the configured maximum during `create_packaged_split`.

Common situations: Choosing a high-cardinality field (user_id, request_id, URL) as a tag field; a bug causing a field to be marked as tag field unintentionally.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/006c60b3f6f0a7ad. Report an issue: GitHub.