{"record":{"id":"006c60b3f6f0a7ad","repo":"quickwit-oss/quickwit","slug":"number-of-unique-terms-for-tag-field","errorCode":null,"errorMessage":"number of unique terms for tag field {} > {}","messagePattern":"number of unique terms for tag field (.+?) > (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-indexing/src/actors/packager.rs","lineNumber":228,"sourceCode":"///\n/// returns None if:\n/// - the number of terms exceed MAX_VALUES_PER_TAG_FIELD\n/// - some of the terms are not value utf8.\n/// - an error occurs.\n///\n/// Returns None may hurt split pruning and affects performance,\n/// but it does not affect Quickwit's result validity.\nfn try_extract_terms(\n    named_field: &NamedField,\n    inv_indexes: &[Arc<InvertedIndexReader>],\n    max_terms: usize,\n) -> anyhow::Result<Vec<String>> {\n    let num_terms = inv_indexes\n        .iter()\n        .map(|inv_index| inv_index.terms().num_terms())\n        .sum::<usize>();\n    if num_terms > max_terms {\n        bail!(\n            \"number of unique terms for tag field {} > {}\",\n            named_field.name,\n            max_terms\n        );\n    }\n    let mut terms = Vec::with_capacity(num_terms);\n    for inv_index in inv_indexes {\n        let mut terms_streamer = inv_index.terms().stream()?;\n        while let Some((term_data, _)) = terms_streamer.next() {\n            let term = match named_field.field_type {\n                FieldType::U64(_) => u64_from_term_data(term_data)?.to_string(),\n                FieldType::I64(_) => {\n                    tantivy::u64_to_i64(u64_from_term_data(term_data)?).to_string()\n                }\n                FieldType::F64(_) => {\n                    tantivy::u64_to_f64(u64_from_term_data(term_data)?).to_string()\n                }\n                FieldType::Bool(_) => match u64_from_term_data(term_data)? {","sourceCodeStart":210,"sourceCodeEnd":246,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-indexing/src/actors/packager.rs#L210-L246","documentation":"`try_extract_terms` builds tag term lists from the split's inverted indexes but enforces a per-field cap (`max_terms`) on the number of unique terms summed across indexes. Exceeding it would bloat split metadata, so the extraction fails.","triggerScenarios":"Packaging a split whose tag field (per `tag_fields` config) has more unique indexed terms than the configured maximum during `create_packaged_split`.","commonSituations":"Choosing a high-cardinality field (user_id, request_id, URL) as a tag field; a bug causing a field to be marked as tag field unintentionally.","solutions":["Remove the high-cardinality field from `tag_fields` in the index config.","Increase the max-terms threshold if you genuinely need many tags (within operational limits).","Pre-aggregate or bucket the field (e.g. hash_mod routing, truncated values) to lower cardinality."],"exampleFix":"// before (index config)\ntag_fields: [\"request_id\"]\n// after\ntag_fields: [\"service\", \"level\"]","handlingStrategy":"validation","validationCode":"// before configuring tag fields, estimate cardinality\nfn cardinality_within_limit(terms_per_index: &[usize], max_terms: usize) -> bool {\n    terms_per_index.iter().sum::<usize>() <= max_terms\n}","typeGuard":null,"tryCatchPattern":"match create_packaged_split(...) {\n    Ok(p) => p,\n    Err(e) if e.to_string().contains(\"number of unique terms\") => {\n        // drop the offending field from tag_fields and repackage\n        ...\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Only mark low-cardinality fields (service, level, host) as tag_fields.","Never use IDs or high-entropy fields as tag fields.","Review tag_fields configuration whenever the schema gains new fields."],"tags":["packaging","tags","cardinality"],"backgroundTag":"value-out-of-range","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}