quickwit-oss/quickwit · error · anyhow::Error

tags collection is only allowed on text fields with the `raw

Error message

tags collection is only allowed on text fields with the `raw` tokenizer

What it means

For text (Str) fields, tags are only allowed when the field is indexed with the `raw` tokenizer. Tokenized text fields have multiple token representations of a value, so tag equality filtering would be unreliable; validate_tag enforces the raw tokenizer for Str fields.

Source

Thrown at quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs:327

/// - the field must be indexed.
fn validate_tag(tag_field_name: &str, schema: &Schema) -> Result<(), anyhow::Error> {
    if tag_field_name.starts_with('.') || tag_field_name.starts_with("\\.") {
        bail!("tag field `{tag_field_name}` should not start with a `.`");
    }
    if tag_field_name.ends_with('.') {
        bail!("tag field `{tag_field_name}` should not end with a `.`");
    }
    let field = schema
        .get_field(tag_field_name)
        .with_context(|| format!("unknown tag field: `{tag_field_name}`"))?;
    let field_type = schema.get_field_entry(field).field_type();
    match field_type {
        FieldType::Str(options) => {
            let tokenizer_opt = options
                .get_indexing_options()
                .map(|text_options: &tantivy::schema::TextFieldIndexing| text_options.tokenizer());
            if tokenizer_opt != Some(RAW_TOKENIZER_NAME) {
                bail!("tags collection is only allowed on text fields with the `raw` tokenizer");
            }
        }
        FieldType::U64(_) | FieldType::I64(_) => {
            // u64 and i64 are accepted as tags.
        }
        _ => {
            // We avoid the bytes / bool / f64 types,
            // as they are generally speaking poor tags and we want to avoid
            // bugs associated to the multiplicity of their representation.
            //
            // (Tags are relying heavily on string manipulation and we want to
            // avoid a "ZRP because you searched you searched for 0.100 instead of 0.1",
            // or `myflag:1`, `myflag:True` instead of `myflag:true`.
            bail!(
                "tags collection is not allowed on `{}` fields",
                field_type.value_type().name().to_lowercase()
            )
        }

View on GitHub (pinned to a39730c5cd)

Solutions

1. Configure the field's text indexing with `tokenizer: raw`.
2. Or use a separate raw-tokenized field (or u64/i64 field) as the tag field.

Example fix

# before
- name: service
  type: text
  indexed: true

tag_fields: [service]

# after
- name: service
  type: text
  fieldnorms: false
  text_type:
    index: true
    tokenizer: raw

tag_fields: [service]

fn check_text_tags_use_raw(cfg: &serde_yaml::Value) -> Result<(), String> {
    let tags: Vec<&str> = cfg["doc_mapping"]["tag_fields"].as_sequence()
        .map(|s| s.iter().filter_map(|v| v.as_str()).collect()).unwrap_or_default();
    if let Some(fields) = cfg["doc_mapping"]["field_mappings"].as_sequence() {
        for f in fields {
            let name = f["name"].as_str().unwrap_or_default();
            if tags.contains(&name) && f["type"].as_str() == Some("text") {
                let tok = f["text_type"]["tokenizer"].as_str().unwrap_or("default");
                if tok != "raw" {
                    return Err(format!("tag field `{name}` (text) must use the raw tokenizer"));
                }
            }
        }
    }
    Ok(())
}

if let Err(e) = QuickwitDocMapper::try_from(&index_config) {
    if e.to_string().contains("raw tokenizer") {
        eprintln!("Set tokenizer: raw on tag text fields: {e}");
    }
    return Err(e);
}

When it happens

Trigger: `QuickwitDocMapper::try_from` where a `tag_fields` entry is a Str field whose text indexing options use a tokenizer other than `raw` (e.g. the default tokenizer), or has no indexing options at all.

Common situations: Making a normal full-text field (e.g. `message`) a tag field, forgetting that tags require raw tokenization.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• duplicated custom tokenizer: `{}`
• custom tokenizer name `{}` should be different from built-in
• tag field `{tag_field_name}` should not start with a `.`
• tag field `{tag_field_name}` should not end with a `.`
• tags collection is not allowed on `{}` fields

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/9388702d813a6bd0. Report an issue: GitHub.