retention policy requires a timestamp field, but doc mapping

Error message

retention policy requires a timestamp field, but doc mapping does not declare one

What it means

A retention policy deletes documents based on their timestamp field. validate_index_config therefore requires that, whenever a retention policy is present, the doc mapping declares a timestamp_field; otherwise there is no column to evaluate retention against, and config build fails.

Source

Thrown at quickwit/quickwit-config/src/index_config/mod.rs:692

/// to a method on `IndexConfig` so we can reuse it for validating index templates.
pub(super) fn validate_index_config(
    doc_mapping: &DocMapping,
    indexing_settings: &IndexingSettings,
    search_settings: &SearchSettings,
    retention_policy_opt: &Option<RetentionPolicy>,
) -> anyhow::Result<()> {
    // Note: this needs a deep refactoring to separate the doc mapping configuration,
    // and doc mapper implementations.
    // TODO see if we should store the byproducton the IndexConfig.
    build_doc_mapper(doc_mapping, search_settings)?;

    indexing_settings.merge_policy.validate()?;
    indexing_settings.resources.validate()?;

    if let Some(retention_policy) = retention_policy_opt {
        retention_policy.validate()?;

        ensure!(
            doc_mapping.timestamp_field.is_some(),
            "retention policy requires a timestamp field, but doc mapping does not declare one"
        );
    }
    Ok(())
}

/// Returns the updated doc mapping and a boolean indicating whether a mutation occurred.
///
/// The logic goes as follows:
/// 1. If the new doc mapping is the same as the current doc mapping, ignoring their UIDs, returns
///    the current doc mapping and `false`, indicating that no mutation occurred.
/// 2. If the new doc mapping is different from the current doc mapping, verifies the following
///    constraints before returning the new doc mapping and `true`, indicating that a mutation
///    occurred:
///    - The doc mapping UID should differ from the current one
///    - The timestamp field should remain the same
///    - The tokenizers should be a superset of the current tokenizers

View on GitHub (pinned to a39730c5cd)

Solutions

Declare `timestamp_field` in doc_mapping pointing at a valid datetime field
Or remove the retention policy if time-based deletion is not needed
Ensure the referenced field has a datetime fast/typed mapping so retention can evaluate it

Example fix

# before
doc_mapping:
  field_mappings:
    - name: ts
      type: datetime
retention:
  period: 30 days
# after
doc_mapping:
  timestamp_field: ts
  field_mappings:
    - name: ts
      type: datetime
retention:
  period: 30 days

Defensive patterns

Strategy: validation

Validate before calling

# Python precheck
if cfg.get("retention") and not cfg.get("doc_mapping", {}).get("timestamp_field"):
    raise ValueError("retention requires doc_mapping.timestamp_field")

Prevention

Declare timestamp_field whenever you use retention, partitioning or pruning features
Validate index configs with quickwit's config validation before submitting
Template retention+timestamp together in index config generators

When it happens

Trigger: Submitting an index config (via CreateIndex, IndexConfig::load, or config update validation) that sets a `retention` policy while `doc_mapping.timestamp_field` is absent.

Common situations: Adding a retention policy to an existing index that never had a timestamp field; copying a retention snippet into a logs config without declaring timestamp_field; schema generated from data that omitted the timestamp field.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/bfb5ba865fb5f72e. Report an issue: GitHub.