apache/druid · error · RE

Index metadata doesn't exist for segment [%s]. Try providing

Error message

Index metadata doesn't exist for segment [%s]. Try providing explicit rollup, queryGranularity, dimensionsSpec, and metricsSpec.

What it means

CompactionTask.ExistingSegmentAnalyzer reads segment metadata to infer rollup/queryGranularity/dimensions/metrics. Older segments (or tombstone-adjacent storage) can lack index metadata; without it the analyzer cannot auto-derive the compaction spec, so it throws a Redis/RE telling the user to specify those fields explicitly.

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java:1037

    public void fetchAndProcessIfNeeded()
    {
      if (!shouldDownloadSegments()) {
        // Nothing to do; short-circuit and don't fetch segments.
        return;
      }

      multiValuedDimensions = new HashSet<>();
      final List<Pair<DataSegment, Supplier<ResourceHolder<QueryableIndex>>>> segments = sortSegmentsListNewestFirst();

      for (Pair<DataSegment, Supplier<ResourceHolder<QueryableIndex>>> segmentPair : segments) {
        final DataSegment dataSegment = segmentPair.lhs;

        try (final ResourceHolder<QueryableIndex> queryableIndexHolder = segmentPair.rhs.get()) {
          final QueryableIndex index = queryableIndexHolder.get();

          if (index != null) { // Avoid tombstones (null QueryableIndex)
            if (index.getMetadata() == null) {
              throw new RE(
                  "Index metadata doesn't exist for segment [%s]. Try providing explicit rollup, "
                  + "queryGranularity, dimensionsSpec, and metricsSpec.", dataSegment.getId()
              );
            }

            processRollup(index);
            processQueryGranularity(index);
            processDimensionsSpec(index);
            processMetricsSpec(index);
            processMultiValuedDimensions(index);
            processProjections(index);
          }
        }
      }
    }

    public Boolean getRollup()
    {

View on GitHub (pinned to 9b90983fd2)

Solutions

Provide explicit rollup, queryGranularity, dimensionsSpec, and metricsSpec in the compaction tuningConfig/spec so metadata isn't needed.
Re-ingest the legacy segments with a current Druid version to regenerate metadata, then compact.
Exclude the affected segments/intervals from compaction until they are rewritten.

Example fix

// before: relying on auto-detection
{ "type": "compaction", ... } // no granularitySpec
// after
{
  "granularitySpec": {
    "segmentGranularity": "day",
    "queryGranularity": "none",
    "rollup": false
  },
  "dimensionsSpec": { "dimensions": ["dim1"] },
  "metricsSpec": [{ "type": "count", "name": "count" }]
}

Defensive patterns

Strategy: fallback

Validate before calling

// pre-check one segment
QueryableIndex idx = ...;
if (idx == null || idx.getMetadata() == null) {
  // require explicit rollup/queryGranularity/dimensionsSpec/metricsSpec in compaction config
}

Try / catch

try { runCompaction(spec); } catch (RE e) { if (e.getMessage().contains("Index metadata doesn't exist")) { add explicit schema fields and rerun; } else throw e; }

Prevention

For segments from old Druid versions, always specify the full schema in compaction specs.
Re-ingest legacy data to regenerate index metadata before enabling auto-detection.
Spot-check QueryableIndex.getMetadata() on a sample segment before configuring auto-compaction.

When it happens

Trigger: Compacting segments written by very old Druid versions (pre-metadata) or otherwise missing QueryableIndex.getMetadata(); running auto-compaction over segments lacking metadata and no explicit dimensionsSpec/metricsSpec/rollup/queryGranularity in the compaction config.

Common situations: Clusters upgraded across major versions with long-lived historical segments; segments produced by external tools; compaction configs relying on auto-detection for legacy data.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/4ed9dd78473cca1f. Report an issue: GitHub.