apache/druid · error · IllegalStateException

interval %s, bucketId %s mismatched shard specs: %s and %s

Error message

interval %s, bucketId %s mismatched shard specs: %s and %s

What it means

PartialGenericSegmentMergeTask builds a map from (interval, bucketId) to ShardSpec from its input partitions. If two partitions share the same interval and bucketId but carry different ShardSpecs, the shard layout is inconsistent and the task throws ISE rather than merging conflicting data.

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/PartialGenericSegmentMergeTask.java:99

    this.intervalAndIntegerToShardSpec = createIntervalAndIntegerToShardSpec(
        ingestionSchema.getIOConfig().getPartitionLocations()
    );
  }

  private static Table<Interval, Integer, BuildingShardSpec<?>> createIntervalAndIntegerToShardSpec(
      List<PartitionLocation> partitionLocations
  )
  {
    final Table<Interval, Integer, BuildingShardSpec<?>> intervalAndIntegerToShardSpec = HashBasedTable.create();

    partitionLocations.forEach(
        p -> {
          final ShardSpec currShardSpec = intervalAndIntegerToShardSpec.get(p.getInterval(), p.getBucketId());
          if (currShardSpec == null) {
            intervalAndIntegerToShardSpec.put(p.getInterval(), p.getBucketId(), p.getShardSpec());
          } else {
            if (!p.getShardSpec().equals(currShardSpec)) {
              throw new ISE(
                  "interval %s, bucketId %s mismatched shard specs: %s and %s",
                  p.getInterval(),
                  p.getBucketId(),
                  currShardSpec,
                  p.getShardSpec()
              );
            }
          }
        }
    );

    return intervalAndIntegerToShardSpec;
  }

  @JsonProperty("spec")
  private PartialSegmentMergeIngestionSpec getIngestionSchema()
  {
    return ingestionSchema;

View on GitHub (pinned to 9b90983fd2)

Solutions

Kill and rerun the whole supervisor parallel task so all intermediate partial tasks are regenerated consistently.
Compare the shard specs in the message to identify the two conflicting partitions and locate the stale segments.
Ensure partial merge tasks are not reused across different ingestion runs.
Check for duplicated segment push from concurrent failed/retried allocation sub-tasks.

Defensive patterns

Strategy: try-catch

Try / catch

catch (IllegalStateException e) { if (e.getMessage().contains("mismatched shard specs")) { /* abort and rerun the full supervisor task */ } else { throw e; } }

Prevention

Never reuse intermediate partial merge tasks or their input segments across ingestion runs.
Rerun the entire supervisor task when intermediate phases fail.
Avoid manual manipulation of intermediate segments in deep storage.

When it happens

Trigger: An intermediate 'partial generic segment merge' input set (from PartialSegmentAllocation/InputSourcePartitioning) where two input segments for the same interval/bucket were produced with different shard specs - typically from a corrupted or manually manipulated intermediate state, or from mixing outputs of two different allocation runs.

Common situations: Retrying/compacting compounds of intermediate segment merge tasks after a failed run; manually copying or re-submitting partial merge tasks with stale input segments; supervisor task state inconsistency after task retries.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/ecedb70c5215a846. Report an issue: GitHub.