apache/druid · error · IllegalStateException

Mismatched shardSpecs in interval

Error message

Mismatched shardSpecs in interval[%s] for segments[%s]

What it means

SegmentPublisherHelper.annotateShardSpec groups segments per interval and computes shard-spec annotations for publication. If the segments in one interval do not all use the same ShardSpec class, the helper cannot derive a consistent annotation and throws this ISE, since mixed shard-spec types within an interval indicate a corrupted or inconsistent segment set.

Solutions

1. Kill and re-run the affected ingestion so all segments for the interval are generated with the same partitioning scheme
2. Clean up orphaned/partial segments for the interval (e.g. via kill tasks or metadata cleanup) before republishing
3. Ensure partitioning configuration (partitionsSpec) is not changed between retries of the same interval

Example fix

// before
// republishing with changed partitionsSpec over leftover segments -> mixed shard specs
// after
// drop existing segments for the interval, then re-run ingestion with consistent partitionsSpec
"partitionsSpec": {"type": "dynamic"} // consistent across all retries

Map<String, Set<String>> byInterval = segments.values().stream().collect(Collectors.groupingBy(s -> s.getInterval().toString(), Collectors.mapping(s -> s.getShardSpec().getClass().getName(), Collectors.toSet())));
if (byInterval.values().stream().anyMatch(s -> s.size() > 1)) { /* abort and re-run */ }

try { publish(); } catch (ISE e) { log.error("mixed shard specs; kill segments and re-run task", e); killSegmentsForInterval(interval); retryIngestion(); }

When it happens

Trigger: Publishing segments where an interval contains segments with different ShardSpec implementations (e.g. some NumberedShardSpec and some HashBasedNumberedShardSpec) — typically after a failed/aborted partial publish mixed with a re-generated set.

Common situations: Corrupted or hand-edited segment metadata in metadata storage; partially published segments left from a previous failed run with different partitioning; task retries reusing stale segment identifiers with new partitioning config.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Cannot publish segments due to incomplete time chunk for…
• Cannot publish segments with shardSpec
• announceHistoricalSegments failed with null metadata…
• Column is not multi-valued
• index[ ] >= size[ ] or < 0

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/10665a62307e3c76. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/segment/realtime/appenderator/SegmentPublisherHelper.java:68

   * - When segment lock is used, the overwriting task should set the proper size of the atomic update group.
   *   See {@link #annotateAtomicUpdateGroupFn}.
   */
  static Set<DataSegment> annotateShardSpec(Set<DataSegment> segments)
  {
    final Map<Interval, List<DataSegment>> intervalToSegments = new HashMap<>();
    segments.forEach(
        segment -> intervalToSegments.computeIfAbsent(segment.getInterval(), k -> new ArrayList<>()).add(segment)
    );

    for (Entry<Interval, List<DataSegment>> entry : intervalToSegments.entrySet()) {
      final Interval interval = entry.getKey();
      final List<DataSegment> segmentsPerInterval = entry.getValue();
      final ShardSpec firstShardSpec = segmentsPerInterval.get(0).getShardSpec();
      final boolean anyMismatch = segmentsPerInterval.stream().anyMatch(
          segment -> segment.getShardSpec().getClass() != firstShardSpec.getClass()
      );
      if (anyMismatch) {
        throw new ISE(
            "Mismatched shardSpecs in interval[%s] for segments[%s]",
            interval,
            segmentsPerInterval
        );
      }
      final Function<DataSegment, DataSegment> annotateFn;
      if (firstShardSpec instanceof OverwriteShardSpec) {
        annotateFn = annotateAtomicUpdateGroupFn(segmentsPerInterval.size());
      } else if (firstShardSpec instanceof BuildingShardSpec) {
        // sanity check
        // BuildingShardSpec is used in non-appending mode. In this mode,
        // the segments in each interval should have contiguous partitionIds,
        // so that they can be queryable (see PartitionHolder.isComplete()).
        int expectedCorePartitionSetSize = segmentsPerInterval.size();
        int actualCorePartitionSetSize = Math.toIntExact(
            segmentsPerInterval
                .stream()
                .filter(segment -> segment.getShardSpec().getPartitionNum() < expectedCorePartitionSetSize)

View on GitHub (pinned to 9b90983fd2)