apache/druid · error · IllegalStateException

Cannot publish segments with shardSpec

Error message

Cannot publish segments with shardSpec[%s]

What it means

annotateShardSpec supports NumberedShardSpec and SingleDimensionShardSpec annotation paths; NumberedOverwriteShardSpec via core partition handling, and explicitly rejects BucketNumberedShardSpec with this ISE, because bucket-numbered specs (used by MSQ/ingest-internal pipelines) are not publishable through this helper's annotation logic.

Solutions

1. Publish BucketNumberedShardSpec segments through the MSQ/pipeline-specific publishing machinery, not SegmentPublisherHelper
2. Convert bucket-numbered specs to numbered/single-dimension shard specs before publishing, as the MSQ framework does
3. Audit the task code so the correct publisher is selected for the segment type

Example fix

// before
segmentPublisherHelper.annotateShardSpec(segmentsWithBuckets); // throws
// after
// use MSQ stage output publication for BucketNumberedShardSpec segments
msqPublisher.publish(stageSegments);

if (segments.stream().anyMatch(s -> s.getShardSpec() instanceof BucketNumberedShardSpec)) { useMsqPublisher(); }

boolean publishable = !(segment.getShardSpec() instanceof BucketNumberedShardSpec);

try { publish(); } catch (ISE e) { log.error("BucketNumberedShardSpec not publishable here; route to MSQ publisher", e); }

When it happens

Trigger: Publishing segments whose shard specs are BucketNumberedShardSpec through the standard appenderator publish path instead of the intended MSQ publish path.

Common situations: Mixing MSQ-generated segments with the classic task publish path; custom task reusing SegmentPublisherHelper with incompatible shard spec types; framework routing bug passing MSQ segments to a batch task's publisher.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Cannot publish segments due to incomplete time chunk for…
• Mismatched shardSpecs in interval
• announceHistoricalSegments failed with null metadata…
• Column is not multi-valued
• index[ ] >= size[ ] or < 0

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/d5eea5071f4301d2. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/segment/realtime/appenderator/SegmentPublisherHelper.java:102

            segmentsPerInterval
                .stream()
                .filter(segment -> segment.getShardSpec().getPartitionNum() < expectedCorePartitionSetSize)
                .count()
        );
        if (expectedCorePartitionSetSize != actualCorePartitionSetSize) {
          LOG.errorSegments(segmentsPerInterval, "Cannot publish segments due to incomplete time chunk");
          throw new ISE(
              "Cannot publish segments due to incomplete time chunk for interval[%s]. "
              + "Expected [%s] segments in the core partition, but only [%] segments are found. "
              + "See task logs for more details about these segments.",
              interval,
              expectedCorePartitionSetSize,
              actualCorePartitionSetSize
          );
        }
        annotateFn = annotateCorePartitionSetSizeFn(expectedCorePartitionSetSize);
      } else if (firstShardSpec instanceof BucketNumberedShardSpec) {
        throw new ISE("Cannot publish segments with shardSpec[%s]", firstShardSpec);
      } else {
        annotateFn = null;
      }

      if (annotateFn != null) {
        intervalToSegments.put(interval, segmentsPerInterval.stream().map(annotateFn).collect(Collectors.toList()));
      }
    }

    return intervalToSegments.values().stream().flatMap(Collection::stream).collect(Collectors.toSet());
  }

  private static Function<DataSegment, DataSegment> annotateAtomicUpdateGroupFn(int atomicUpdateGroupSize)
  {
    // The segments which are published together consist an atomicUpdateGroup.
    return segment -> {
      final OverwriteShardSpec shardSpec = (OverwriteShardSpec) segment.getShardSpec();
      return segment.withShardSpec(shardSpec.withAtomicUpdateGroupSize((short) atomicUpdateGroupSize));

View on GitHub (pinned to 9b90983fd2)