{"record":{"id":"d5eea5071f4301d2","repo":"apache/druid","slug":"cannot-publish-segments-with-shardspec-s","errorCode":null,"errorMessage":"Cannot publish segments with shardSpec[%s]","messagePattern":"Cannot publish segments with shardSpec\\[(.+?)\\]","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"server/src/main/java/org/apache/druid/segment/realtime/appenderator/SegmentPublisherHelper.java","lineNumber":102,"sourceCode":"            segmentsPerInterval\n                .stream()\n                .filter(segment -> segment.getShardSpec().getPartitionNum() < expectedCorePartitionSetSize)\n                .count()\n        );\n        if (expectedCorePartitionSetSize != actualCorePartitionSetSize) {\n          LOG.errorSegments(segmentsPerInterval, \"Cannot publish segments due to incomplete time chunk\");\n          throw new ISE(\n              \"Cannot publish segments due to incomplete time chunk for interval[%s]. \"\n              + \"Expected [%s] segments in the core partition, but only [%] segments are found. \"\n              + \"See task logs for more details about these segments.\",\n              interval,\n              expectedCorePartitionSetSize,\n              actualCorePartitionSetSize\n          );\n        }\n        annotateFn = annotateCorePartitionSetSizeFn(expectedCorePartitionSetSize);\n      } else if (firstShardSpec instanceof BucketNumberedShardSpec) {\n        throw new ISE(\"Cannot publish segments with shardSpec[%s]\", firstShardSpec);\n      } else {\n        annotateFn = null;\n      }\n\n      if (annotateFn != null) {\n        intervalToSegments.put(interval, segmentsPerInterval.stream().map(annotateFn).collect(Collectors.toList()));\n      }\n    }\n\n    return intervalToSegments.values().stream().flatMap(Collection::stream).collect(Collectors.toSet());\n  }\n\n  private static Function<DataSegment, DataSegment> annotateAtomicUpdateGroupFn(int atomicUpdateGroupSize)\n  {\n    // The segments which are published together consist an atomicUpdateGroup.\n    return segment -> {\n      final OverwriteShardSpec shardSpec = (OverwriteShardSpec) segment.getShardSpec();\n      return segment.withShardSpec(shardSpec.withAtomicUpdateGroupSize((short) atomicUpdateGroupSize));","sourceCodeStart":84,"sourceCodeEnd":120,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/segment/realtime/appenderator/SegmentPublisherHelper.java#L84-L120","documentation":"annotateShardSpec supports NumberedShardSpec and SingleDimensionShardSpec annotation paths; NumberedOverwriteShardSpec via core partition handling, and explicitly rejects BucketNumberedShardSpec with this ISE, because bucket-numbered specs (used by MSQ/ingest-internal pipelines) are not publishable through this helper's annotation logic.","triggerScenarios":"Publishing segments whose shard specs are BucketNumberedShardSpec through the standard appenderator publish path instead of the intended MSQ publish path.","commonSituations":"Mixing MSQ-generated segments with the classic task publish path; custom task reusing SegmentPublisherHelper with incompatible shard spec types; framework routing bug passing MSQ segments to a batch task's publisher.","solutions":["Publish BucketNumberedShardSpec segments through the MSQ/pipeline-specific publishing machinery, not SegmentPublisherHelper","Convert bucket-numbered specs to numbered/single-dimension shard specs before publishing, as the MSQ framework does","Audit the task code so the correct publisher is selected for the segment type"],"exampleFix":"// before\nsegmentPublisherHelper.annotateShardSpec(segmentsWithBuckets); // throws\n// after\n// use MSQ stage output publication for BucketNumberedShardSpec segments\nmsqPublisher.publish(stageSegments);","handlingStrategy":"validation","validationCode":"if (segments.stream().anyMatch(s -> s.getShardSpec() instanceof BucketNumberedShardSpec)) { useMsqPublisher(); }","typeGuard":"boolean publishable = !(segment.getShardSpec() instanceof BucketNumberedShardSpec);","tryCatchPattern":"try { publish(); } catch (ISE e) { log.error(\"BucketNumberedShardSpec not publishable here; route to MSQ publisher\", e); }","preventionTips":["Use the MSQ publishing path for MSQ-generated segments","Never mix MSQ segments with classic task publishers","Add shard-spec type checks before publishing"],"tags":["java","segments","publishing"],"backgroundTag":"unsupported-operation","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}