{"record":{"id":"73cbcf1684767afd","repo":"apache/druid","slug":"cannot-publish-segments-due-to-incomplete-time-chu","errorCode":null,"errorMessage":"Cannot publish segments due to incomplete time chunk for interval[%s]. Expected [%s] segments in the core partition, but only [%] segments are found. See task logs for more details about these segments.","messagePattern":"Cannot publish segments due to incomplete time chunk for interval\\[(.+?)\\]\\. Expected \\[(.+?)\\] segments in the core partition, but only \\[%\\] segments are found\\. See task logs for more details about these segments\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"critical","filePath":"server/src/main/java/org/apache/druid/segment/realtime/appenderator/SegmentPublisherHelper.java","lineNumber":91,"sourceCode":"      }\n      final Function<DataSegment, DataSegment> annotateFn;\n      if (firstShardSpec instanceof OverwriteShardSpec) {\n        annotateFn = annotateAtomicUpdateGroupFn(segmentsPerInterval.size());\n      } else if (firstShardSpec instanceof BuildingShardSpec) {\n        // sanity check\n        // BuildingShardSpec is used in non-appending mode. In this mode,\n        // the segments in each interval should have contiguous partitionIds,\n        // so that they can be queryable (see PartitionHolder.isComplete()).\n        int expectedCorePartitionSetSize = segmentsPerInterval.size();\n        int actualCorePartitionSetSize = Math.toIntExact(\n            segmentsPerInterval\n                .stream()\n                .filter(segment -> segment.getShardSpec().getPartitionNum() < expectedCorePartitionSetSize)\n                .count()\n        );\n        if (expectedCorePartitionSetSize != actualCorePartitionSetSize) {\n          LOG.errorSegments(segmentsPerInterval, \"Cannot publish segments due to incomplete time chunk\");\n          throw new ISE(\n              \"Cannot publish segments due to incomplete time chunk for interval[%s]. \"\n              + \"Expected [%s] segments in the core partition, but only [%] segments are found. \"\n              + \"See task logs for more details about these segments.\",\n              interval,\n              expectedCorePartitionSetSize,\n              actualCorePartitionSetSize\n          );\n        }\n        annotateFn = annotateCorePartitionSetSizeFn(expectedCorePartitionSetSize);\n      } else if (firstShardSpec instanceof BucketNumberedShardSpec) {\n        throw new ISE(\"Cannot publish segments with shardSpec[%s]\", firstShardSpec);\n      } else {\n        annotateFn = null;\n      }\n\n      if (annotateFn != null) {\n        intervalToSegments.put(interval, segmentsPerInterval.stream().map(annotateFn).collect(Collectors.toList()));\n      }","sourceCodeStart":73,"sourceCodeEnd":109,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/segment/realtime/appenderator/SegmentPublisherHelper.java#L73-L109","documentation":"For intervals whose shard specs form a core partition set (NumberedShardSpec), publication requires ALL core partitions to be present. annotateShardSpec counts segments with partitionNum below the expected core partition set size and throws this ISE when fewer are found, because publishing an incomplete atomic set would leave the datasource inconsistent.","triggerScenarios":"A dimension/hash-partitioned task's sub-tasks finished but some segments were never created or were discarded (failed sub-task, worker loss, or dropped replicas), so the core partition set for the interval is incomplete at publish time.","commonSituations":"IndexTask sub-task failure in parallel batch ingestion; worker crash mid-ingestion leaving missing partitions; task reports success but some shard partitions were never appended.","solutions":["Check the logs of the failed/missing sub-task(s) for the interval and re-run the ingestion after fixing the root failure","Verify all partition sub-tasks (partitionNum 0..N-1) actually completed and published; re-run only the failed ones if the framework supports it","Increase task retry/replica settings or fix resource issues (memory, disk, coordinator availability) that killed sub-tasks"],"exampleFix":"// before\n// sub-task for partition 2 failed silently, publish proceeds and throws\n// after\n// ensure taskCompletion for all partitions before publishing\ntasks.stream().allMatch(t -> t.getStatus().isSuccess()) // gate publish on all partitions succeeding","handlingStrategy":"retry","validationCode":"long actual = segments.stream().filter(s -> s.getShardSpec().getPartitionNum() < expectedCoreSize).count();\nif (actual != expectedCoreSize) { /* halt and inspect failed sub-tasks */ }","typeGuard":null,"tryCatchPattern":"try { publish(); } catch (ISE e) { log.error(\"incomplete core partition set for interval\", e); reRunFailedSubTasks(interval); }","preventionTips":["Monitor all parallel sub-tasks to completion before publishing","Set appropriate task replicas/retries for batch ingestion","Alert on sub-task failures in parallel partitioned ingestion"],"tags":["java","segments","publishing","partitioning"],"backgroundTag":"incomplete-partition-set","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}