apache/druid · critical · IllegalStateException

Cannot publish segments due to incomplete time chunk for…

Error message

Cannot publish segments due to incomplete time chunk for interval[%s]. Expected [%s] segments in the core partition, but only [%] segments are found. See task logs for more details about these segments.

What it means

For intervals whose shard specs form a core partition set (NumberedShardSpec), publication requires ALL core partitions to be present. annotateShardSpec counts segments with partitionNum below the expected core partition set size and throws this ISE when fewer are found, because publishing an incomplete atomic set would leave the datasource inconsistent.

Solutions

1. Check the logs of the failed/missing sub-task(s) for the interval and re-run the ingestion after fixing the root failure
2. Verify all partition sub-tasks (partitionNum 0..N-1) actually completed and published; re-run only the failed ones if the framework supports it
3. Increase task retry/replica settings or fix resource issues (memory, disk, coordinator availability) that killed sub-tasks

Example fix

// before
// sub-task for partition 2 failed silently, publish proceeds and throws
// after
// ensure taskCompletion for all partitions before publishing
tasks.stream().allMatch(t -> t.getStatus().isSuccess()) // gate publish on all partitions succeeding

long actual = segments.stream().filter(s -> s.getShardSpec().getPartitionNum() < expectedCoreSize).count();
if (actual != expectedCoreSize) { /* halt and inspect failed sub-tasks */ }

try { publish(); } catch (ISE e) { log.error("incomplete core partition set for interval", e); reRunFailedSubTasks(interval); }

When it happens

Trigger: A dimension/hash-partitioned task's sub-tasks finished but some segments were never created or were discarded (failed sub-task, worker loss, or dropped replicas), so the core partition set for the interval is incomplete at publish time.

Common situations: IndexTask sub-task failure in parallel batch ingestion; worker crash mid-ingestion leaving missing partitions; task reports success but some shard partitions were never appended.

Related errors

• Cannot publish segments with shardSpec
• Mismatched shardSpecs in interval
• announceHistoricalSegments failed with null metadata…
• Column is not multi-valued
• index[ ] >= size[ ] or < 0

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/73cbcf1684767afd. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/segment/realtime/appenderator/SegmentPublisherHelper.java:91

      }
      final Function<DataSegment, DataSegment> annotateFn;
      if (firstShardSpec instanceof OverwriteShardSpec) {
        annotateFn = annotateAtomicUpdateGroupFn(segmentsPerInterval.size());
      } else if (firstShardSpec instanceof BuildingShardSpec) {
        // sanity check
        // BuildingShardSpec is used in non-appending mode. In this mode,
        // the segments in each interval should have contiguous partitionIds,
        // so that they can be queryable (see PartitionHolder.isComplete()).
        int expectedCorePartitionSetSize = segmentsPerInterval.size();
        int actualCorePartitionSetSize = Math.toIntExact(
            segmentsPerInterval
                .stream()
                .filter(segment -> segment.getShardSpec().getPartitionNum() < expectedCorePartitionSetSize)
                .count()
        );
        if (expectedCorePartitionSetSize != actualCorePartitionSetSize) {
          LOG.errorSegments(segmentsPerInterval, "Cannot publish segments due to incomplete time chunk");
          throw new ISE(
              "Cannot publish segments due to incomplete time chunk for interval[%s]. "
              + "Expected [%s] segments in the core partition, but only [%] segments are found. "
              + "See task logs for more details about these segments.",
              interval,
              expectedCorePartitionSetSize,
              actualCorePartitionSetSize
          );
        }
        annotateFn = annotateCorePartitionSetSizeFn(expectedCorePartitionSetSize);
      } else if (firstShardSpec instanceof BucketNumberedShardSpec) {
        throw new ISE("Cannot publish segments with shardSpec[%s]", firstShardSpec);
      } else {
        annotateFn = null;
      }

      if (annotateFn != null) {
        intervalToSegments.put(interval, segmentsPerInterval.stream().map(annotateFn).collect(Collectors.toList()));
      }

View on GitHub (pinned to 9b90983fd2)