apache/druid · error · IllegalStateException

Can't find pushedSegments for segment[%s]

Error message

Can't find pushedSegments for segment[%s]

What it means

Thrown by BatchAppenderatorDriver.pushAndClear when a segment that is still in the 'appending' state for a sequence has no corresponding entry in the map of pushed segments. Every active appending segment must have been pushed before the driver finalizes it; a miss means the push did not cover all active segments.

Source

Thrown at server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderatorDriver.java:176

          "Pushed segments[%s] are different from the requested ones[%s]",
          pushedSegmentIdToSegmentMap.keySet(),
          requestedSegmentIdsForSequences
      );
    }

    synchronized (segments) {
      for (String sequenceName : sequenceNames) {
        final SegmentsForSequence segmentsForSequence = segments.get(sequenceName);
        if (segmentsForSequence == null) {
          throw new ISE("Can't find segmentsForSequence for sequence[%s]", sequenceName);
        }

        segmentsForSequence.getAllSegmentsOfInterval().forEach(segmentsOfInterval -> {
          final SegmentWithState appendingSegment = segmentsOfInterval.getAppendingSegment();
          if (appendingSegment != null) {
            final DataSegment pushedSegment = pushedSegmentIdToSegmentMap.get(appendingSegment.getSegmentIdentifier());
            if (pushedSegment == null) {
              throw new ISE("Can't find pushedSegments for segment[%s]", appendingSegment.getSegmentIdentifier());
            }

            segmentsOfInterval.finishAppendingToCurrentActiveSegment(
                segmentWithState -> segmentWithState.pushAndDrop(pushedSegment)
            );
          }
        });
      }
    }

    return segmentsAndCommitMetadata;
  }

  /**
   * Publish all segments.
   *
   * @param segmentsToBeOverwritten segments which can be overwritten by new segments published by the given publisher
   * @param publisher               segment publisher

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Compare the logged pushed segment set against the appending segment identifiers to find which segment was never pushed
  2. Check deep-storage push logs for failures on the missing segment and address the storage error
  3. Retry the batch task; the transactional model reruns pushes cleanly
  4. If reproducible, inspect the Appenderator/segment allocation logic for identifier mismatches (e.g. changed shard specs)
Defensive patterns

Strategy: validation

Validate before calling

// confirm all appending segments have been pushed before finalizing
Set<SegmentIdWithShardSpec> pending = driver.getActiveAppendingSegments();
Set<SegmentIdWithShardSpec> pushed = lastPushResult.keySet();
if (!pushed.containsAll(pending)) {
  throw new IllegalStateException("Unpushed appending segments: " + Sets.difference(pending, pushed));
}

Try / catch

try {
  driver.pushAllAndClear(publisher, commitFn);
} catch (ISE e) {
  if (e.getMessage().startsWith("Can't find pushedSegments for segment")) {
    // a push failed silently; abort and retry the whole batch
    throw new TaskAbortedException(e);
  }
  throw e;
}

Prevention

When it happens

Trigger: During pushAndClear, an appending SegmentWithState exists whose SegmentIdWithShardSpec is absent from pushedSegmentIdToSegmentMap — i.e. the Appenderator pushed fewer segments than there are active appending segments.

Common situations: Partial push failures where some segments were dropped before push; mismatch between the segment identifiers registered in the driver and those produced by the Appenderator; shard spec generation changes mid-task; custom Appenderator bugs.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/daa5f46791f60d35. Report an issue: GitHub.