apache/druid · error · IllegalStateException

Pushed segments[%s] are different from the requested ones[%s

Error message

Pushed segments[%s] are different from the requested ones[%s]

What it means

Thrown by BatchAppenderatorDriver.pushAndClear when the set of segment IDs actually pushed to deep storage differs from the set of segment IDs requested for the pending sequences. Druid expects push operations to be deterministic: every segment requested must be pushed exactly once. A mismatch means the batch task's internal state diverged from deep storage, so the driver aborts rather than publish inconsistent metadata.

Source

Thrown at server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderatorDriver.java:157

    final Set<SegmentIdWithShardSpec> requestedSegmentIdsForSequences = getAppendingSegments(sequenceNames);

    final ListenableFuture<SegmentsAndCommitMetadata> future = Futures.transformAsync(
        pushInBackground(null, requestedSegmentIdsForSequences, false),
        (AsyncFunction<SegmentsAndCommitMetadata, SegmentsAndCommitMetadata>) this::dropInBackground,
        MoreExecutors.directExecutor()
    );

    final SegmentsAndCommitMetadata segmentsAndCommitMetadata =
        pushAndClearTimeoutMs == 0L ? future.get() : future.get(pushAndClearTimeoutMs, TimeUnit.MILLISECONDS);

    // Sanity check
    final Map<SegmentIdWithShardSpec, DataSegment> pushedSegmentIdToSegmentMap = segmentsAndCommitMetadata
        .getSegments()
        .stream()
        .collect(Collectors.toMap(SegmentIdWithShardSpec::fromDataSegment, Function.identity()));

    if (!pushedSegmentIdToSegmentMap.keySet().equals(requestedSegmentIdsForSequences)) {
      throw new ISE(
          "Pushed segments[%s] are different from the requested ones[%s]",
          pushedSegmentIdToSegmentMap.keySet(),
          requestedSegmentIdsForSequences
      );
    }

    synchronized (segments) {
      for (String sequenceName : sequenceNames) {
        final SegmentsForSequence segmentsForSequence = segments.get(sequenceName);
        if (segmentsForSequence == null) {
          throw new ISE("Can't find segmentsForSequence for sequence[%s]", sequenceName);
        }

        segmentsForSequence.getAllSegmentsOfInterval().forEach(segmentsOfInterval -> {
          final SegmentWithState appendingSegment = segmentsOfInterval.getAppendingSegment();
          if (appendingSegment != null) {
            final DataSegment pushedSegment = pushedSegmentIdToSegmentMap.get(appendingSegment.getSegmentIdentifier());
            if (pushedSegment == null) {

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Inspect the logged pushed vs requested segment ID sets and compare to identify which segments are missing or extra
2. Check for push failures to deep storage in the task logs (retriable push errors, S3/HDFS issues) and fix the storage backend
3. Retry the task; batch ingestion is transactional so a clean rerun usually clears transient divergence
4. If reproducible, report/inspect the Appenderator implementation for a bug in how segments are mapped to sequences

// before calling pushAllAndClear
Set<SegmentIdWithShardSpec> expected = sequences.stream()
    .flatMap(seq -> driver.getSegments(seq).stream())
    .map(SegmentIdWithShardSpec::fromDataSegment)
    .collect(Collectors.toSet());
if (expected.isEmpty()) { throw new IllegalStateException("No segments pending push"); }

try {
  driver.pushAllAndClear(publisher, commitFn);
} catch (ISE e) {
  if (e.getMessage().contains("different from the requested")) {
    // abort task and retry whole batch; do not attempt partial publish
    throw new TaskAbortedException(e);
  }
  throw e;
}

When it happens

Trigger: Calling pushAllAndClear (which delegates to pushAndClear) after the underlying Appenderator pushed a different set of segments than the sequence-name-derived requestedSegmentIdsForSequences — e.g. segments were dropped, failed to push, or were written by a different sequence than expected.

Common situations: Batch ingestion tasks (batch ingestion SQL/native) hitting storage failures that partially push segments; race conditions or state resets mid-push; bugs in custom Appenderator implementations; supervisor/task restarts that desynchronize the driver's segment map.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Can't find segmentsForSequence for sequence[%s]
• Can't find pushedSegments for segment[%s]
• ColumnCapacityExceededException
• Cannot deserialize type[%s] to an RoaringBitmap64Counter:
• Index[%d] >= size[%d]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/03e3b61bd85dc8b1. Report an issue: GitHub.