{"record":{"id":"03e3b61bd85dc8b1","repo":"apache/druid","slug":"pushed-segments-s-are-different-from-the-request","errorCode":null,"errorMessage":"Pushed segments[%s] are different from the requested ones[%s]","messagePattern":"Pushed segments\\[(.+?)\\] are different from the requested ones\\[(.+?)\\]","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderatorDriver.java","lineNumber":157,"sourceCode":"    final Set<SegmentIdWithShardSpec> requestedSegmentIdsForSequences = getAppendingSegments(sequenceNames);\n\n    final ListenableFuture<SegmentsAndCommitMetadata> future = Futures.transformAsync(\n        pushInBackground(null, requestedSegmentIdsForSequences, false),\n        (AsyncFunction<SegmentsAndCommitMetadata, SegmentsAndCommitMetadata>) this::dropInBackground,\n        MoreExecutors.directExecutor()\n    );\n\n    final SegmentsAndCommitMetadata segmentsAndCommitMetadata =\n        pushAndClearTimeoutMs == 0L ? future.get() : future.get(pushAndClearTimeoutMs, TimeUnit.MILLISECONDS);\n\n    // Sanity check\n    final Map<SegmentIdWithShardSpec, DataSegment> pushedSegmentIdToSegmentMap = segmentsAndCommitMetadata\n        .getSegments()\n        .stream()\n        .collect(Collectors.toMap(SegmentIdWithShardSpec::fromDataSegment, Function.identity()));\n\n    if (!pushedSegmentIdToSegmentMap.keySet().equals(requestedSegmentIdsForSequences)) {\n      throw new ISE(\n          \"Pushed segments[%s] are different from the requested ones[%s]\",\n          pushedSegmentIdToSegmentMap.keySet(),\n          requestedSegmentIdsForSequences\n      );\n    }\n\n    synchronized (segments) {\n      for (String sequenceName : sequenceNames) {\n        final SegmentsForSequence segmentsForSequence = segments.get(sequenceName);\n        if (segmentsForSequence == null) {\n          throw new ISE(\"Can't find segmentsForSequence for sequence[%s]\", sequenceName);\n        }\n\n        segmentsForSequence.getAllSegmentsOfInterval().forEach(segmentsOfInterval -> {\n          final SegmentWithState appendingSegment = segmentsOfInterval.getAppendingSegment();\n          if (appendingSegment != null) {\n            final DataSegment pushedSegment = pushedSegmentIdToSegmentMap.get(appendingSegment.getSegmentIdentifier());\n            if (pushedSegment == null) {","sourceCodeStart":139,"sourceCodeEnd":175,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderatorDriver.java#L139-L175","documentation":"Thrown by BatchAppenderatorDriver.pushAndClear when the set of segment IDs actually pushed to deep storage differs from the set of segment IDs requested for the pending sequences. Druid expects push operations to be deterministic: every segment requested must be pushed exactly once. A mismatch means the batch task's internal state diverged from deep storage, so the driver aborts rather than publish inconsistent metadata.","triggerScenarios":"Calling pushAllAndClear (which delegates to pushAndClear) after the underlying Appenderator pushed a different set of segments than the sequence-name-derived requestedSegmentIdsForSequences — e.g. segments were dropped, failed to push, or were written by a different sequence than expected.","commonSituations":"Batch ingestion tasks (batch ingestion SQL/native) hitting storage failures that partially push segments; race conditions or state resets mid-push; bugs in custom Appenderator implementations; supervisor/task restarts that desynchronize the driver's segment map.","solutions":["Inspect the logged pushed vs requested segment ID sets and compare to identify which segments are missing or extra","Check for push failures to deep storage in the task logs (retriable push errors, S3/HDFS issues) and fix the storage backend","Retry the task; batch ingestion is transactional so a clean rerun usually clears transient divergence","If reproducible, report/inspect the Appenderator implementation for a bug in how segments are mapped to sequences"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// before calling pushAllAndClear\nSet<SegmentIdWithShardSpec> expected = sequences.stream()\n    .flatMap(seq -> driver.getSegments(seq).stream())\n    .map(SegmentIdWithShardSpec::fromDataSegment)\n    .collect(Collectors.toSet());\nif (expected.isEmpty()) { throw new IllegalStateException(\"No segments pending push\"); }","typeGuard":null,"tryCatchPattern":"try {\n  driver.pushAllAndClear(publisher, commitFn);\n} catch (ISE e) {\n  if (e.getMessage().contains(\"different from the requested\")) {\n    // abort task and retry whole batch; do not attempt partial publish\n    throw new TaskAbortedException(e);\n  }\n  throw e;\n}","preventionTips":["Call pushAllAndClear exactly once per batch task, after all appends are complete","Monitor deep-storage push logs for silent/retried push failures","Do not mutate or drop segments between append and push phases","Test batch tasks against the actual deep-storage backend before production runs"],"tags":["java","druid","ingestion","internal-state"],"backgroundTag":"internal-invariant-violation","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}