{"record":{"id":"5990851ba6241148","repo":"apache/druid","slug":"stream-ingestion-task-unexpectedly-attempted-to-ov","errorCode":null,"errorMessage":"Stream ingestion task unexpectedly attempted to overwrite segments: %s","messagePattern":"Stream ingestion task unexpectedly attempted to overwrite segments: (.+?)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SequenceMetadata.java","lineNumber":360,"sourceCode":"        TaskToolbox toolbox,\n        boolean useTransaction\n    )\n    {\n      this.runner = runner;\n      this.toolbox = toolbox;\n      this.useTransaction = useTransaction;\n    }\n\n    @Override\n    public SegmentPublishResult publishAnnotatedSegments(\n        @Nullable Set<DataSegment> mustBeNullOrEmptyOverwriteSegments,\n        Set<DataSegment> segmentsToPush,\n        @Nullable Object commitMetadata,\n        SegmentSchemaMapping segmentSchemaMapping\n    ) throws IOException\n    {\n      if (mustBeNullOrEmptyOverwriteSegments != null && !mustBeNullOrEmptyOverwriteSegments.isEmpty()) {\n        throw new ISE(\n            \"Stream ingestion task unexpectedly attempted to overwrite segments: %s\",\n            SegmentUtils.commaSeparatedIdentifiers(mustBeNullOrEmptyOverwriteSegments)\n        );\n      }\n      final Map<?, ?> commitMetaMap = (Map<?, ?>) Preconditions.checkNotNull(commitMetadata, \"commitMetadata\");\n      final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> finalPartitions =\n          runner.deserializePartitionsFromMetadata(\n              toolbox.getJsonMapper(),\n              commitMetaMap.get(SeekableStreamIndexTaskRunner.METADATA_PUBLISH_PARTITIONS)\n          );\n\n      // Sanity check, we should only be publishing things that match our desired end state.\n      if (!getEndOffsets().equals(finalPartitions.getPartitionSequenceNumberMap())) {\n        throw new ISE(\n            \"Driver for sequence[%s] attempted to publish invalid metadata[%s].\",\n            SequenceMetadata.this.toString(),\n            commitMetadata\n        );","sourceCodeStart":342,"sourceCodeEnd":378,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SequenceMetadata.java#L342-L378","documentation":"SequenceMetadata.publishAnnotatedSegments asserts an invariant: stream ingestion tasks must not overwrite existing segments, so the internal mustBeNullOrEmptyOverwriteVersions/mustBeNullOrEmptyOverwriteSegments marker must always be empty (or null) at publish time. If it contains segments, a code path has violated the streaming contract that segment versions/publishing are strictly append-only, and the runner throws an IllegalStateException listing the offending segment identifiers.","triggerScenarios":"Calling publishAnnotatedSegments (from SeekableStreamIndexTaskRunner's publish path) while mustBeNullOrEmptyOverwriteSegments is non-empty — i.e. the task computed overwrite segments it should never produce: e.g. a streaming task configured with a non-null overwrite tombstone, corrupted metadata, or a bug/extension modifying the segment set before publishing.","commonSituations":"Streaming tasks whose specs or tuning produce overwrite versions due to misused segment-granularity/append-to-existing settings; data corruption in task metadata after crash/replay; custom extensions or code changes feeding extra segments into the publish transaction.","solutions":["Inspect the listed segment identifiers and find which spec/tuning options caused overwrite segments; remove any overwrite-tombstone or non-append configuration from the streaming task.","Reset the supervisor and restart the affected tasks so fresh SequenceMetadata is built without stale overwrite markers.","Upgrade Druid if this reproduces on standard specs — it indicates a bug in the runner/extension publishing path.","Ensure no custom code/extensions mutate the segments set passed to the publish transaction."],"exampleFix":"// before: streaming spec triggering overwrite paths\n\"tuningConfig\": {\"type\": \"kafka\", \"appendable\": false, ...}\n// after: keep streaming tasks append-only\n\"tuningConfig\": {\"type\": \"kafka\", \"appendable\": true, ...}","handlingStrategy":"validation","validationCode":"// streaming tasks must be append-only; check spec before submission\nassert spec.getTuningConfig().isAppendable() : \"streaming tasks cannot overwrite segments\";","typeGuard":null,"tryCatchPattern":"try { runTask(); } catch (ISE e) { if (e.getMessage().contains(\"unexpectedly attempted to overwrite segments\")) { escalateAsDruidBug(e); } else { throw e; } }","preventionTips":["Keep streaming task specs appendable (no overwrite tombstones)","Avoid custom code that mutates the segment set before publish","Upgrade Druid if standard specs trigger this invariant violation"],"tags":["druid","streaming","segments","internal-invariant"],"backgroundTag":"internal-invariant-violation","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}