apache/druid · critical · IllegalStateException

Driver for sequence[%s] attempted to publish invalid metadat

Error message

Driver for sequence[%s] attempted to publish invalid metadata[%s].

What it means

During segment publishing, SeekableStreamIndexTaskRunner hands SequenceMetadata the commit metadata (partition->sequence offsets) it stored. Before publishing segments, SequenceMetadata verifies that the final partitions implied by the stored commit metadata exactly match this SequenceMetadata's endOffsets. A mismatch means the task's in-memory publish state and its persisted commit metadata disagree, so publishing is aborted with an IllegalStateException to prevent committing wrong offsets.

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SequenceMetadata.java:374

        SegmentSchemaMapping segmentSchemaMapping
    ) throws IOException
    {
      if (mustBeNullOrEmptyOverwriteSegments != null && !mustBeNullOrEmptyOverwriteSegments.isEmpty()) {
        throw new ISE(
            "Stream ingestion task unexpectedly attempted to overwrite segments: %s",
            SegmentUtils.commaSeparatedIdentifiers(mustBeNullOrEmptyOverwriteSegments)
        );
      }
      final Map<?, ?> commitMetaMap = (Map<?, ?>) Preconditions.checkNotNull(commitMetadata, "commitMetadata");
      final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> finalPartitions =
          runner.deserializePartitionsFromMetadata(
              toolbox.getJsonMapper(),
              commitMetaMap.get(SeekableStreamIndexTaskRunner.METADATA_PUBLISH_PARTITIONS)
          );

      // Sanity check, we should only be publishing things that match our desired end state.
      if (!getEndOffsets().equals(finalPartitions.getPartitionSequenceNumberMap())) {
        throw new ISE(
            "Driver for sequence[%s] attempted to publish invalid metadata[%s].",
            SequenceMetadata.this.toString(),
            commitMetadata
        );
      }

      final TaskAction<SegmentPublishResult> action;

      if (segmentsToPush.isEmpty()) {
        // If a task ingested no data but made progress reading through its assigned partitions,
        // we publish no segments but still need to update the supervisor with the current offsets
        SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> startPartitions =
            new SeekableStreamStartSequenceNumbers<>(
                finalPartitions.getStream(),
                getStartOffsets(),
                exclusiveStartPartitions
            );
        if (isMetadataUnchanged(startPartitions, finalPartitions)) {

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Compare the metadata[%s] value in the message with the task's expected end offsets; if stale, reset the supervisor/sequence so tasks restart from consistent offsets
2. Kill and re-launch the affected tasks so a fresh SequenceMetadata and commit metadata are generated
3. Check whether a checkpoint request (supervisor checkpoint API) was issued with offsets inconsistent with the running sequence and re-checkpoint correctly
4. If reproducible, verify no duplicate publishing path stores commit metadata for the wrong sequence; upgrade Druid if a known bug

Example fix

// before: publishing with mismatched commit metadata
runner.publishAnnotatedSegments(toolbox, staleCommitMetaMap);
// after: verify sequence matches end offsets before publishing
if (!sequenceMetadata.getEndOffsets().equals(commitMetaMap.get(METADATA_PUBLISH_PARTITIONS))) {
  // checkpoint/reset or skip publishing for this stale metadata
}

if (!sequenceMetadata.getEndOffsets().equals(commitMeta.getPartitionSequenceNumberMap())) {
  throw new IllegalStateException("Commit metadata does not match current sequence end offsets; reset or re-checkpoint before publishing");
}

try {
  sequenceMetadata.publishAnnotatedSegments(toolbox, commitMetaMap);
} catch (IllegalStateException e) {
  log.error(e, "Invalid publish metadata; aborting publish and resetting sequence");
  // trigger supervisor reset / task restart
}

When it happens

Trigger: publishAnnotatedSegments is called with a commitMetaMap whose METADATA_PUBLISH_PARTITIONS entry does not equal getEndOffsets() of this SequenceMetadata — e.g. publishing after a checkpoint changed the sequence's end offsets, or replaying/storing stale commit metadata from a previous sequence.

Common situations: Kafka/Kinesis tasks crashing and resuming with persisted commit metadata from an older checkpoint; replica tasks with divergent checkpoints; bugs or manual edits to task storage that leave stale metadata rows behind.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Unsupported keyFormat. KafkaInputformat only supports input
• New sequence startOffset[%s] does not equal expected prior o
• Exclusive start partitions[%s] for new sequence don't match
• Resume command was not accepted within 5 seconds
• Record sequenceNumber[%s] is smaller than current sequenceNu

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/7c5174e0d639f0d3. Report an issue: GitHub.