apache/druid · error · IllegalStateException

New sequence startOffset[%s] does not equal expected prior o

Error message

New sequence startOffset[%s] does not equal expected prior offset[%s]

What it means

When a SeekableStream (Kafka/Kinesis) index task resumes from persisted checkpoints, getOrCreateSequenceMetadata verifies that each new sequence's start offsets exactly equal the end offsets recorded in the latest existing SequenceMetadata for each partition. If the new start offset differs from the prior end offset for a partition, it throws this IllegalStateException because the stream would be re-read from a wrong point (data loss/duplication).

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java:1338

  }

  private void addSequence(final SequenceMetadata<PartitionIdType, SequenceOffsetType> sequenceMetadata)
  {
    sequencesLock.lock();
    try {
      final SequenceMetadata<PartitionIdType, SequenceOffsetType> latestSequence =
          getLastSequenceMetadataOrNull(sequences);

      // Sanity check that the start of the new sequence matches up with the end of the prior sequence.
      for (Map.Entry<PartitionIdType, SequenceOffsetType> entry : sequenceMetadata.getStartOffsets().entrySet()) {
        final PartitionIdType partition = entry.getKey();
        final SequenceOffsetType startOffset = entry.getValue();

        if (latestSequence != null) {
          final SequenceOffsetType priorOffset = latestSequence.getEndOffsets().get(partition);

          if (!startOffset.equals(priorOffset)) {
            throw new ISE(
                "New sequence startOffset[%s] does not equal expected prior offset[%s]",
                startOffset,
                priorOffset
            );
          }
        }
      }

      if (!isEndOffsetExclusive() && latestSequence != null) {
        if (!latestSequence.getEndOffsets().keySet().equals(sequenceMetadata.getExclusiveStartPartitions())) {
          throw new ISE(
              "Exclusive start partitions[%s] for new sequence don't match to the prior offset[%s]",
              sequenceMetadata.getExclusiveStartPartitions(),
              latestSequence
          );
        }
      }

View on GitHub (pinned to 9b90983fd2)

Solutions

Reset the supervisor's offsets to match current stream state (POST /druid/indexer/v1/supervisor/<id>/reset) so checkpoints and sequence metadata are rebuilt.
Ensure you did not manually alter checkpoint JSON; restore consistent checkpoints where new-sequence startOffsets equal prior-sequence endOffsets.
If partitions were added/removed in the source topic, restart the supervisor so sequence metadata is regenerated for the new partition layout.
Verify task's persisted checkpoints in the tasks storage / database and correct them to align endOffsets with the new start offsets.

Example fix

// before: new sequence startOffsets don't match prior endOffsets
{"startOffsets": {"0": 100}} // prior endOffsets: {"0": 250}
// after: reset supervisor to rebuild checkpoints
curl -X POST 'http://overlord:8087/druid/indexer/v1/supervisor/my-supervisor/reset'

Defensive patterns

Strategy: validation

Validate before calling

// before resuming, verify checkpoints are contiguous
// new sequence startOffsets must equal prior sequence endOffsets per partition
if (!newSeq.getStartOffsets().equals(latestSeq.getEndOffsets())) { resetSupervisor(); }

Try / catch

try { resumeSupervisor(id); } catch (IllegalStateException e) { if (e.getMessage().contains("does not equal expected prior offset")) { resetSupervisor(id); } else { throw e; } }

Prevention

Never hand-edit checkpoint metadata
Use the supervisor reset endpoint after offset manipulation
Avoid changing topic offsets while tasks are mid-sequence

When it happens

Trigger: Task restart/resume where the checkpoints saved by a prior task generation don't line up: sequenceMetadata endOffsets for partition P differ from the startOffsets of the new sequence entry, typically after manually editing checkpoints, restoring from an older supervisor state, or a producer-consumer offset mismatch.

Common situations: Manually modified checkpoint file / task storage; resetting offsets via Kafka tooling while a task's sequence metadata still holds old offsets; supervisor resume after task failure where repartitioning changed partition IDs; restoring tasks from a different cluster.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/46bb9f033e48410f. Report an issue: GitHub.