apache/druid · error · IllegalStateException

Exclusive start partitions[%s] for new sequence don't match

Error message

Exclusive start partitions[%s] for new sequence don't match to the prior offset[%s]

What it means

During task resume, SeekableStreamIndexTaskRunner checks that the exclusive start partitions of the new SequenceMetadata equal the partition set of the latest prior sequence's endOffsets (when the end offsets are exclusive). If the partition sets differ, it throws this IllegalStateException, since resuming on a different partition set than the previous sequence would skip or duplicate partitions.

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java:1349

        final PartitionIdType partition = entry.getKey();
        final SequenceOffsetType startOffset = entry.getValue();

        if (latestSequence != null) {
          final SequenceOffsetType priorOffset = latestSequence.getEndOffsets().get(partition);

          if (!startOffset.equals(priorOffset)) {
            throw new ISE(
                "New sequence startOffset[%s] does not equal expected prior offset[%s]",
                startOffset,
                priorOffset
            );
          }
        }
      }

      if (!isEndOffsetExclusive() && latestSequence != null) {
        if (!latestSequence.getEndOffsets().keySet().equals(sequenceMetadata.getExclusiveStartPartitions())) {
          throw new ISE(
              "Exclusive start partitions[%s] for new sequence don't match to the prior offset[%s]",
              sequenceMetadata.getExclusiveStartPartitions(),
              latestSequence
          );
        }
      }

      // Actually do the add.
      sequences.add(sequenceMetadata);
    }
    finally {
      sequencesLock.unlock();
    }
  }

  private void removeSequence(final SequenceMetadata<PartitionIdType, SequenceOffsetType> sequenceMetadata)
  {
    sequencesLock.lock();

View on GitHub (pinned to 9b90983fd2)

Solutions

POST /druid/indexer/v1/supervisor/<id>/reset to rebuild checkpoints and sequence metadata consistent with the current partition layout.
Re-create/adjust the supervisor spec's partitions/topic to match the current stream partition set and restart.
If resharding is expected, upgrade tasks so they handle new partitions gracefully rather than resuming stale sequence metadata.
Audit the saved checkpoints (task storage) and correct the partition list to match the prior sequence's endOffsets keys.

Example fix

// before: spec partitions stale after topic repartition
"ioConfig": {"topic": "events", "partitions": [0, 1]}
// after: reset supervisor then update spec to current partitions
curl -X POST 'http://overlord:8087/druid/indexer/v1/supervisor/events-supervisor/reset'
"ioConfig": {"topic": "events", "partitions": [0, 1, 2]}

Defensive patterns

Strategy: validation

Validate before calling

// confirm partition set unchanged since the last sequence
if (!latestSeq.getEndOffsets().keySet().equals(newSeq.getExclusiveStartPartitions())) { resetSupervisor(); }

Try / catch

try { resumeSupervisor(id); } catch (IllegalStateException e) { if (e.getMessage().contains("don't match to the prior offset")) { resetSupervisor(id); } else { throw e; } }

Prevention

Plan for resharding: stop supervisor, reset, update partitions, restart
Verify topic/shard layout after any re-partition before resuming tasks
Do not restore task state from a different topic/cluster

When it happens

Trigger: Task restart/resume where sequenceMetadata.getExclusiveStartPartitions() no longer matches latestSequence.getEndOffsets().keySet() — e.g. the topic was re-partitioned (partitions added/removed) between the two sequences, or checkpoints were manually edited so the partition list changed.

Common situations: Kafka topic re-created or expanded (partitions added) while a task was stopped; Kinesis resharding (shards split/merged) between task runs; hand-edited checkpoint JSON with wrong partition IDs; resuming tasks backed up from a different topic.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/f5239dfb31e6da16. Report an issue: GitHub.