apache/druid · error · IllegalStateException

Record sequenceNumber[%s] is smaller than current sequenceNu

Error message

Record sequenceNumber[%s] is smaller than current sequenceNumber[%s] for partition[%s]

What it means

SeekableStreamIndexTaskRunner.verifyProjectedOffsets compares the offset of an incoming record against the task's current (persisted) offset for the partition. Records from the stream must be at or after the current offset; a record whose sequence number is smaller than the current one means the stream is replaying already-processed data or offsets are corrupted, so the task throws an IllegalStateException instead of silently duplicating data.

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java:2236

   */
  private boolean verifyRecordInRange(
      final PartitionIdType partition,
      final SequenceOffsetType recordOffset
  )
  {
    // Verify that the record is at least as high as its currOffset.
    final SequenceOffsetType currOffset = Preconditions.checkNotNull(
        currOffsets.get(partition),
        "Current offset is null for partition[%s]",
        partition
    );

    final OrderedSequenceNumber<SequenceOffsetType> recordSequenceNumber = createSequenceNumber(recordOffset);
    final OrderedSequenceNumber<SequenceOffsetType> currentSequenceNumber = createSequenceNumber(currOffset);

    final int comparisonToCurrent = recordSequenceNumber.compareTo(currentSequenceNumber);
    if (comparisonToCurrent < 0) {
      throw new ISE(
          "Record sequenceNumber[%s] is smaller than current sequenceNumber[%s] for partition[%s]",
          recordOffset,
          currOffset,
          partition
      );
    }

    // Check if the record has already been read.
    if (isRecordAlreadyRead(partition, recordOffset)) {
      return false;
    }

    // Finally, check if this record comes before the endOffsets for this partition.
    return isMoreToReadBeforeReadingRecord(recordSequenceNumber.get(), endOffsets.get(partition));
  }

  /**
   * checks if the input seqNum marks end of shard. Used by Kinesis only

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Reset the supervisor (POST /druid/indexer/v1/supervisor/<id>/reset) so offsets are rebuilt consistently with the current stream position.
2. Restore/repair the task's persisted checkpoints so currOffset matches where the record supplier actually starts.
3. Ensure you never move source topic/stream offsets backward while tasks are running.
4. Verify partition ordering guarantees (single writer per partition, correct partition key) so records don't arrive with decreasing sequence numbers.

Example fix

// before: source offsets reset backward under a running task
kafka-consumer-groups.sh --reset-offsets --to-earliest ...
// after: stop/reset supervisor first
curl -X POST 'http://overlord:8087/druid/indexer/v1/supervisor/my-supervisor/reset'

// ensure source offsets are never moved backward while a task runs
// e.g. before resetting Kafka offsets: stop the supervisor first
curl -X POST http://overlord:8087/druid/indexer/v1/supervisor/<id>/suspend

try { ingest(); } catch (ISE e) { if (e.getMessage().contains("is smaller than current sequenceNumber")) { resetSupervisor(); } else { throw e; } }

When it happens

Trigger: During addRecord/ingestion, recordOffset for partition is less than currOffset — e.g. after checkpoint restore the current offset is ahead of what the record supplier delivers, the topic/stream was reset to an earlier point, or records arrive out of order on a partition that isn't actually strictly ordered.

Common situations: Manually resetting Kafka offsets backward while a task is mid-run; using a non-earliest sequence number policy on Kinesis after resharding; corrupted checkpoints restored from backup; delivering records from a producer writing older offsets after a failover.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• New sequence startOffset[%s] does not equal expected prior o
• Exclusive start partitions[%s] for new sequence don't match
• Resume command was not accepted within 5 seconds
• Unable to create RecordSupplier: %s
• Expected instance of %s, got %s

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/1d3ff983f3d12a58. Report an issue: GitHub.