apache/iceberg · error · IllegalStateException

Invalid starting record offset %d for file %d from CombinedS

Error message

Invalid starting record offset %d for file %d from CombinedScanTask: %s

What it means

DataIterator.seek throws this IllegalStateException when the requested starting record offset exceeds the number of records actually available while skipping through the current file of a CombinedScanTask — i.e. the iterator ran out of records before consuming startingRecordOffset items. It indicates the offset (typically produced by the split/offset computation or residual filtering) is inconsistent with the file's row count. The message includes the offset, file index, and the full CombinedScanTask for diagnosis.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/source/DataIterator.java:98

    // skip files
    Preconditions.checkState(
        startingFileOffset < combinedTask.files().size(),
        "Invalid starting file offset %s for combined scan task with %s files: %s",
        startingFileOffset,
        combinedTask.files().size(),
        combinedTask);
    for (long i = 0L; i < startingFileOffset; ++i) {
      tasks.next();
      fileOffset += 1;
    }

    updateCurrentIterator();
    // skip records within the file
    for (long i = 0; i < startingRecordOffset; ++i) {
      if (currentFileHasNext() && hasNext()) {
        next();
      } else {
        throw new IllegalStateException(
            String.format(
                Locale.ROOT,
                "Invalid starting record offset %d for file %d from CombinedScanTask: %s",
                startingRecordOffset,
                startingFileOffset,
                combinedTask));
      }
    }
  }

  @Override
  public boolean hasNext() {
    updateCurrentIterator();
    return currentIterator.hasNext();
  }

  @Override
  public T next() {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Ensure the CombinedScanTask used at seek time is the exact one the offset was computed against (same snapshot).
2. Recompute the starting offset against the current snapshot instead of reusing stale checkpoint offsets after table maintenance (compaction/rewrite).
3. Verify position merging logic (file offset + record offset) when implementing custom source resume logic.
4. Catch IllegalStateException around DataIterator creation and fall back to re-reading the split from its start.

Example fix

// before: stale offset after compaction
long offset = previousState.startingRecordOffset; // computed on old files
iterator.seek(fileOffset, offset);
// after: validate offset against current file
long fileRecords = asFilePaths(currentFile).positionCount();
if (offset >= fileRecords) {
  offset = 0; // or recompute from current snapshot
}
iterator.seek(fileOffset, offset);

// validate offset before seek
long available = /* records in the current file for this split */;
if (startingRecordOffset >= available) {
  throw new IllegalArgumentException("Offset " + startingRecordOffset + " exceeds file records " + available);
}

try {
  dataIterator.seek(fileOffset, recordOffset);
} catch (IllegalStateException e) {
  LOG.warn("Stale offset; re-reading split from start", e);
  dataIterator = newDataIterator(combinedTask);
}

When it happens

Trigger: Calling seek() with a startingRecordOffset larger than the file's record count, e.g. when combining a residual filter (delete files / filter residual rows) with a stale offset, or an off-by-one from a checkpointed reader position on a changed file set.

Common situations: Restoring Flink source state from a checkpoint whose task splits no longer match the current table snapshots (files rewritten by compaction), or constructing DataIterator manually with a wrong fileOffset/recordOffset pair.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Failed to list partitions of table %s
• Failed to plan files for main index
• Failed to plan files for main index
• Unknown serialize version:
• Unknown read version: ${readVersion}

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d264728c37fe2d49. Report an issue: GitHub.