apache/iceberg · error · IllegalStateException

Incoming records violate the writer assumption that records

Error message

Incoming records violate the writer assumption that records are clustered by spec and by partition within each spec. Either cluster the incoming records or switch to fanout writers.
Encountered records that belong to already closed files:
spec <spec>

What it means

ClusteredWriter.write enforces that incoming records are clustered by partition spec: once a spec has been completed (its writer closed), records for that spec must not reappear. When a record's specId belongs to an already-closed spec, it throws IllegalStateException describing the offending spec.

Source

Thrown at core/src/main/java/org/apache/iceberg/io/ClusteredWriter.java:76

  protected abstract FileWriter<T, R> newWriter(PartitionSpec spec, StructLike partition);

  protected abstract void addResult(R result);

  protected abstract R aggregatedResult();

  @Override
  public void write(T row, PartitionSpec spec, StructLike partition) {
    if (!spec.equals(currentSpec)) {
      if (currentSpec != null) {
        closeCurrentWriter();
        completedSpecIds.add(currentSpec.specId());
        completedPartitions.clear();
      }

      if (completedSpecIds.contains(spec.specId())) {
        String errorCtx = String.format("spec %s", spec);
        throw new IllegalStateException(NOT_CLUSTERED_ROWS_ERROR_MSG_TEMPLATE + errorCtx);
      }

      StructType partitionType = spec.partitionType();

      this.currentSpec = spec;
      this.partitionComparator = Comparators.forType(partitionType);
      this.completedPartitions = StructLikeSet.create(partitionType);
      // copy the partition key as the key object may be reused
      this.currentPartition = StructLikeUtil.copy(partition);
      this.currentWriter = newWriter(currentSpec, currentPartition);

    } else if (partition != currentPartition
        && partitionComparator.compare(partition, currentPartition) != 0) {
      closeCurrentWriter();
      completedPartitions.add(currentPartition);

      if (completedPartitions.contains(partition)) {
        String errorCtx =

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Sort or repartition records by spec and partition before writing (e.g., a distributed sort by spec id then partition).
2. Switch to a fanout writer (e.g., FanoutDataWriter/FanoutPositionDeleteWriter) which keeps one open writer per spec/partition.
3. Split the input into separate write tasks per spec so each clustered writer sees only one spec.

Example fix

// before
ClusteredDataWriter<T> writer = new ClusteredDataWriter<>(...); // assumes clustered input
records.forEach(writer::write);
// after
FanoutDataWriter<T> writer = new FanoutDataWriter<>(...); // tolerates interleaved specs
records.forEach(writer::write);

// ensure input is clustered by spec id before writing
records = records.sorted(Comparator.comparing(r -> specIdOf(r)));

try {
  writer.write(record);
} catch (IllegalStateException e) {
  // records not clustered: switch to fanout writer or sort input
}

When it happens

Trigger: Writing records to a clustered (non-fanout) writer where spec A, then spec B, then spec A again occur in the input stream, causing the writer for spec A to be reopened after it was closed.

Common situations: Unsorted/unclustered input from joins or multiple table sources with different partition specs feeding one writer; using ClusteredWriter (e.g., ClusteredDataWriter) where records mix specs intermittently instead of using a fanout writer.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Incoming records violate the writer assumption that records
• Failed to close equality delta writer
• Already closed files for partition:
• Cannot bucket by type:
• Identity transform is not supported

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d61c2ad072d571d5. Report an issue: GitHub.