apache/iceberg · error · java.lang.RuntimeException

Rewrite data file error.

Error message

Rewrite data file error.

What it means

RewriteDataFilesAction.execute() wraps any failure from the distributed RowDataRewriter (reading, rewriting, or committing data files) in a generic RuntimeException with the message 'Rewrite data file error.' It signals that a Flink job stage rewriting data files failed; the original cause is attached and must be inspected for the real problem (e.g. read errors, checkpoint failures, commit conflicts).

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java:62

        "Flink does not support compaction on row lineage enabled tables (V3+)");
  }

  @Override
  protected FileIO fileIO() {
    return table().io();
  }

  @Override
  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
    int size = combinedScanTasks.size();
    int parallelism = Math.min(size, maxParallelism);
    DataStream<CombinedScanTask> dataStream = env.fromData(combinedScanTasks);
    RowDataRewriter rowDataRewriter =
        new RowDataRewriter(table(), caseSensitive(), fileIO(), encryptionManager());
    try {
      return rowDataRewriter.rewriteDataForTasks(dataStream, parallelism);
    } catch (Exception e) {
      throw new RuntimeException("Rewrite data file error.", e);
    }
  }

  @Override
  protected RewriteDataFilesAction self() {
    return this;
  }

  public RewriteDataFilesAction maxParallelism(int parallelism) {
    Preconditions.checkArgument(parallelism > 0, "Invalid max parallelism %s", parallelism);
    this.maxParallelism = parallelism;
    return this;
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Inspect the attached cause `e` (print full stack trace) — the message itself is generic and the real error is nested.
2. Verify task manager memory and parallelism are adequate for the data volume being rewritten.
3. Validate the table can be read fully (e.g. run a scan) to rule out corrupted/inaccessible data files.
4. Retry the action; Flink transient failures (network, checkpoint timeouts) are common causes.
5. Upgrade/check Flink-Iceberg version compatibility if the failure is reproducible on valid tables.

Example fix

// before
rewriteAction.execute();
// after
try {
  rewriteAction.execute();
} catch (RuntimeException e) {
  LOG.error("Rewrite failed", e.getCause()); // inspect nested cause for the real error
}

// before executing
RewriteDataFilesAction action = actions.rewriteDataFiles(table);
table.refresh(); // ensure metadata is current
// optionally validate readability:
table.newScan().planFiles().forEach(f -> Preconditions.checkNotNull(f));

try {
  action.execute();
} catch (RuntimeException e) {
  Throwable cause = e.getCause();
  LOG.error("Rewrite data files failed: {}", cause == null ? e : cause.getMessage(), cause);
  throw cause instanceof RuntimeException ? (RuntimeException) cause : e;
}

When it happens

Trigger: Calling RewriteDataFilesAction (RewriteDataFiles Spark-style action in Flink) and the rewriteDataForTasks DataStream job throws any Exception: parquet/orc read failures, out-of-memory in RowDataRewriter, schema mismatch, or commit failures in the sink.

Common situations: Running the rewrite action on tables with corrupted files, on tables whose schema changed, or with insufficient task manager memory; also job-level failures like checkpoint loss or parallelism misconfiguration.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Staging snapshot %s on branch '%s' removes data files; equal
• Rewrite data file error.
• [For table {} with {}[{}] at {}]: Exception processing {}
• Failure during rewrite process for group {}
• Illegal table name:

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b09de1170ea5e56d. Report an issue: GitHub.