[For table {} with {}[{}] at {}]: Failed to plan data file r

Error message

[For table {} with {}[{}] at {}]: Failed to plan data file rewrite groups

What it means

DataFileRewritePlanner.processElement() builds a BinPackRewriteFilePlanner over the table's current snapshot and emits PlannedGroup records. Any exception while loading the table, resolving the branch snapshot, initializing the planner with rewriter options, executing the plan, or iterating groups is caught, logged with the maintenance MESSAGE_PREFIX, and forwarded to the TaskResultAggregator error stream. No PlannedGroups are emitted for that trigger cycle.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DataFileRewritePlanner.java:184

          taskName,
          taskIndex,
          ctx.timestamp(),
          groups.size(),
          groups);
      plannedGroupsCounter.inc(groups.size());

      for (RewriteFileGroup group : groups) {
        LOG.info(
            DataFileRewritePlanner.MESSAGE_PREFIX + "Emitting {}",
            tableName,
            taskName,
            taskIndex,
            ctx.timestamp(),
            group);
        out.collect(new PlannedGroup(table, groupsPerCommit, group, branch));
      }
    } catch (Exception e) {
      LOG.warn(
          DataFileRewritePlanner.MESSAGE_PREFIX + "Failed to plan data file rewrite groups",
          tableName,
          taskName,
          taskIndex,
          ctx.timestamp(),
          e);
      ctx.output(TaskResultAggregator.ERROR_STREAM, e);
      errorCounter.inc();
    }
  }

  @Override
  public void close() throws Exception {
    super.close();
    tableLoader.close();
  }

  public static class PlannedGroup {

View on GitHub (pinned to 86d9c8fc54)

Solutions

Check the logged cause — planner.init option errors state the offending property; fix RewriterConfig values (min-file-size < max-file-size, positive partial-progress.max-commits)
Verify the configured branch exists and the table/catalog is reachable from the Flink cluster
Re-run the maintenance job; the operator records the error via the error counter and the next trigger re-plans
Validate table location and catalog options in TableLoader (Hive/Hadoop/REST catalog config)
Confirm the filterSupplier returns a valid serializable Expression and does not throw

Example fix

// before: invalid rewriter options crash planning every trigger
Map<String, String> opts = ImmutableMap.of(
    "rewrite-data-files.min-file-size", "1GB",
    "rewrite-data-files.max-file-size", "512MB"); // min > max

// after: valid bin-pack options
Map<String, String> opts = ImmutableMap.of(
    "rewrite-data-files.min-file-size", "128MB",
    "rewrite-data-files.max-file-size", "1GB",
    "partial-progress.max-commits", "10");

Defensive patterns

Strategy: validation

Validate before calling

// validate options and branch before building the maintenance job
Table table = tableLoader.loadTable();
Preconditions.checkArgument(table.snapshot(branch) != null || table.snapshots().isEmpty(),
    "Branch %s does not exist", branch);
int maxCommits = Integer.parseInt(opts.get("partial-progress.max-commits"));
Preconditions.checkArgument(maxCommits > 0, "partial-progress.max-commits must be > 0");
Preconditions.checkArgument(minFileSizeBytes < maxFileSizeBytes, "min-file-size must be < max-file-size");

Try / catch

// planner errors surface on the error side-output stream; gate retries on it
DataStream<Exception> errors = result.getSideOutput(TaskResultAggregator.ERROR_STREAM);
errors.process((ProcessFunction<Exception, Void>) (e, ctx) -> {
  LOG.warn("Rewrite planning failed, next trigger will re-plan", e);
  return null;
});

Prevention

Validate rewriter options (min/max file sizes, partial-progress.max-commits) before job submission — planner.init rejects invalid values at trigger time
Verify the branch name exists after any table renames or branch management operations
Test table loadability (catalog URL, credentials) from the Flink runtime environment
Ensure the filterSupplier returns a valid serializable Expression and does not throw

When it happens

Trigger: tableLoader.loadTable() or SerializableTable.copyOf fails (catalog unavailable, bad table name); table.snapshot(branch) fails for a nonexistent branch; planner.init(rewriterOptions) rejects an invalid option value; filterSupplier.get() throws; planner.plan() fails reading manifests/metrics; partialProgressMaxCommits <= 0 breaking the divide into groupsPerCommit.

Common situations: Typo'd or contradictory rewriter options (min-file-size > max-file-size, non-numeric partial-progress.max-commits) rejected at init; branch renamed/deleted; catalog outage or expired credentials on the TaskManager; table moved or dropped between job submission and trigger.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/fbe9563fe4597e0b. Report an issue: GitHub.