apache/beam · error · RuntimeException

Failed to plan deleted rows tasks

Error message

Failed to plan deleted rows tasks

What it means

processSnapshotForDeletedRowsTasks plans tasks for rows deleted via delete files in a snapshot. Exceptions encountered while reading delete manifests or building deleted-row tasks are wrapped into this RuntimeException.

Solutions

1. Inspect the wrapped cause to find which manifest or delete file failed and verify its availability
2. Retry the pipeline — transient storage errors are the most frequent cause
3. Check for concurrent table maintenance (compaction, expiration) racing with the changelog scan and coordinate/serialize those operations

Example fix

// before
table.newIncrementalChangelogScan() ... // run concurrently with rewriteDataFiles/compaction
// after
// pause table maintenance jobs, or run the scan against a snapshot range untouched by compaction
scan = table.newIncrementalChangelogScan().fromSnapshotId(preCompactionSnap);

// preflight: read all delete manifests in range
for (ManifestFile m : deleteManifestsInRange) {
  try (FileIO io = table.io()) {
    io.newInputFile(m.path()).exists();
  }
}

try {
  planDeletedRowsTasks(...);
} catch (RuntimeException e) {
  if (e.getMessage().equals("Failed to plan deleted rows tasks")) {
    // inspect e.getCause(); retry transient storage failures
  } else throw e;
}

When it happens

Trigger: Calling planDeletedRowsTasks when a delete manifest entry cannot be read, its delete file is missing, or task construction (residual evaluation, schema/spec resolution) fails.

Common situations: Delete files orphaned by concurrent compaction; object-store I/O errors while opening manifests; incompatible delete-file format after an Iceberg upgrade.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to collect file statuses for snapshot
• Failed to read delete manifest
• Position delete index cardinality exceeds Integer.MAX_VALUE
• A schema is required to write non-schema'd data.
• A sink must inherit iobase.Sink, iobase.NativeSink, or be a…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/90b23443cc489580. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/iceberg/BeamBaseIncrementalChangelogScan.java:687

                  return ResidualEvaluator.of(spec, residualFilter, isCaseSensitive());
                });

        tasks.add(
            new BaseDeletedRowsScanTask(
                changeOrdinal,
                snapshot.snapshotId(),
                dataFile.copy(shouldKeepStats()),
                addedDeletes,
                existingDeletes,
                schemaString,
                specString,
                residuals));

        // Mark this file as processed for this snapshot
        alreadyProcessedPaths.add(filePath);
      }
    } catch (Exception e) {
      throw new RuntimeException("Failed to plan deleted rows tasks", e);
    }
  }

  private boolean shouldKeepStats() {
    Set<Integer> columns = columnsToKeepStats();
    return columns != null && !columns.isEmpty();
  }

  /**
   * Loads delete files from manifests by parsing each manifest.
   *
   * @param manifests the delete manifests to load
   * @return list of delete files
   */
  private Iterable<DeleteFile> loadDeleteFiles(
      List<ManifestFile> manifests, Long targetSnapshotId) {
    Queue<DeleteFile> allDeleteFiles = new ConcurrentLinkedQueue<>();

View on GitHub (pinned to 12126d8942)