apache/iceberg · error · UnsupportedOperationException

Unable to build the manifest files dataframe. The end versio

Error message

Unable to build the manifest files dataframe. The end version in use may contain invalid snapshots. Please choose an earlier version without invalid snapshots.

What it means

RewriteTablePathSparkAction.manifestsToRewrite builds a Spark DataFrame of manifest files for the version range being copied. If building/collecting that dataframe fails for any reason (invalid snapshots, missing metadata files in the copy range), it is rethrown as UnsupportedOperationException telling the user to pick an earlier end version without invalid snapshots.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java:552

      Table endStaticTable = newStaticTable(endVersionName, table.io());
      Dataset<Row> lastVersionFiles = manifestDS(endStaticTable).select("path");
      if (startMetadata == null) {
        return Sets.newHashSet(lastVersionFiles.distinct().as(Encoders.STRING()).collectAsList());
      } else {
        Set<Long> deltaSnapshotIds =
            deltaSnapshots.stream().map(Snapshot::snapshotId).collect(Collectors.toSet());
        return Sets.newHashSet(
            lastVersionFiles
                .distinct()
                .filter(
                    functions
                        .column(ManifestFile.SNAPSHOT_ID.name())
                        .isInCollection(deltaSnapshotIds))
                .as(Encoders.STRING())
                .collectAsList());
      }
    } catch (Exception e) {
      throw new UnsupportedOperationException(
          "Unable to build the manifest files dataframe. The end version in use may contain invalid snapshots. "
              + "Please choose an earlier version without invalid snapshots.",
          e);
    }
  }

  public static class RewriteContentFileResult extends RewriteResult<ContentFile<?>> {
    @Override
    public RewriteContentFileResult append(RewriteResult<ContentFile<?>> r1) {
      this.copyPlan().addAll(r1.copyPlan());
      this.toRewrite().addAll(r1.toRewrite());
      r1.rewrittenManifestLengths().forEach(this::addRewrittenManifestLength);
      return this;
    }

    public RewriteContentFileResult appendDataFile(RewriteResult<DataFile> r1) {
      this.copyPlan().addAll(r1.copyPlan());
      this.toRewrite().addAll(r1.toRewrite());

View on GitHub (pinned to 86d9c8fc54)

Solutions

Choose an earlier end snapshot/version that predates the invalid snapshots
Validate the snapshot chain (table.snapshots(), metadata JSON) and repair or roll back to a valid snapshot (rollbackTo)
Restore missing metadata files from backup or re-register the table at a valid metadata location
Inspect the underlying cause exception chained in the UnsupportedOperationException

Example fix

// before
SparkActions.get(table).rewriteTablePath(source, target).rewriteManifests(); // end version has invalid snapshot
// after: roll back to a valid snapshot first
table.manageSnapshots().rollbackTo(validSnapshotId);
SparkActions.get(table).rewriteTablePath(source, target).execute();

Defensive patterns

Strategy: validation

Validate before calling

// validate snapshot chain before rewriting table path
table.snapshots().forEach(s -> {
  Preconditions.checkArgument(s.snapshotId() > 0, "Invalid snapshot in chain");
  // ensure metadata files referenced exist in the copied range
});

Try / catch

try { action.rewriteManifests(); } catch (UnsupportedOperationException e) { /* pick earlier end version per message */ }

Prevention

Verify all snapshots in the chosen version range are valid and their metadata files exist
Avoid deleting/expiring metadata files before a path rewrite
Roll back to a known-good snapshot before copying the table path

When it happens

Trigger: Calling rewriteTablePath() (manifestsToRewrite stage) with an end version whose snapshot chain contains invalid/unreadable snapshots, causing the manifest-files dataframe construction or collectAsList to throw.

Common situations: Copying a table path up to a snapshot ID referencing expired or corrupted metadata; using a snapshot range that spans manually deleted metadata files; malformed ancestor chain after failed commits.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/6cf76f2a5e292175. Report an issue: GitHub.