apache/iceberg · error · UnsupportedOperationException

Unable to build the manifest files dataframe. The end versio

Error message

Unable to build the manifest files dataframe. The end version in use may contain invalid snapshots. Please choose an earlier version without invalid snapshots.

What it means

RewriteTablePathSparkAction.manifestsToRewrite builds a Spark DataFrame of manifest files to copy and wraps any failure (Spark query, snapshot lookup, etc.) in UnsupportedOperationException, advising the user to pick an earlier end version that contains no invalid snapshots.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java:552

      Table endStaticTable = newStaticTable(endVersionName, table.io());
      Dataset<Row> lastVersionFiles = manifestDS(endStaticTable).select("path");
      if (startMetadata == null) {
        return Sets.newHashSet(lastVersionFiles.distinct().as(Encoders.STRING()).collectAsList());
      } else {
        Set<Long> deltaSnapshotIds =
            deltaSnapshots.stream().map(Snapshot::snapshotId).collect(Collectors.toSet());
        return Sets.newHashSet(
            lastVersionFiles
                .distinct()
                .filter(
                    functions
                        .column(ManifestFile.SNAPSHOT_ID.name())
                        .isInCollection(deltaSnapshotIds))
                .as(Encoders.STRING())
                .collectAsList());
      }
    } catch (Exception e) {
      throw new UnsupportedOperationException(
          "Unable to build the manifest files dataframe. The end version in use may contain invalid snapshots. "
              + "Please choose an earlier version without invalid snapshots.",
          e);
    }
  }

  public static class RewriteContentFileResult extends RewriteResult<ContentFile<?>> {
    @Override
    public RewriteContentFileResult append(RewriteResult<ContentFile<?>> r1) {
      this.copyPlan().addAll(r1.copyPlan());
      this.toRewrite().addAll(r1.toRewrite());
      r1.rewrittenManifestLengths().forEach(this::addRewrittenManifestLength);
      return this;
    }

    public RewriteContentFileResult appendDataFile(RewriteResult<DataFile> r1) {
      this.copyPlan().addAll(r1.copyPlan());
      this.toRewrite().addAll(r1.toRewrite());

View on GitHub (pinned to 86d9c8fc54)

Solutions

Choose an earlier end snapshot/version without invalid snapshots (e.g. via table.history() or rollBackTo).
Inspect snapshots with `spark.table(...).snapshots()` and remove/repair invalid snapshot references (e.g. removeSnapshots).
Read the wrapped cause (getCause()) to identify the concrete failing query and fix that underlying issue.

Example fix

// before
actions.rewriteTablePath(table).startVersion("v1").endVersion("v9").execute();
// after
// v9 contained invalid snapshots; use a known-good end version
actions.rewriteTablePath(table).startVersion("v1").endVersion("v7").execute();

Defensive patterns

Strategy: validation

Validate before calling

table.snapshots().forEach(s -> { /* validate each snapshot in [start, end] resolves */ });

Try / catch

try { action.execute(); } catch (UnsupportedOperationException e) { Throwable cause = e.getCause(); /* inspect cause, choose earlier end version */ }

Prevention

Choose end versions from healthy snapshot chains (verify with snapshots() metadata table)
Avoid targeting versions after interrupted expire/rewrite operations
Read the cause chain to diagnose the actual failing query

When it happens

Trigger: Calling manifestsToRewrite() with a rewriteTablePath end version whose snapshot chain contains invalid snapshots (e.g. dangling snapshot references, corrupted metadata) causing the manifest-collection query to throw.

Common situations: Migrating tables with interrupted expireSnapshots or partially-committed rewrites; end-version set to a snapshot with invalid ancestors; metadata corruption from failed operations.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f35f784038096b7e. Report an issue: GitHub.