apache/iceberg · error · UnsupportedOperationException

Unable to build the manifest files dataframe. The end versio

Error message

Unable to build the manifest files dataframe. The end version in use may contain invalid snapshots. Please choose an earlier version without invalid snapshots.

What it means

RewriteTablePathSparkAction.manifestsToRewrite builds a dataframe of manifests between the start and end versions and collects the snapshot IDs of delta snapshots. Any exception during this (invalid/expired snapshots, unreadable metadata, query failure) is wrapped in an UnsupportedOperationException saying the end version may contain invalid snapshots and advising an earlier version.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java:552

      Table endStaticTable = newStaticTable(endVersionName, table.io());
      Dataset<Row> lastVersionFiles = manifestDS(endStaticTable).select("path");
      if (startMetadata == null) {
        return Sets.newHashSet(lastVersionFiles.distinct().as(Encoders.STRING()).collectAsList());
      } else {
        Set<Long> deltaSnapshotIds =
            deltaSnapshots.stream().map(Snapshot::snapshotId).collect(Collectors.toSet());
        return Sets.newHashSet(
            lastVersionFiles
                .distinct()
                .filter(
                    functions
                        .column(ManifestFile.SNAPSHOT_ID.name())
                        .isInCollection(deltaSnapshotIds))
                .as(Encoders.STRING())
                .collectAsList());
      }
    } catch (Exception e) {
      throw new UnsupportedOperationException(
          "Unable to build the manifest files dataframe. The end version in use may contain invalid snapshots. "
              + "Please choose an earlier version without invalid snapshots.",
          e);
    }
  }

  public static class RewriteContentFileResult extends RewriteResult<ContentFile<?>> {
    @Override
    public RewriteContentFileResult append(RewriteResult<ContentFile<?>> r1) {
      this.copyPlan().addAll(r1.copyPlan());
      this.toRewrite().addAll(r1.toRewrite());
      r1.rewrittenManifestLengths().forEach(this::addRewrittenManifestLength);
      return this;
    }

    public RewriteContentFileResult appendDataFile(RewriteResult<DataFile> r1) {
      this.copyPlan().addAll(r1.copyPlan());
      this.toRewrite().addAll(r1.toRewrite());

View on GitHub (pinned to 86d9c8fc54)

Solutions

Pick an earlier, valid endVersion (or newer startVersion) so the delta snapshot range contains only valid snapshots, and re-run.
Verify the chosen snapshots exist: SELECT snapshot_id FROM <table>.snapshots / inspect metadata JSON, and ensure expireSnapshots isn't running concurrently.
Ensure the source table metadata and manifest files are fully readable (permissions, FileIO config) before re-running.
If snapshots were legitimately expired mid-migration, restart the copy procedure from a consistent starting version.

Example fix

// before
CALL catalog.system.rewrite_table_path(
  table => 'db.t', new_table => 'db.t_copy', end_version => 845129391); // version has expired snapshots
// after
CALL catalog.system.rewrite_table_path(
  table => 'db.t', new_table => 'db.t_copy', end_version => 845128000); // valid, non-expired version

Defensive patterns

Strategy: validation

Validate before calling

// Verify the end version's snapshots exist and are valid before copying
MetadataTableOperator.; // Scala/SQL alternative:
spark.sql(s"SELECT * FROM prod_catalog.db.t.snapshots").collect()
  .foreach(r => require(!r.isNullAt(0), s"Snapshot in range missing"));

Try / catch

try {
  proc.run();
} catch (UnsupportedOperationException e) {
  if (e.getMessage().contains("Unable to build the manifest files dataframe")) {
    LOG.error("Pick an earlier end_version with valid snapshots; check expireSnapshots activity", e);
  } else throw e;
}

Prevention

Never run expireSnapshots on the source table while a path rewrite is in progress or about to read it.
Record the snapshot/version ID of the end version before starting and use the snapshots metadata table to confirm validity.
Copy/migrate from an immutable, quiesced source table.

When it happens

Trigger: Calling rewriteTablePath (or the rewrite_table_path stored procedure) with an endVersion whose snapshot chain includes invalid/expired snapshots, corrupted metadata, or manifests that cannot be resolved by the delta-snapshot query.

Common situations: End version points to a snapshot removed by expireSnapshots or rewrite on the source table; source metadata files partially missing after a partial copy; referencing a version across a schema/format-affecting operation that broke the chain.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d55b899592c131b8. Report an issue: GitHub.