{"record":{"id":"d55b899592c131b8","repo":"apache/iceberg","slug":"unable-to-build-the-manifest-files-dataframe-the","errorCode":null,"errorMessage":"Unable to build the manifest files dataframe. The end version in use may contain invalid snapshots. Please choose an earlier version without invalid snapshots.","messagePattern":"Unable to build the manifest files dataframe\\. The end version in use may contain invalid snapshots\\. Please choose an earlier version without invalid snapshots\\.","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java","lineNumber":552,"sourceCode":"      Table endStaticTable = newStaticTable(endVersionName, table.io());\n      Dataset<Row> lastVersionFiles = manifestDS(endStaticTable).select(\"path\");\n      if (startMetadata == null) {\n        return Sets.newHashSet(lastVersionFiles.distinct().as(Encoders.STRING()).collectAsList());\n      } else {\n        Set<Long> deltaSnapshotIds =\n            deltaSnapshots.stream().map(Snapshot::snapshotId).collect(Collectors.toSet());\n        return Sets.newHashSet(\n            lastVersionFiles\n                .distinct()\n                .filter(\n                    functions\n                        .column(ManifestFile.SNAPSHOT_ID.name())\n                        .isInCollection(deltaSnapshotIds))\n                .as(Encoders.STRING())\n                .collectAsList());\n      }\n    } catch (Exception e) {\n      throw new UnsupportedOperationException(\n          \"Unable to build the manifest files dataframe. The end version in use may contain invalid snapshots. \"\n              + \"Please choose an earlier version without invalid snapshots.\",\n          e);\n    }\n  }\n\n  public static class RewriteContentFileResult extends RewriteResult<ContentFile<?>> {\n    @Override\n    public RewriteContentFileResult append(RewriteResult<ContentFile<?>> r1) {\n      this.copyPlan().addAll(r1.copyPlan());\n      this.toRewrite().addAll(r1.toRewrite());\n      r1.rewrittenManifestLengths().forEach(this::addRewrittenManifestLength);\n      return this;\n    }\n\n    public RewriteContentFileResult appendDataFile(RewriteResult<DataFile> r1) {\n      this.copyPlan().addAll(r1.copyPlan());\n      this.toRewrite().addAll(r1.toRewrite());","sourceCodeStart":534,"sourceCodeEnd":570,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java#L534-L570","documentation":"RewriteTablePathSparkAction.manifestsToRewrite builds a dataframe of manifests between the start and end versions and collects the snapshot IDs of delta snapshots. Any exception during this (invalid/expired snapshots, unreadable metadata, query failure) is wrapped in an UnsupportedOperationException saying the end version may contain invalid snapshots and advising an earlier version.","triggerScenarios":"Calling rewriteTablePath (or the rewrite_table_path stored procedure) with an endVersion whose snapshot chain includes invalid/expired snapshots, corrupted metadata, or manifests that cannot be resolved by the delta-snapshot query.","commonSituations":"End version points to a snapshot removed by expireSnapshots or rewrite on the source table; source metadata files partially missing after a partial copy; referencing a version across a schema/format-affecting operation that broke the chain.","solutions":["Pick an earlier, valid endVersion (or newer startVersion) so the delta snapshot range contains only valid snapshots, and re-run.","Verify the chosen snapshots exist: SELECT snapshot_id FROM <table>.snapshots / inspect metadata JSON, and ensure expireSnapshots isn't running concurrently.","Ensure the source table metadata and manifest files are fully readable (permissions, FileIO config) before re-running.","If snapshots were legitimately expired mid-migration, restart the copy procedure from a consistent starting version."],"exampleFix":"// before\nCALL catalog.system.rewrite_table_path(\n  table => 'db.t', new_table => 'db.t_copy', end_version => 845129391); // version has expired snapshots\n// after\nCALL catalog.system.rewrite_table_path(\n  table => 'db.t', new_table => 'db.t_copy', end_version => 845128000); // valid, non-expired version","handlingStrategy":"validation","validationCode":"// Verify the end version's snapshots exist and are valid before copying\nMetadataTableOperator.; // Scala/SQL alternative:\nspark.sql(s\"SELECT * FROM prod_catalog.db.t.snapshots\").collect()\n  .foreach(r => require(!r.isNullAt(0), s\"Snapshot in range missing\"));","typeGuard":null,"tryCatchPattern":"try {\n  proc.run();\n} catch (UnsupportedOperationException e) {\n  if (e.getMessage().contains(\"Unable to build the manifest files dataframe\")) {\n    LOG.error(\"Pick an earlier end_version with valid snapshots; check expireSnapshots activity\", e);\n  } else throw e;\n}","preventionTips":["Never run expireSnapshots on the source table while a path rewrite is in progress or about to read it.","Record the snapshot/version ID of the end version before starting and use the snapshots metadata table to confirm validity.","Copy/migrate from an immutable, quiesced source table."],"tags":["spark","rewrite-table-path","invalid-snapshot","table-copy"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}