{"record":{"id":"8223e9f89e35b675","repo":"apache/iceberg","slug":"unable-to-determine-whether-certain-files-are-orph","errorCode":null,"errorMessage":"Unable to determine whether certain files are orphan. Metadata references files that match listed/provided files except for authority/scheme. Please, inspect the conflicting authorities/schemes and provide which of them are equal by further configuring the action via equalSchemes() and equalAuthorities() methods. Set the prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that remaining conflicting authorities/schemes are different. It will be impossible to recover deleted files. Conflicting authorities/schemes: %s.","messagePattern":"Unable to determine whether certain files are orphan\\. Metadata references files that match listed/provided files except for authority/scheme\\. Please, inspect the conflicting authorities/schemes and provide which of them are equal by further configuring the action via equalSchemes\\(\\) and equalAuthorities\\(\\) methods\\. Set the prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that remaining conflicting authorities/schemes are different\\. It will be impossible to recover deleted files\\. Conflicting authorities/schemes: %s\\.","errorType":"validation","errorClass":"ValidationException","httpStatus":null,"severity":"error","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java","lineNumber":372,"sourceCode":"\n    SetAccumulator<Pair<String, String>> conflicts = new SetAccumulator<>();\n    actualFileIdentDS.sparkSession().sparkContext().register(conflicts);\n\n    Column joinCond = actualFileIdentDS.col(\"path\").equalTo(validFileIdentDS.col(\"path\"));\n\n    Dataset<String> orphanFileDS =\n        actualFileIdentDS\n            .joinWith(validFileIdentDS, joinCond, \"leftouter\")\n            .mapPartitions(new FindOrphanFiles(prefixMismatchMode, conflicts), Encoders.STRING());\n\n    // Cache and force computation to populate conflicts accumulator\n    orphanFileDS = orphanFileDS.cache();\n\n    try {\n      orphanFileDS.count();\n\n      if (prefixMismatchMode == PrefixMismatchMode.ERROR && !conflicts.value().isEmpty()) {\n        throw new ValidationException(\n            \"Unable to determine whether certain files are orphan. Metadata references files that\"\n                + \" match listed/provided files except for authority/scheme. Please, inspect the\"\n                + \" conflicting authorities/schemes and provide which of them are equal by further\"\n                + \" configuring the action via equalSchemes() and equalAuthorities() methods. Set the\"\n                + \" prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting\"\n                + \" authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that\"\n                + \" remaining conflicting authorities/schemes are different. It will be impossible to\"\n                + \" recover deleted files. Conflicting authorities/schemes: %s.\",\n            conflicts.value());\n      }\n\n      return orphanFileDS;\n    } catch (Exception e) {\n      orphanFileDS.unpersist();\n      throw e;\n    }\n  }\n","sourceCodeStart":354,"sourceCodeEnd":390,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java#L354-L390","documentation":"DeleteOrphanFilesSparkAction threw a ValidationException while deciding which files are orphan. Some files listed from storage match files referenced in table metadata only if you ignore the authority/scheme part of the URI, so Iceberg refuses to guess whether they are safe to delete. It reports the conflicting authorities/schemes and asks you to declare which prefixes are equivalent (equalSchemes/equalAuthorities) or choose a PrefixMismatchMode of NONE (ignore) or DELETE (delete anyway).","triggerScenarios":"Running DELETE ORPHAN FILES (SparkActions.deleteOrphanFiles(table)) with default PrefixMismatchMode.ERROR where location URIs compare equal except for scheme/authority — e.g. s3://bucket vs s3a://bucket, or two bucket aliases like s3a://bkt vs s3a://alias-bkt, or filesystem:// vs no-authority relative URIs.","commonSituations":"Tables written via different Hadoop FileSystems (s3n/s3a/s3 legacy), storage migration or table rename across bucket aliases, multi-cluster setups where the same object store is addressed with different endpoint authorities, or running the action while a table was copied to a new location prefix.","solutions":["Compare the reported conflicting authorities/schemes; if they truly point to the same storage, call .equalSchemes(Map.of(\"s3\",\"s3a\")).equalAuthorities(Map.of(\"bkt\",\"alias-bkt\")) on the action before execute.","If conflicting locations should simply be ignored, set prefix mismatch mode to NONE: .prefixMismatchMode(PrefixMismatchMode.NONE).","Only if you are certain remaining conflicting locations are different storage, set .prefixMismatchMode(PrefixMismatchMode.DELETE) — files at those locations may be permanently deleted.","Fix the table metadata locations (e.g. rewrite table path / migrate to a consistent prefix) so listings and metadata use one canonical scheme/authority."],"exampleFix":"// before\nDeleteOrphanFiles.Result r = SparkActions.get(spark)\n    .deleteOrphanFiles(table)\n    .execute();\n// after\nDeleteOrphanFiles.Result r = SparkActions.get(spark)\n    .deleteOrphanFiles(table)\n    .equalSchemes(ImmutableMap.of(\"s3\", \"s3a\"))\n    .equalAuthorities(ImmutableMap.of(\"old-alias\", \"real-bucket\"))\n    .execute();","handlingStrategy":"validation","validationCode":"// Inspect metadata locations vs listing prefixes before running\nTableMetadata meta = table.operations().current();\nSet<String> locs = new HashSet<>();\nmeta.streams().forEach(null); // placeholder\nSystem.out.println(table.location()); // compare with the location you will list; if schemes/authorities differ, configure:\ndeleteOrphanFiles(table).equalSchemes(Map.of(\"s3\",\"s3a\")).equalAuthorities(Map.of(\"old\",\"new\")).execute();","typeGuard":null,"tryCatchPattern":"try {\n  action.execute();\n} catch (ValidationException e) {\n  if (e.getMessage().contains(\"orphan\")) {\n    action.prefixMismatchMode(PrefixMismatchMode.NONE).execute(); // after inspecting conflicts\n  } else throw e;\n}","preventionTips":["Keep one canonical URI scheme/authority per storage system in all table locations (prefer s3a:// or s3:// consistently).","After storage migrations, run with PrefixMismatchMode.NONE first and inspect reported conflicts before ever using DELETE.","Register all equivalent prefixes explicitly with equalSchemes/equalAuthorities as part of runbook."],"tags":["spark","delete-orphan-files","validation","uri-prefix"],"backgroundTag":"invalid-argument-value","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}