{"record":{"id":"a1da5e5cfcda2f1f","repo":"apache/iceberg","slug":"unable-to-determine-whether-certain-files-are-orph-a1da5e","errorCode":null,"errorMessage":"Unable to determine whether certain files are orphan. Metadata references files that match listed/provided files except for authority/scheme. Please, inspect the conflicting authorities/schemes and provide which of them are equal by further configuring the action via equalSchemes() and equalAuthorities() methods. Set the prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that remaining conflicting authorities/schemes are different. It will be impossible to recover deleted files. Conflicting authorities/schemes: %s.","messagePattern":"Unable to determine whether certain files are orphan\\. Metadata references files that match listed/provided files except for authority/scheme\\. Please, inspect the conflicting authorities/schemes and provide which of them are equal by further configuring the action via equalSchemes\\(\\) and equalAuthorities\\(\\) methods\\. Set the prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that remaining conflicting authorities/schemes are different\\. It will be impossible to recover deleted files\\. Conflicting authorities/schemes: %s\\.","errorType":"validation","errorClass":"org.apache.iceberg.exceptions.ValidationException","httpStatus":null,"severity":"error","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java","lineNumber":372,"sourceCode":"\n    SetAccumulator<Pair<String, String>> conflicts = new SetAccumulator<>();\n    actualFileIdentDS.sparkSession().sparkContext().register(conflicts);\n\n    Column joinCond = actualFileIdentDS.col(\"path\").equalTo(validFileIdentDS.col(\"path\"));\n\n    Dataset<String> orphanFileDS =\n        actualFileIdentDS\n            .joinWith(validFileIdentDS, joinCond, \"leftouter\")\n            .mapPartitions(new FindOrphanFiles(prefixMismatchMode, conflicts), Encoders.STRING());\n\n    // Cache and force computation to populate conflicts accumulator\n    orphanFileDS = orphanFileDS.cache();\n\n    try {\n      orphanFileDS.count();\n\n      if (prefixMismatchMode == PrefixMismatchMode.ERROR && !conflicts.value().isEmpty()) {\n        throw new ValidationException(\n            \"Unable to determine whether certain files are orphan. Metadata references files that\"\n                + \" match listed/provided files except for authority/scheme. Please, inspect the\"\n                + \" conflicting authorities/schemes and provide which of them are equal by further\"\n                + \" configuring the action via equalSchemes() and equalAuthorities() methods. Set the\"\n                + \" prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting\"\n                + \" authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that\"\n                + \" remaining conflicting authorities/schemes are different. It will be impossible to\"\n                + \" recover deleted files. Conflicting authorities/schemes: %s.\",\n            conflicts.value());\n      }\n\n      return orphanFileDS;\n    } catch (Exception e) {\n      orphanFileDS.unpersist();\n      throw e;\n    }\n  }\n","sourceCodeStart":354,"sourceCodeEnd":390,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java#L354-L390","documentation":"DeleteOrphanFiles compares listed files with files referenced in metadata. When some paths match except for differing scheme or authority (e.g. s3a:// vs s3://, or different endpoint hostnames), the action cannot safely decide if they are orphan, and with PrefixMismatchMode.ERROR (the default) it throws this ValidationException listing the conflicting pairs.","triggerScenarios":"Running remove_orphan_files on a table whose metadata file locations and listed directory locations differ in scheme or authority (s3a vs s3, different S3 endpoint, IP vs hostname), while prefixMismatchMode is ERROR.","commonSituations":"Object storage endpoint changes (S3 endpoint/IP in authority); switching filesystem implementations (s3a vs s3); multi-cloud storage aliases; EMR vs self-hosted endpoint configuration differences.","solutions":["Inspect the listed conflicting authorities/schemes; declare equal ones via equalSchemes() / equalAuthorities() (or equalSchemes/equalAuthorities params in SQL)","Set prefix_mismatch_mode to NONE to ignore remaining conflicting locations (safe: no deletion of those)","Set prefix_mismatch_mode to DELETE only if absolutely certain the conflicting prefixes are the same storage; deleted files are unrecoverable"],"exampleFix":"// before\nCALL iceberg.system.remove_orphan_files(table => 'db.t') -- fails on s3a vs s3\n// after\nCALL iceberg.system.remove_orphan_files(\n  table => 'db.t',\n  equal_schemes => map('s3', 's3a'),\n  prefix_mismatch_mode => 'NONE');","handlingStrategy":"validation","validationCode":"// Pre-check: compare metadata path prefixes with listed prefixes\n// Declare equal schemes/authorities instead of relying on default ERROR mode\nCALL iceberg.system.remove_orphan_files(\n  table => 'db.t',\n  equal_schemes => map('s3', 's3a'),\n  equal_authorities => map('bucket1.s3.amazonaws.com', 'bucket1.s3.us-west-2.amazonaws.com'),\n  prefix_mismatch_mode => 'NONE');","typeGuard":null,"tryCatchPattern":"try { orphanFilesAction.execute(); } catch (ValidationException e) { /* re-run with equalSchemes/equalAuthorities or prefix_mismatch_mode=NONE */ }","preventionTips":["Use a consistent scheme and authority (endpoint) when accessing table locations","Configure equalSchemes/equalAuthorities up front when you know of storage aliases","Prefer prefix_mismatch_mode=NONE unless you can prove conflicting prefixes are the same storage","Remember deleted orphan files are unrecoverable — always dry-run/dry-verify first"],"tags":["spark","orphan-files","storage","prefix-mismatch","s3"],"backgroundTag":"invalid-argument-value","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}