{"record":{"id":"f9509d3ddeb98d82","repo":"apache/iceberg","slug":"unable-to-determine-whether-certain-files-are-orph-f9509d","errorCode":null,"errorMessage":"Unable to determine whether certain files are orphan. Metadata references files that match listed/provided files except for authority/scheme. Please, inspect the conflicting authorities/schemes and provide which of them are equal by further configuring the action via equalSchemes() and equalAuthorities() methods. Set the prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that remaining conflicting authorities/schemes are different. It will be impossible to recover deleted files. Conflicting authorities/schemes: %s.","messagePattern":"Unable to determine whether certain files are orphan\\. Metadata references files that match listed/provided files except for authority/scheme\\. Please, inspect the conflicting authorities/schemes and provide which of them are equal by further configuring the action via equalSchemes\\(\\) and equalAuthorities\\(\\) methods\\. Set the prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that remaining conflicting authorities/schemes are different\\. It will be impossible to recover deleted files\\. Conflicting authorities/schemes: %s\\.","errorType":"validation","errorClass":"ValidationException","httpStatus":null,"severity":"error","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java","lineNumber":372,"sourceCode":"\n    SetAccumulator<Pair<String, String>> conflicts = new SetAccumulator<>();\n    actualFileIdentDS.sparkSession().sparkContext().register(conflicts);\n\n    Column joinCond = actualFileIdentDS.col(\"path\").equalTo(validFileIdentDS.col(\"path\"));\n\n    Dataset<String> orphanFileDS =\n        actualFileIdentDS\n            .joinWith(validFileIdentDS, joinCond, \"leftouter\")\n            .mapPartitions(new FindOrphanFiles(prefixMismatchMode, conflicts), Encoders.STRING());\n\n    // Cache and force computation to populate conflicts accumulator\n    orphanFileDS = orphanFileDS.cache();\n\n    try {\n      orphanFileDS.count();\n\n      if (prefixMismatchMode == PrefixMismatchMode.ERROR && !conflicts.value().isEmpty()) {\n        throw new ValidationException(\n            \"Unable to determine whether certain files are orphan. Metadata references files that\"\n                + \" match listed/provided files except for authority/scheme. Please, inspect the\"\n                + \" conflicting authorities/schemes and provide which of them are equal by further\"\n                + \" configuring the action via equalSchemes() and equalAuthorities() methods. Set the\"\n                + \" prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting\"\n                + \" authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that\"\n                + \" remaining conflicting authorities/schemes are different. It will be impossible to\"\n                + \" recover deleted files. Conflicting authorities/schemes: %s.\",\n            conflicts.value());\n      }\n\n      return orphanFileDS;\n    } catch (Exception e) {\n      orphanFileDS.unpersist();\n      throw e;\n    }\n  }\n","sourceCodeStart":354,"sourceCodeEnd":390,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java#L354-L390","documentation":"DeleteOrphanFiles aborts when prefix-mismatch mode is ERROR and metadata file locations differ from listed files only by authority/scheme (e.g. s3a vs s3, or two S3 endpoints). The action cannot safely decide orphanhood, so it fails asking you to declare equal schemes/authorities or change the mode.","triggerScenarios":"Running RemoveOrphanFilesSparkAction where table metadata paths and listed paths use different URI authority/scheme (e.g. metadata written with s3a:// but listing via s3://, or path-style vs virtual-hosted endpoints) with default PrefixMismatchMode.ERROR.","commonSituations":"Cluster migration (Hadoop S3A to Iceberg S3FileIO s3://), storage migration, mixing S3A with the Iceberg S3 catalog, bucket aliasing/CDN prefixes.","solutions":["Call .equalSchemes(map) and .equalAuthorities(map) declaring which schemes/authorities are equivalent (e.g. equalSchemes(ImmutableMap.of(\"s3\",\"s3a\")))","Set .prefixMismatchMode(PrefixMismatchMode.NONE) to skip conflicting locations instead of deleting them (safe default)","Only set PrefixMismatchMode.DELETE if absolutely certain the conflicting prefixes point to the same storage, since mismatches would then be treated as orphans and permanently deleted","Fix the underlying cause so metadata and listing use the same scheme/authority"],"exampleFix":"// before\nSparkActions.get(spark).deleteOrphanFiles(table).execute();\n// after\nSparkActions.get(spark).deleteOrphanFiles(table)\n    .prefixMismatchMode(PrefixMismatchMode.NONE)\n    .equalSchemes(ImmutableMap.of(\"s3\", \"s3a\"))\n    .equalAuthorities(ImmutableMap.of(\"endpoint-a\", \"endpoint-b\"))\n    .execute();","handlingStrategy":"validation","validationCode":"// compare metadata location scheme/authority with listed files' scheme/authority before running\nString scheme = URI.create(table.location()).getScheme();\nif (!scheme.equals(listingScheme)) { configureEqualSchemesOrNoneMode(); }","typeGuard":null,"tryCatchPattern":"try { action.execute(); } catch (ValidationException e) { /* re-run with equalSchemes/equalAuthorities or PrefixMismatchMode.NONE */ }","preventionTips":["Keep metadata and listing prefixes consistent","Declare equalSchemes/equalAuthorities when migrating S3A<->S3FileIO","Default to PrefixMismatchMode.NONE until prefixes are verified","Never use DELETE mode without verifying storage identity"],"tags":["spark","delete-orphan-files","prefix-mismatch","s3"],"backgroundTag":"invalid-config-value","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}