{"record":{"id":"f52e10d031fa6135","repo":"apache/iceberg","slug":"unable-to-determine-whether-certain-files-are-orph-f52e10","errorCode":null,"errorMessage":"Unable to determine whether certain files are orphan. Found in filesystem: {} and in table: {}","messagePattern":"Unable to determine whether certain files are orphan\\. Found in filesystem: (.+?) and in table: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/OrphanFilesDetector.java","lineNumber":147,"sourceCode":"      } else {\n        FileURI actual =\n            new FileURI(\n                new Path(foundInFileSystem.value()).toUri(), equalSchemes, equalAuthorities);\n        if (hasMismatch(actual, foundInTablesList)) {\n          if (prefixMismatchMode == DeleteOrphanFiles.PrefixMismatchMode.DELETE) {\n            out.collect(foundInFileSystem.value());\n          } else if (prefixMismatchMode == DeleteOrphanFiles.PrefixMismatchMode.ERROR) {\n            ValidationException validationException =\n                new ValidationException(\n                    \"Unable to determine whether certain files are orphan. \"\n                        + \"Metadata references files that match listed/provided files except for authority/scheme. \"\n                        + \"Please, inspect the conflicting authorities/schemes and provide which of them are equal \"\n                        + \"by further configuring the action via equalSchemes() and equalAuthorities() methods. \"\n                        + \"Set the prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting \"\n                        + \"authorities/schemes or to 'DELETE' if you are ABSOLUTELY confident that remaining conflicting \"\n                        + \"authorities/schemes are different. It will be impossible to recover deleted files. \"\n                        + \"Conflicting authorities/schemes\");\n            LOG.warn(\n                \"Unable to determine whether certain files are orphan. Found in filesystem: {} and in table: {}\",\n                actual,\n                StringUtils.join(foundInTablesList, \",\"),\n                validationException);\n            ctx.output(\n                org.apache.iceberg.flink.maintenance.api.DeleteOrphanFiles.ERROR_STREAM,\n                validationException);\n          }\n        }\n      }\n    }\n\n    clearState();\n  }\n\n  private boolean hasMismatch(FileURI actual, List<FileURI> foundInTablesList) {\n    return foundInTablesList.stream()\n        .noneMatch(valid -> valid.schemeMatch(actual) && valid.authorityMatch(actual));","sourceCodeStart":129,"sourceCodeEnd":165,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/OrphanFilesDetector.java#L129-L165","documentation":"OrphanFilesDetector.onTimer compares files found on the filesystem with files referenced by the table. When locations have conflicting scheme/authority representations (e.g. s3a://bucket/path vs s3://bucket/path, or different authorities meaning the same store), a ValidationException is raised because the detector cannot safely decide which files are orphan. The warning logs the conflicting sets, emits the exception to the DeleteOrphanFiles ERROR_STREAM, and instructs the user to configure equalSchemes()/equalAuthorities() or set prefix mismatch mode.","triggerScenarios":"Raised in onTimer when prefix-mismatch validation finds filesystem locations whose scheme/authority differ from table-referenced locations and the mismatch mode is not NONE/DELETE and no equality mappings were configured via equalSchemes() or equalAuthorities().","commonSituations":"Tables written with s3:// but the orphan job configured with s3a://; mixed HDFS HA authority forms (nameservice vs host:port); EMR vs vanilla Hadoop default schemes; migrating storage without normalizing location URIs.","solutions":["Configure DeleteOrphanFiles.equalSchemes(\"s3\", \"s3a\") and/or equalAuthorities(...) to declare which prefixes are equivalent","Set prefix mismatch mode to NONE to skip conflicting locations (safe default)","Set prefix mismatch mode to DELETE only when you are absolutely certain the conflicting authorities/schemes are different storages — deleted files cannot be recovered","Normalize the table's location URIs so all writers use the same scheme/authority"],"exampleFix":"// before\nDeleteOrphanFiles.Builder builder = DeleteOrphanFiles.builder()...\n// after\nbuilder.equalSchemes(Map.of(\"s3\", \"s3a\"))\n       .equalAuthorities(Map.of(\"internal-ns\", \"namenode:8020\"))\n       .prefixMismatchMode(DeleteOrphanFiles.PrefixMismatchMode.NONE);","handlingStrategy":"validation","validationCode":"// Pre-check that all location prefixes agree before enabling deletes\nSet<String> schemes = locations.stream().map(l -> new Path(l).toUri().getScheme()).collect(Collectors.toSet());\nif (schemes.size() > 1) { configure equalSchemes(...) or set PrefixMismatchMode.NONE; }","typeGuard":null,"tryCatchPattern":"// Never use PrefixMismatchMode.DELETE unless prefixes were verified equivalent\ntry { detectorRun(); } catch (ValidationException e) { configureEqualSchemesAndAuthorities(e); }","preventionTips":["Configure equalSchemes()/equalAuthorities() when the same store is reachable under multiple URIs","Default prefix mismatch mode to NONE; reserve DELETE for verified-distinct stores","Normalize table location URIs across all writers","Treat this warning as blocking — orphan detection results are incomplete"],"tags":["flink","orphan-files","location-uri","validation"],"backgroundTag":"conflicting-config-options","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}