apache/iceberg · error · ValidationException

Unable to determine whether certain files are orphan. Metada

Error message

Unable to determine whether certain files are orphan. Metadata references files that match listed/provided files except for authority/scheme. Please, inspect the conflicting authorities/schemes and provide which of them are equal by further configuring the action via equalSchemes() and equalAuthorities() methods. Set the prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that remaining conflicting authorities/schemes are different. It will be impossible to recover deleted files. Conflicting authorities/schemes: %s.

What it means

DeleteOrphanFilesSparkAction threw a ValidationException while deciding which files are orphan. Some files listed from storage match files referenced in table metadata only if you ignore the authority/scheme part of the URI, so Iceberg refuses to guess whether they are safe to delete. It reports the conflicting authorities/schemes and asks you to declare which prefixes are equivalent (equalSchemes/equalAuthorities) or choose a PrefixMismatchMode of NONE (ignore) or DELETE (delete anyway).

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:372

    SetAccumulator<Pair<String, String>> conflicts = new SetAccumulator<>();
    actualFileIdentDS.sparkSession().sparkContext().register(conflicts);

    Column joinCond = actualFileIdentDS.col("path").equalTo(validFileIdentDS.col("path"));

    Dataset<String> orphanFileDS =
        actualFileIdentDS
            .joinWith(validFileIdentDS, joinCond, "leftouter")
            .mapPartitions(new FindOrphanFiles(prefixMismatchMode, conflicts), Encoders.STRING());

    // Cache and force computation to populate conflicts accumulator
    orphanFileDS = orphanFileDS.cache();

    try {
      orphanFileDS.count();

      if (prefixMismatchMode == PrefixMismatchMode.ERROR && !conflicts.value().isEmpty()) {
        throw new ValidationException(
            "Unable to determine whether certain files are orphan. Metadata references files that"
                + " match listed/provided files except for authority/scheme. Please, inspect the"
                + " conflicting authorities/schemes and provide which of them are equal by further"
                + " configuring the action via equalSchemes() and equalAuthorities() methods. Set the"
                + " prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting"
                + " authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that"
                + " remaining conflicting authorities/schemes are different. It will be impossible to"
                + " recover deleted files. Conflicting authorities/schemes: %s.",
            conflicts.value());
      }

      return orphanFileDS;
    } catch (Exception e) {
      orphanFileDS.unpersist();
      throw e;
    }
  }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Compare the reported conflicting authorities/schemes; if they truly point to the same storage, call .equalSchemes(Map.of("s3","s3a")).equalAuthorities(Map.of("bkt","alias-bkt")) on the action before execute.
2. If conflicting locations should simply be ignored, set prefix mismatch mode to NONE: .prefixMismatchMode(PrefixMismatchMode.NONE).
3. Only if you are certain remaining conflicting locations are different storage, set .prefixMismatchMode(PrefixMismatchMode.DELETE) — files at those locations may be permanently deleted.
4. Fix the table metadata locations (e.g. rewrite table path / migrate to a consistent prefix) so listings and metadata use one canonical scheme/authority.

Example fix

// before
DeleteOrphanFiles.Result r = SparkActions.get(spark)
    .deleteOrphanFiles(table)
    .execute();
// after
DeleteOrphanFiles.Result r = SparkActions.get(spark)
    .deleteOrphanFiles(table)
    .equalSchemes(ImmutableMap.of("s3", "s3a"))
    .equalAuthorities(ImmutableMap.of("old-alias", "real-bucket"))
    .execute();

// Inspect metadata locations vs listing prefixes before running
TableMetadata meta = table.operations().current();
Set<String> locs = new HashSet<>();
meta.streams().forEach(null); // placeholder
System.out.println(table.location()); // compare with the location you will list; if schemes/authorities differ, configure:
deleteOrphanFiles(table).equalSchemes(Map.of("s3","s3a")).equalAuthorities(Map.of("old","new")).execute();

try {
  action.execute();
} catch (ValidationException e) {
  if (e.getMessage().contains("orphan")) {
    action.prefixMismatchMode(PrefixMismatchMode.NONE).execute(); // after inspecting conflicts
  } else throw e;
}

When it happens

Trigger: Running DELETE ORPHAN FILES (SparkActions.deleteOrphanFiles(table)) with default PrefixMismatchMode.ERROR where location URIs compare equal except for scheme/authority — e.g. s3://bucket vs s3a://bucket, or two bucket aliases like s3a://bkt vs s3a://alias-bkt, or filesystem:// vs no-authority relative URIs.

Common situations: Tables written via different Hadoop FileSystems (s3n/s3a/s3 legacy), storage migration or table rename across bucket aliases, multi-cluster setups where the same object store is addressed with different endpoint authorities, or running the action while a table was copied to a new location prefix.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Illegal file type: %s
• Could not list sub directories, reached maximum depth:
• Cannot remove orphan files with an interval less than 24 hou
• Illegal file type: %s
• Unable to determine whether certain files are orphan. Metada

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/8223e9f89e35b675. Report an issue: GitHub.