apache/iceberg · error · ValidationException

Unable to determine whether certain files are orphan. Metada

Error message

Unable to determine whether certain files are orphan. Metadata references files that match listed/provided files except for authority/scheme. Please, inspect the conflicting authorities/schemes and provide which of them are equal by further configuring the action via equalSchemes() and equalAuthorities() methods. Set the prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that remaining conflicting authorities/schemes are different. It will be impossible to recover deleted files. Conflicting authorities/schemes: %s.

What it means

DeleteOrphanFiles aborts when prefix-mismatch mode is ERROR and metadata file locations differ from listed files only by authority/scheme (e.g. s3a vs s3, or two S3 endpoints). The action cannot safely decide orphanhood, so it fails asking you to declare equal schemes/authorities or change the mode.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:372

    SetAccumulator<Pair<String, String>> conflicts = new SetAccumulator<>();
    actualFileIdentDS.sparkSession().sparkContext().register(conflicts);

    Column joinCond = actualFileIdentDS.col("path").equalTo(validFileIdentDS.col("path"));

    Dataset<String> orphanFileDS =
        actualFileIdentDS
            .joinWith(validFileIdentDS, joinCond, "leftouter")
            .mapPartitions(new FindOrphanFiles(prefixMismatchMode, conflicts), Encoders.STRING());

    // Cache and force computation to populate conflicts accumulator
    orphanFileDS = orphanFileDS.cache();

    try {
      orphanFileDS.count();

      if (prefixMismatchMode == PrefixMismatchMode.ERROR && !conflicts.value().isEmpty()) {
        throw new ValidationException(
            "Unable to determine whether certain files are orphan. Metadata references files that"
                + " match listed/provided files except for authority/scheme. Please, inspect the"
                + " conflicting authorities/schemes and provide which of them are equal by further"
                + " configuring the action via equalSchemes() and equalAuthorities() methods. Set the"
                + " prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting"
                + " authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that"
                + " remaining conflicting authorities/schemes are different. It will be impossible to"
                + " recover deleted files. Conflicting authorities/schemes: %s.",
            conflicts.value());
      }

      return orphanFileDS;
    } catch (Exception e) {
      orphanFileDS.unpersist();
      throw e;
    }
  }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Call .equalSchemes(map) and .equalAuthorities(map) declaring which schemes/authorities are equivalent (e.g. equalSchemes(ImmutableMap.of("s3","s3a")))
2. Set .prefixMismatchMode(PrefixMismatchMode.NONE) to skip conflicting locations instead of deleting them (safe default)
3. Only set PrefixMismatchMode.DELETE if absolutely certain the conflicting prefixes point to the same storage, since mismatches would then be treated as orphans and permanently deleted
4. Fix the underlying cause so metadata and listing use the same scheme/authority

Example fix

// before
SparkActions.get(spark).deleteOrphanFiles(table).execute();
// after
SparkActions.get(spark).deleteOrphanFiles(table)
    .prefixMismatchMode(PrefixMismatchMode.NONE)
    .equalSchemes(ImmutableMap.of("s3", "s3a"))
    .equalAuthorities(ImmutableMap.of("endpoint-a", "endpoint-b"))
    .execute();

// compare metadata location scheme/authority with listed files' scheme/authority before running
String scheme = URI.create(table.location()).getScheme();
if (!scheme.equals(listingScheme)) { configureEqualSchemesOrNoneMode(); }

try { action.execute(); } catch (ValidationException e) { /* re-run with equalSchemes/equalAuthorities or PrefixMismatchMode.NONE */ }

When it happens

Trigger: Running RemoveOrphanFilesSparkAction where table metadata paths and listed paths use different URI authority/scheme (e.g. metadata written with s3a:// but listing via s3://, or path-style vs virtual-hosted endpoints) with default PrefixMismatchMode.ERROR.

Common situations: Cluster migration (Hadoop S3A to Iceberg S3FileIO s3://), storage migration, mixing S3A with the Iceberg S3 catalog, bucket aliasing/CDN prefixes.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• Unable to determine whether certain files are orphan. Metada
• Unable to determine whether certain files are orphan. Metada
• Could not list sub directories, reached maximum depth:
• Could not list sub directories, reached maximum depth: ${MAX
• Delete failed for {}: {}

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f9509d3ddeb98d82. Report an issue: GitHub.