apache/iceberg · error · org.apache.iceberg.exceptions.ValidationException

Unable to determine whether certain files are orphan. Metada

Error message

Unable to determine whether certain files are orphan. Metadata references files that match listed/provided files except for authority/scheme. Please, inspect the conflicting authorities/schemes and provide which of them are equal by further configuring the action via equalSchemes() and equalAuthorities() methods. Set the prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that remaining conflicting authorities/schemes are different. It will be impossible to recover deleted files. Conflicting authorities/schemes: %s.

What it means

DeleteOrphanFiles compares listed files with files referenced in metadata. When some paths match except for differing scheme or authority (e.g. s3a:// vs s3://, or different endpoint hostnames), the action cannot safely decide if they are orphan, and with PrefixMismatchMode.ERROR (the default) it throws this ValidationException listing the conflicting pairs.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:372

    SetAccumulator<Pair<String, String>> conflicts = new SetAccumulator<>();
    actualFileIdentDS.sparkSession().sparkContext().register(conflicts);

    Column joinCond = actualFileIdentDS.col("path").equalTo(validFileIdentDS.col("path"));

    Dataset<String> orphanFileDS =
        actualFileIdentDS
            .joinWith(validFileIdentDS, joinCond, "leftouter")
            .mapPartitions(new FindOrphanFiles(prefixMismatchMode, conflicts), Encoders.STRING());

    // Cache and force computation to populate conflicts accumulator
    orphanFileDS = orphanFileDS.cache();

    try {
      orphanFileDS.count();

      if (prefixMismatchMode == PrefixMismatchMode.ERROR && !conflicts.value().isEmpty()) {
        throw new ValidationException(
            "Unable to determine whether certain files are orphan. Metadata references files that"
                + " match listed/provided files except for authority/scheme. Please, inspect the"
                + " conflicting authorities/schemes and provide which of them are equal by further"
                + " configuring the action via equalSchemes() and equalAuthorities() methods. Set the"
                + " prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting"
                + " authorities/schemes or to 'DELETE' iff you are ABSOLUTELY confident that"
                + " remaining conflicting authorities/schemes are different. It will be impossible to"
                + " recover deleted files. Conflicting authorities/schemes: %s.",
            conflicts.value());
      }

      return orphanFileDS;
    } catch (Exception e) {
      orphanFileDS.unpersist();
      throw e;
    }
  }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Inspect the listed conflicting authorities/schemes; declare equal ones via equalSchemes() / equalAuthorities() (or equalSchemes/equalAuthorities params in SQL)
2. Set prefix_mismatch_mode to NONE to ignore remaining conflicting locations (safe: no deletion of those)
3. Set prefix_mismatch_mode to DELETE only if absolutely certain the conflicting prefixes are the same storage; deleted files are unrecoverable

Example fix

// before
CALL iceberg.system.remove_orphan_files(table => 'db.t') -- fails on s3a vs s3
// after
CALL iceberg.system.remove_orphan_files(
  table => 'db.t',
  equal_schemes => map('s3', 's3a'),
  prefix_mismatch_mode => 'NONE');

// Pre-check: compare metadata path prefixes with listed prefixes
// Declare equal schemes/authorities instead of relying on default ERROR mode
CALL iceberg.system.remove_orphan_files(
  table => 'db.t',
  equal_schemes => map('s3', 's3a'),
  equal_authorities => map('bucket1.s3.amazonaws.com', 'bucket1.s3.us-west-2.amazonaws.com'),
  prefix_mismatch_mode => 'NONE');

try { orphanFilesAction.execute(); } catch (ValidationException e) { /* re-run with equalSchemes/equalAuthorities or prefix_mismatch_mode=NONE */ }

When it happens

Trigger: Running remove_orphan_files on a table whose metadata file locations and listed directory locations differ in scheme or authority (s3a vs s3, different S3 endpoint, IP vs hostname), while prefixMismatchMode is ERROR.

Common situations: Object storage endpoint changes (S3 endpoint/IP in authority); switching filesystem implementations (s3a vs s3); multi-cloud storage aliases; EMR vs self-hosted endpoint configuration differences.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Unable to determine whether certain files are orphan. Metada
• Cannot remove orphan files with an interval less than 24 hou
• Cannot remove orphan files with an interval less than 24 hou
• Cannot remove orphan files with an interval less than 24 hou
• Table refresh failed

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/a1da5e5cfcda2f1f. Report an issue: GitHub.