apache/iceberg · warning

Unable to determine whether certain files are orphan. Found

Error message

Unable to determine whether certain files are orphan. Found in filesystem: {} and in table: {}

What it means

OrphanFilesDetector.onTimer compares files found on the filesystem with files referenced by the table. When locations have conflicting scheme/authority representations (e.g. s3a://bucket/path vs s3://bucket/path, or different authorities meaning the same store), a ValidationException is raised because the detector cannot safely decide which files are orphan. The warning logs the conflicting sets, emits the exception to the DeleteOrphanFiles ERROR_STREAM, and instructs the user to configure equalSchemes()/equalAuthorities() or set prefix mismatch mode.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/OrphanFilesDetector.java:147

      } else {
        FileURI actual =
            new FileURI(
                new Path(foundInFileSystem.value()).toUri(), equalSchemes, equalAuthorities);
        if (hasMismatch(actual, foundInTablesList)) {
          if (prefixMismatchMode == DeleteOrphanFiles.PrefixMismatchMode.DELETE) {
            out.collect(foundInFileSystem.value());
          } else if (prefixMismatchMode == DeleteOrphanFiles.PrefixMismatchMode.ERROR) {
            ValidationException validationException =
                new ValidationException(
                    "Unable to determine whether certain files are orphan. "
                        + "Metadata references files that match listed/provided files except for authority/scheme. "
                        + "Please, inspect the conflicting authorities/schemes and provide which of them are equal "
                        + "by further configuring the action via equalSchemes() and equalAuthorities() methods. "
                        + "Set the prefix mismatch mode to 'NONE' to ignore remaining locations with conflicting "
                        + "authorities/schemes or to 'DELETE' if you are ABSOLUTELY confident that remaining conflicting "
                        + "authorities/schemes are different. It will be impossible to recover deleted files. "
                        + "Conflicting authorities/schemes");
            LOG.warn(
                "Unable to determine whether certain files are orphan. Found in filesystem: {} and in table: {}",
                actual,
                StringUtils.join(foundInTablesList, ","),
                validationException);
            ctx.output(
                org.apache.iceberg.flink.maintenance.api.DeleteOrphanFiles.ERROR_STREAM,
                validationException);
          }
        }
      }
    }

    clearState();
  }

  private boolean hasMismatch(FileURI actual, List<FileURI> foundInTablesList) {
    return foundInTablesList.stream()
        .noneMatch(valid -> valid.schemeMatch(actual) && valid.authorityMatch(actual));

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Configure DeleteOrphanFiles.equalSchemes("s3", "s3a") and/or equalAuthorities(...) to declare which prefixes are equivalent
2. Set prefix mismatch mode to NONE to skip conflicting locations (safe default)
3. Set prefix mismatch mode to DELETE only when you are absolutely certain the conflicting authorities/schemes are different storages — deleted files cannot be recovered
4. Normalize the table's location URIs so all writers use the same scheme/authority

Example fix

// before
DeleteOrphanFiles.Builder builder = DeleteOrphanFiles.builder()...
// after
builder.equalSchemes(Map.of("s3", "s3a"))
       .equalAuthorities(Map.of("internal-ns", "namenode:8020"))
       .prefixMismatchMode(DeleteOrphanFiles.PrefixMismatchMode.NONE);

// Pre-check that all location prefixes agree before enabling deletes
Set<String> schemes = locations.stream().map(l -> new Path(l).toUri().getScheme()).collect(Collectors.toSet());
if (schemes.size() > 1) { configure equalSchemes(...) or set PrefixMismatchMode.NONE; }

// Never use PrefixMismatchMode.DELETE unless prefixes were verified equivalent
try { detectorRun(); } catch (ValidationException e) { configureEqualSchemesAndAuthorities(e); }

When it happens

Trigger: Raised in onTimer when prefix-mismatch validation finds filesystem locations whose scheme/authority differ from table-referenced locations and the mismatch mode is not NONE/DELETE and no equality mappings were configured via equalSchemes() or equalAuthorities().

Common situations: Tables written with s3:// but the orphan job configured with s3a://; mixed HDFS HA authority forms (nameservice vs host:port); EMR vs vanilla Hadoop default schemes; migrating storage without normalizing location URIs.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

• Invalid precision: ${precision}
• Invalid precision: ${precision}
• Database properties should not contain key: 'comment'.
• Invalid primary key '%s'. A primary key must not contain dup
• Invalid primary key '%s'. Column '%s' does not exist.

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f52e10d031fa6135. Report an issue: GitHub.