apache/iceberg · warning

Exception listing files for

Error message

Exception listing files for {} at {}

What it means

A WARN log emitted when the ListFileSystemFiles operator fails while listing files at a location during orphan-file detection. The exception is routed to the DeleteOrphanFiles ERROR_STREAM side output and the errorCounter is incremented, so downstream committers can fail the task; the listing attempt for that directory is abandoned.

Solutions

1. Verify the table/root location exists and is correct; recreate or fix the path if it was moved or deleted.
2. Grant the Flink job's principal LIST permission on the directory (HDFS ACL, S3/IAM s3:ListBucket).
3. Check filesystem availability and retry; the operator emits the error to the side output so the maintenance task can be safely rerun.
4. Reduce listing pressure (fewer concurrent workers, shallower prefixes) if throttling is the cause.

Example fix

// before: assume directory always listable
Files.listRecursive(location, ...);
// after: precheck existence before running the task
FileSystem fs = new Path(location).getFileSystem(conf);
Preconditions.checkArgument(fs.exists(new Path(location)), "Location %s missing", location);

org.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.FileSystem.get(conf);
org.apache.hadoop.fs.Path p = new org.apache.hadoop.fs.Path(location);
if (!fs.exists(p) || !fs.getFileStatus(p).isDirectory()) {
  throw new IllegalArgumentException("Cannot list: location missing or not a directory: " + location);
}

boolean listable(FileSystem fs, Path dir) throws IOException {
  return fs.exists(dir) && fs.getFileStatus(dir).isDirectory();
}

try {
  listRecursive(location, out);
} catch (Exception e) {
  ctx.output(DeleteOrphanFiles.ERROR_STREAM, e);
  errorCounter.inc();
}

When it happens

Trigger: processElement() calls Hadoop/FileSystem listStatus (recursive listing) on the table location and any Exception occurs — directory does not exist, permission denied, or filesystem I/O error. Emitted with the location and processing timestamp.

Common situations: Table location path typo or removed/moved directory; missing HDFS/S3 list permissions for the Flink job's user; S3 throttling on deep prefixes; filesystem outage or network partition.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Exception listing files for
• Exception listing files for
• Could not list sub directories, reached maximum depth:
• Exception planning scan for
• Exception planning scan for

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/e1679e2acf056fdb. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/ListFileSystemFiles.java:122

            "Cannot use prefix listing with FileIO %s which does not support prefix operations.",
            io);

        FileSystemWalker.listDirRecursivelyWithFileIO(
            (SupportsPrefixOperations) io, location, specs, predicate, out::collect);
      } else {
        Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;
        FileSystemWalker.listDirRecursivelyWithHadoop(
            location,
            specs,
            predicate,
            configuration,
            Integer.MAX_VALUE,
            Integer.MAX_VALUE,
            dir -> {},
            out::collect);
      }
    } catch (Exception e) {
      LOG.warn("Exception listing files for {} at {}", location, ctx.timestamp(), e);
      ctx.output(DeleteOrphanFiles.ERROR_STREAM, e);
      errorCounter.inc();
    }
  }

  @Override
  public void close() throws Exception {
    super.close();
    tableLoader.close();
  }
}

View on GitHub (pinned to 86d9c8fc54)