apache/iceberg · warning

Exception listing files for

Error message

Exception listing files for {} at {}

What it means

ListFileSystemFiles.processElement lists filesystem locations for the DeleteOrphanFiles maintenance flow and emits each found file. When the listing operation throws, the exception is logged as a warning with the location and timestamp, emitted to the DeleteOrphanFiles ERROR_STREAM side output, and counted in errorCounter — the job keeps running so other locations can still be scanned.

Solutions

1. Consume the DeleteOrphanFiles.ERROR_STREAM side output in your job to inspect the actual exception
2. Verify the configured location exists and the scheme/authority is correct (s3a:// vs s3://)
3. Fix filesystem permissions/IAM policies for the TaskManager's identity
4. If listing huge prefixes, tune listing parallelism or reduce scope; retry on next trigger since errorCounter tracks failures

// Before running the action, verify the location exists
FileSystem fs = new Path(location).getFileSystem(conf);
Preconditions.checkArgument(fs.exists(new Path(location)), "Location missing: " + location);

// Consume the side output where the exception is forwarded
SingleOutputStreamOperator<?> result = ...;
result.getSideOutput(DeleteOrphanFiles.ERROR_STREAM)
      .executeAndCollect().forEachRemaining(e -> log.error("orphan-scan listing failed", e));

When it happens

Trigger: Raised in processElement when the FileSystem listFiles/recursive walk over `location` throws — e.g. FileNotFoundException for a missing directory, permission errors, or object-store listing failures (throttling, invalid prefix).

Common situations: Configured location does not exist yet (first run before any writes); S3/GCS list throttling on huge prefixes; wrong path/scheme configuration (s3a vs s3); HDFS NameNode unavailability; IAM/permission gaps for the Flink task user.

Related errors

• Exception listing files for
• Exception listing files for
• Could not list sub directories, reached maximum depth:
• Exception planning scan for
• Exception planning scan for

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/2f60bdd6315bde7d. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/ListFileSystemFiles.java:122

            "Cannot use prefix listing with FileIO %s which does not support prefix operations.",
            io);

        FileSystemWalker.listDirRecursivelyWithFileIO(
            (SupportsPrefixOperations) io, location, specs, predicate, out::collect);
      } else {
        Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;
        FileSystemWalker.listDirRecursivelyWithHadoop(
            location,
            specs,
            predicate,
            configuration,
            Integer.MAX_VALUE,
            Integer.MAX_VALUE,
            dir -> {},
            out::collect);
      }
    } catch (Exception e) {
      LOG.warn("Exception listing files for {} at {}", location, ctx.timestamp(), e);
      ctx.output(DeleteOrphanFiles.ERROR_STREAM, e);
      errorCounter.inc();
    }
  }

  @Override
  public void close() throws Exception {
    super.close();
    tableLoader.close();
  }
}

View on GitHub (pinned to 86d9c8fc54)