{"record":{"id":"2f60bdd6315bde7d","repo":"apache/iceberg","slug":"exception-listing-files-for-at-2f60bd","errorCode":null,"errorMessage":"Exception listing files for {} at {}","messagePattern":"Exception listing files for (.+?) at (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/ListFileSystemFiles.java","lineNumber":122,"sourceCode":"            \"Cannot use prefix listing with FileIO %s which does not support prefix operations.\",\n            io);\n\n        FileSystemWalker.listDirRecursivelyWithFileIO(\n            (SupportsPrefixOperations) io, location, specs, predicate, out::collect);\n      } else {\n        Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;\n        FileSystemWalker.listDirRecursivelyWithHadoop(\n            location,\n            specs,\n            predicate,\n            configuration,\n            Integer.MAX_VALUE,\n            Integer.MAX_VALUE,\n            dir -> {},\n            out::collect);\n      }\n    } catch (Exception e) {\n      LOG.warn(\"Exception listing files for {} at {}\", location, ctx.timestamp(), e);\n      ctx.output(DeleteOrphanFiles.ERROR_STREAM, e);\n      errorCounter.inc();\n    }\n  }\n\n  @Override\n  public void close() throws Exception {\n    super.close();\n    tableLoader.close();\n  }\n}\n","sourceCodeStart":104,"sourceCodeEnd":134,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/ListFileSystemFiles.java#L104-L134","documentation":"ListFileSystemFiles.processElement lists filesystem locations for the DeleteOrphanFiles maintenance flow and emits each found file. When the listing operation throws, the exception is logged as a warning with the location and timestamp, emitted to the DeleteOrphanFiles ERROR_STREAM side output, and counted in errorCounter — the job keeps running so other locations can still be scanned.","triggerScenarios":"Raised in processElement when the FileSystem listFiles/recursive walk over `location` throws — e.g. FileNotFoundException for a missing directory, permission errors, or object-store listing failures (throttling, invalid prefix).","commonSituations":"Configured location does not exist yet (first run before any writes); S3/GCS list throttling on huge prefixes; wrong path/scheme configuration (s3a vs s3); HDFS NameNode unavailability; IAM/permission gaps for the Flink task user.","solutions":["Consume the DeleteOrphanFiles.ERROR_STREAM side output in your job to inspect the actual exception","Verify the configured location exists and the scheme/authority is correct (s3a:// vs s3://)","Fix filesystem permissions/IAM policies for the TaskManager's identity","If listing huge prefixes, tune listing parallelism or reduce scope; retry on next trigger since errorCounter tracks failures"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Before running the action, verify the location exists\nFileSystem fs = new Path(location).getFileSystem(conf);\nPreconditions.checkArgument(fs.exists(new Path(location)), \"Location missing: \" + location);","typeGuard":null,"tryCatchPattern":"// Consume the side output where the exception is forwarded\nSingleOutputStreamOperator<?> result = ...;\nresult.getSideOutput(DeleteOrphanFiles.ERROR_STREAM)\n      .executeAndCollect().forEachRemaining(e -> log.error(\"orphan-scan listing failed\", e));","preventionTips":["Use a consistent scheme (s3a://) everywhere in table locations and action config","Verify the location exists before the first maintenance run","Check IAM/filesystem permissions for the Flink task user","Consume the ERROR_STREAM side output in every maintenance job for visibility"],"tags":["flink","filesystem","listing","orphan-files"],"backgroundTag":"file-listing-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}