{"record":{"id":"e1679e2acf056fdb","repo":"apache/iceberg","slug":"exception-listing-files-for-at","errorCode":null,"errorMessage":"Exception listing files for {} at {}","messagePattern":"Exception listing files for (.+?) at (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/ListFileSystemFiles.java","lineNumber":122,"sourceCode":"            \"Cannot use prefix listing with FileIO %s which does not support prefix operations.\",\n            io);\n\n        FileSystemWalker.listDirRecursivelyWithFileIO(\n            (SupportsPrefixOperations) io, location, specs, predicate, out::collect);\n      } else {\n        Predicate<FileStatus> predicate = file -> file.getModificationTime() < olderThanTimestamp;\n        FileSystemWalker.listDirRecursivelyWithHadoop(\n            location,\n            specs,\n            predicate,\n            configuration,\n            Integer.MAX_VALUE,\n            Integer.MAX_VALUE,\n            dir -> {},\n            out::collect);\n      }\n    } catch (Exception e) {\n      LOG.warn(\"Exception listing files for {} at {}\", location, ctx.timestamp(), e);\n      ctx.output(DeleteOrphanFiles.ERROR_STREAM, e);\n      errorCounter.inc();\n    }\n  }\n\n  @Override\n  public void close() throws Exception {\n    super.close();\n    tableLoader.close();\n  }\n}\n","sourceCodeStart":104,"sourceCodeEnd":134,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/ListFileSystemFiles.java#L104-L134","documentation":"A WARN log emitted when the ListFileSystemFiles operator fails while listing files at a location during orphan-file detection. The exception is routed to the DeleteOrphanFiles ERROR_STREAM side output and the errorCounter is incremented, so downstream committers can fail the task; the listing attempt for that directory is abandoned.","triggerScenarios":"processElement() calls Hadoop/FileSystem listStatus (recursive listing) on the table location and any Exception occurs — directory does not exist, permission denied, or filesystem I/O error. Emitted with the location and processing timestamp.","commonSituations":"Table location path typo or removed/moved directory; missing HDFS/S3 list permissions for the Flink job's user; S3 throttling on deep prefixes; filesystem outage or network partition.","solutions":["Verify the table/root location exists and is correct; recreate or fix the path if it was moved or deleted.","Grant the Flink job's principal LIST permission on the directory (HDFS ACL, S3/IAM s3:ListBucket).","Check filesystem availability and retry; the operator emits the error to the side output so the maintenance task can be safely rerun.","Reduce listing pressure (fewer concurrent workers, shallower prefixes) if throttling is the cause."],"exampleFix":"// before: assume directory always listable\nFiles.listRecursive(location, ...);\n// after: precheck existence before running the task\nFileSystem fs = new Path(location).getFileSystem(conf);\nPreconditions.checkArgument(fs.exists(new Path(location)), \"Location %s missing\", location);","handlingStrategy":"validation","validationCode":"org.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.FileSystem.get(conf);\norg.apache.hadoop.fs.Path p = new org.apache.hadoop.fs.Path(location);\nif (!fs.exists(p) || !fs.getFileStatus(p).isDirectory()) {\n  throw new IllegalArgumentException(\"Cannot list: location missing or not a directory: \" + location);\n}","typeGuard":"boolean listable(FileSystem fs, Path dir) throws IOException {\n  return fs.exists(dir) && fs.getFileStatus(dir).isDirectory();\n}","tryCatchPattern":"try {\n  listRecursive(location, out);\n} catch (Exception e) {\n  ctx.output(DeleteOrphanFiles.ERROR_STREAM, e);\n  errorCounter.inc();\n}","preventionTips":["Verify the table location and filesystem config (region, defaultFS) before launching the task.","Grant LIST permission to the Flink job's user/role on all table prefixes.","Watch for S3 503/SlowDown on deep prefixes; reduce concurrent list workers if needed.","Consume the ERROR_STREAM side output so listing failures surface as task failures rather than silent skips."],"tags":["flink","file-listing","filesystem","orphan-files","permission-denied"],"backgroundTag":"file-read-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}