{"record":{"id":"fce85fbb4e41b443","repo":"apache/seatunnel","slug":"failed-during-target-bulk-listing-for-path-s","errorCode":null,"errorMessage":"Failed during target_bulk_listing for path=%s","messagePattern":"Failed during target_bulk_listing for path=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/UpdateFileMetadataLoader.java","lineNumber":104,"sourceCode":"                for (Map.Entry<Path, List<Request>> group : bulkGroups) {\n                    long started = System.nanoTime();\n                    Map<String, Request> wanted = new HashMap<>();\n                    for (Request request : group.getValue()) {\n                        wanted.put(new Path(request.targetPath).getName(), request);\n                    }\n                    try {\n                        session.list(\n                                group.getKey(),\n                                status -> {\n                                    Request request = wanted.get(status.getPath().getName());\n                                    if (request != null) {\n                                        ordered[request.order] = status;\n                                    }\n                                });\n                    } catch (FileNotFoundException ignored) {\n                        // A missing target directory means every source file in the group is new.\n                    } catch (IOException e) {\n                        throw new IOException(\n                                \"Failed during target_bulk_listing for path=\"\n                                        + mask(group.getKey()),\n                                e);\n                    } finally {\n                        bulkNanos += System.nanoTime() - started;\n                    }\n                }\n            }\n        }\n\n        PointResult pointResult = loadPoints(pointRequests, target, ordered, parallelism);\n        return new Result(\n                Arrays.asList(ordered),\n                bulkGroups.size(),\n                pointRequests.size(),\n                pointResult.peakConcurrency,\n                pointResult.peakInFlight,\n                bulkNanos,","sourceCodeStart":86,"sourceCodeEnd":122,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/UpdateFileMetadataLoader.java#L86-L122","documentation":"UpdateFileMetadataLoader.load performs a bulk status lookup on target (destination) directories to detect which files already exist; an IOException other than FileNotFoundException is rethrown as 'Failed during target_bulk_listing for path=...'. The path is masked for logging. It means the bulk metadata listing against the target filesystem failed.","triggerScenarios":"Listing a target directory whose storage backend throws IOException: HDFS NameNode unreachable, S3 throttling (HTTP 5xx), token/credential expiry, or transient network errors. (Missing target directory is explicitly tolerated as 'all files new'.)","commonSituations":"Incremental/append sync jobs where the destination storage is degraded; throttled object storage under high listing concurrency; expired Kerberos/STS credentials mid-job.","solutions":["Check target storage health/connectivity from worker nodes and retry the job","Inspect the cause chain for the storage-specific error (throttling, auth, timeout) and address it (backoff, IAM/Kerberos credentials)","Lower listing concurrency or add retry/backoff if object storage rate-limits","Verify target path permissions are readable"],"exampleFix":"// before\n// job failing on S3 503 during bulk listing\n// after\n// enable S3 requester retry/backoff or reduce parallelism, then rerun the sync job","handlingStrategy":"retry","validationCode":"// probe target listing before starting the job\ntry {\n    fs.listStatus(targetPath);\n} catch (IOException e) {\n    throw new IllegalStateException(\"Target filesystem unhealthy before job: \" + targetPath, e);\n}","typeGuard":null,"tryCatchPattern":"try {\n    loader.load(requests);\n} catch (IOException e) {\n    if (e.getMessage().startsWith(\"Failed during target_bulk_listing\")) {\n        // retry with backoff; check cause for throttling/auth\n        retryWithBackoff(e, 3);\n    } else throw e;\n}","preventionTips":["Validate target filesystem access and credentials before job submission","Configure client retry/backoff for object storage (S3 503 handling)","Renew long-lived tokens (Kerberos/STS) for jobs longer than token lifetime"],"tags":["io","hdfs","listing","metadata"],"backgroundTag":"file-read-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}