apache/seatunnel · error · IOException

Failed during source_listing setup/scan for protocol=%s, pat

Error message

Failed during source_listing setup/scan for protocol=%s, path=%s

What it means

AbstractReadStrategy.getFileNamesByPath wraps any IOException from listing/setting up the Hadoop filesystem scan into an IOException naming the URI scheme and the (userinfo-masked) path. It indicates the file discovery phase itself failed — network, permissions, missing filesystem implementation, or bad URI — not a problem with individual files' content.

Source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/AbstractReadStrategy.java:183

    public List<String> getFileNamesByPath(String path) throws IOException {
        List<FileStatus> candidates = new ArrayList<>();
        FileDiscoveryScanner.ScanStats scanStats;
        try (FileStatusListingSession session =
                hadoopFileSystemProxy.openFileStatusListingSession()) {
            scanStats =
                    FileDiscoveryScanner.scan(
                            new Path(path),
                            recursiveFileScan,
                            session,
                            this::isSourceCandidate,
                            candidates::add);
        } catch (IOException e) {
            Path sourcePath = new Path(path);
            String protocol =
                    sourcePath.toUri().getScheme() == null
                            ? hadoopFileSystemProxy.getScheme()
                            : sourcePath.toUri().getScheme();
            throw new IOException(
                    "Failed during source_listing setup/scan for protocol="
                            + protocol
                            + ", path="
                            + maskUriUserInfo(path),
                    e);
        }

        List<FileInfo> fileInfoList = new ArrayList<>(candidates.size());
        UpdateFileMetadataLoader.Result updateResult = null;
        long skipped = 0;
        if (enableUpdateSync) {
            if (targetHadoopFileSystemProxy == null) {
                initTargetHadoopFileSystemProxy();
            }
            List<UpdateFileMetadataLoader.Request> requests = new ArrayList<>(candidates.size());
            for (int i = 0; i < candidates.size(); i++) {
                String sourceFilePath = candidates.get(i).getPath().toString();
                String relativePath = resolveRelativePath(sourceRootPath, sourceFilePath);

View on GitHub (pinned to cf67b549a7)

Solutions

Check the protocol in the message and confirm the matching Hadoop filesystem dependency/config is present.
Test connectivity and credentials: `hdfs dfs -ls` or an s3 client list with the same settings as the job.
Verify the path URI is well-formed and the scheme matches an available filesystem.
Inspect the wrapped cause (the `e` chained in the IOException) for the exact storage-level error.

Example fix

// before
path = "s3a://bucket/data"  // no s3a filesystem jars on classpath
// after
# add seatunnel-hadoop3-3.1.4-uber jar / hadoop-aws to plugin dir, then
path = "s3a://bucket/data"

Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-flight listing with the same Hadoop conf the job uses
Path p = new Path(path);
FileSystem fs = p.getFileSystem(conf);
fs.listStatus(p); // throws early with a clearer stack

Try / catch

try {
    // source read
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("Failed during source_listing setup/scan")) {
        log.error("File discovery failed; check protocol jars, connectivity and permissions for path", e.getCause());
    }
    throw e;
}

Prevention

Ship the correct Hadoop filesystem uber jars (hdfs/s3a/oss) in the plugin dir.
Pre-flight list the path with the same credentials before job launch.
Mask-free test: resolve scheme explicitly in the URI.

When it happens

Trigger: An IOException is raised while creating the Hadoop Path/FileSystem or listing statuses in getFileNamesByPath (e.g. unknown scheme, connection failure to HDFS/S3, permission denied); the strategy rethrows with protocol= and path= context (AbstractReadStrategy.java:183).

Common situations: Missing s3a/oss filesystem jars on the classpath (UnknownHostException/NoClassDefFoundError chain); HDFS NameNode unreachable; IAM/kerberos credentials missing; path URI malformed.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/eae4ac6ee10fbee4. Report an issue: GitHub.