apache/seatunnel · warning

Failed to close file discovery resources for plugin

Error message

Failed to close file discovery resources for plugin {}

What it means

A WARN from BaseFileSourceConfig.closeDiscoveryReadStrategy: the temporary ReadStrategy used only for file discovery could not be closed (its close() threw an IOException). Split enumeration already produced file paths; only underlying resources (filesystem handles, HDFS clients) of the discovery strategy may leak.

Solutions

1. Check filesystem stability and credentials at job start; rerun the job once the cluster/storage is healthy
2. Ensure the storage plugin (hadoop-s3, oss, etc.) credentials are valid for the whole enumeration window
3. If it recurs, verify the connector version's ReadStrategy.close() implementation for known cleanup bugs
4. Treat as non-fatal: enumeration results are already complete

// preflight: check storage reachable with job credentials
// hadoop fs -ls hdfs://nn:8020/base/path  (or aws s3 ls s3://bucket/path)

// enumeration already succeeded; only cleanup failed.
// treat as informational; escalate only if job start also throws IOException

When it happens

Trigger: getFilePathsForSplitEnumerator creates a throwaway discovery ReadStrategy, calls it, then closes it in a finally; close() throws IOException because the Hadoop/S3 filesystem was already closed, credentials were revoked, or the remote store returned an I/O error during cleanup.

Common situations: HDFS NameNode restarts or S3 throttling during job startup; Kerberos token expiry between discovery and close; sharing filesystem caches that another component closed concurrently.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• delete transaction directory
• Failed to clean empty transaction parent directory
• CanalJson file does not support this compress type
• Cannot rename file from
• check connectivity failed,

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/86b47b47ed6cf9ec. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseFileSourceConfig.java:125

    public List<String> getFilePathsForSplitEnumerator() {
        if (fileDiscoveryDeferred) {
            ReadStrategy discoveryReadStrategy =
                    ReadStrategyFactory.of(baseFileSourceConfig, getHadoopConfig());
            try {
                return discoverFilePaths(discoveryReadStrategy);
            } finally {
                closeDiscoveryReadStrategy(discoveryReadStrategy);
            }
        }
        return filePaths;
    }

    private void closeDiscoveryReadStrategy(ReadStrategy discoveryReadStrategy) {
        try {
            discoveryReadStrategy.close();
        } catch (IOException e) {
            log.warn("Failed to close file discovery resources for plugin {}", getPluginName(), e);
        }
    }

    private List<String> discoverFilePaths() {
        return discoverFilePaths(readStrategy);
    }

    private List<String> discoverFilePaths(ReadStrategy discoveryReadStrategy) {
        String rootPath = baseFileSourceConfig.get(FileBaseSourceOptions.FILE_PATH);
        long startTime = System.currentTimeMillis();
        try {
            List<String> discoveredFilePaths = discoveryReadStrategy.getFileNamesByPath(rootPath);
            log.info(
                    "File source discovery finished: plugin={}, path={}, files={}, cost={}ms",
                    getPluginName(),
                    safeDiscoveryRootContext,
                    discoveredFilePaths.size(),
                    System.currentTimeMillis() - startTime);

View on GitHub (pinned to cf67b549a7)