What causes "Error during file discovery" in apache/seatunnel?

Check the logged stack trace to see which path failed listing and why.

Error during file discovery

Error message

Error during file discovery

What it means

FileCollectReader.discoverNewFiles wraps the directory-scan loop in a try/catch. When skipOnError is enabled in the agent file-collect config, any exception during discovery (unreadable directory, listing IO error) is only logged as a warning 'Error during file discovery'; otherwise it escalates to 'File discovery failed'.

Source

Thrown at seatunnel-edge-agent/seatunnel-edge-agent-connector/src/main/java/org/apache/seatunnel/edge/agent/connector/file/FileCollectReader.java:272

        }
        lastDiscoveryMs = now;
        try {
            List<Path> newFiles = globResolver.resolveNew();
            for (Path file : newFiles) {
                if (activeCursors.containsKey(file)) {
                    continue;
                }
                String pathStr = file.toAbsolutePath().toString();
                EdgeSourcePosition pos = resolvePosition(pathStr);

                FileTailCursor cursor = openCursor(file, pos);
                activeCursors.put(file, cursor);
                lineCounters.put(file, restoredLineNumber(pos));
                LOG.info("Discovered new file: {}", file);
            }
        } catch (Exception e) {
            if (config.isSkipOnError()) {
                LOG.warn("Error during file discovery", e);
            } else {
                throw new RuntimeException("File discovery failed", e);
            }
        }
    }

    private void closeInactiveCursors(long nowMs) {
        Iterator<Map.Entry<Path, FileTailCursor>> it = activeCursors.entrySet().iterator();
        while (it.hasNext()) {
            Map.Entry<Path, FileTailCursor> entry = it.next();
            Path filePath = entry.getKey();
            FileTailCursor cursor = entry.getValue();
            if (nowMs - cursor.lastActivityMs() > config.getCloseInactiveMs()) {
                LOG.debug("Closing inactive cursor: {}", filePath);
                clearFileState(filePath, "inactive timeout");
                try {
                    cursor.close();
                } catch (Exception ignored) {

View on GitHub (pinned to cf67b549a7)

Solutions

Check the logged stack trace to see which path failed listing and why.
Restore read permissions on the watched directory for the agent's OS user.
Verify the watched directory still exists (or recreate it); some watchers auto-recreate, directory scans do not.
If data loss from skipping is unacceptable, set skipOnError=false so the reader fails fast and can be restarted from committed offsets.

Example fix

// before
FileCollect {
  path = "/var/log/app"
  skip_on_error = true
}
// after: fail fast to surface real discovery problems
FileCollect {
  path = "/var/log/app"
  skip_on_error = false
}

Defensive patterns

Strategy: retry

Validate before calling

// Pre-flight: ensure the watch directory is readable
File dir = config.getWatchPath().toFile();
if (!dir.isDirectory() || !dir.canRead()) {
    throw new IllegalStateException("Watch path missing or unreadable: " + dir);
}

Try / catch

// If skipOnError=false
try {
    reader.poll();
} catch (RuntimeException e) {
    if (e.getMessage() != null && e.getMessage().contains("File discovery failed")) {
        // alert, backoff, and restart the reader from last committed offsets
    }
}

Prevention

Verify directory permissions for the agent user after every deploy.
Avoid deleting watched directories during collection; rotate files in place.
Set skipOnError=false in environments where losing discovery cycles is unacceptable.
Monitor the agent logs for repeated 'Error during file discovery' warnings.

When it happens

Trigger: poll() -> discoverNewFiles() hits an IO error listing the watch directory: directory deleted or permissions changed mid-run, filesystem errors, or a path that stops being listable while the reader is active.

Common situations: Log files being rotated/archived by another process while collecting, the watched directory removed on NFS mounts, or insufficient read permissions after a redeploy running as a different user.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/a461a83fac8b8488. Report an issue: GitHub.