apache/seatunnel · critical · RuntimeException

File discovery failed

Error message

File discovery failed

What it means

FileCollectReader.discoverNewFiles scans the configured glob for new files inside poll(). If discovery throws and the input's on-error policy is "fail" (i.e. not skip-on-error), it wraps the cause in a RuntimeException("File discovery failed") to propagate the failure to the reader loop.

Solutions

1. Inspect the wrapped cause exception for the underlying filesystem error
2. Set input.on-error = "skip" to log and continue instead of failing
3. Verify the glob path exists and the agent user has read permission
4. Reconnect/remount the storage the glob points to

Example fix

# before
input.on-error = "fail"
# after
input.on-error = "skip"  # tolerate transient discovery errors

Path dir = Paths.get(globDir);
if (!Files.isDirectory(dir)) throw new IllegalStateException("glob directory missing: " + dir);
Files.isDirectory(dir) && Files.isReadable(dir); // probe access before starting the reader

try { reader.poll(); } catch (RuntimeException e) { if ("File discovery failed".equals(e.getMessage())) { log.error("discovery failed", e.getCause()); /* remediate path/permissions or restart */ } else { throw e; } }

When it happens

Trigger: A glob pointing to a deleted or permission-denied directory, filesystem I/O errors during listing, or any exception thrown by the path matcher while poll() is running.

Common situations: Network mounts going offline, removable media unmounted, directories removed while the agent runs, or glob patterns with invalid syntax.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Checking catalog path
• Circular condition chain detected
• Condition for option
• Dropping table exception.
• Error during file discovery

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/f14f8d0d7a02ac99. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-edge-agent/seatunnel-edge-agent-connector/src/main/java/org/apache/seatunnel/edge/agent/connector/file/FileCollectReader.java:274

        try {
            List<Path> newFiles = globResolver.resolveNew();
            for (Path file : newFiles) {
                if (activeCursors.containsKey(file)) {
                    continue;
                }
                String pathStr = file.toAbsolutePath().toString();
                EdgeSourcePosition pos = resolvePosition(pathStr);

                FileTailCursor cursor = openCursor(file, pos);
                activeCursors.put(file, cursor);
                lineCounters.put(file, restoredLineNumber(pos));
                LOG.info("Discovered new file: {}", file);
            }
        } catch (Exception e) {
            if (config.isSkipOnError()) {
                LOG.warn("Error during file discovery", e);
            } else {
                throw new RuntimeException("File discovery failed", e);
            }
        }
    }

    private void closeInactiveCursors(long nowMs) {
        Iterator<Map.Entry<Path, FileTailCursor>> it = activeCursors.entrySet().iterator();
        while (it.hasNext()) {
            Map.Entry<Path, FileTailCursor> entry = it.next();
            Path filePath = entry.getKey();
            FileTailCursor cursor = entry.getValue();
            if (nowMs - cursor.lastActivityMs() > config.getCloseInactiveMs()) {
                LOG.debug("Closing inactive cursor: {}", filePath);
                clearFileState(filePath, "inactive timeout");
                try {
                    cursor.close();
                } catch (Exception ignored) {
                }
                globResolver.forget(filePath);

View on GitHub (pinned to cf67b549a7)