{"record":{"id":"a461a83fac8b8488","repo":"apache/seatunnel","slug":"error-during-file-discovery","errorCode":null,"errorMessage":"Error during file discovery","messagePattern":"Error during file discovery","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"seatunnel-edge-agent/seatunnel-edge-agent-connector/src/main/java/org/apache/seatunnel/edge/agent/connector/file/FileCollectReader.java","lineNumber":272,"sourceCode":"        }\n        lastDiscoveryMs = now;\n        try {\n            List<Path> newFiles = globResolver.resolveNew();\n            for (Path file : newFiles) {\n                if (activeCursors.containsKey(file)) {\n                    continue;\n                }\n                String pathStr = file.toAbsolutePath().toString();\n                EdgeSourcePosition pos = resolvePosition(pathStr);\n\n                FileTailCursor cursor = openCursor(file, pos);\n                activeCursors.put(file, cursor);\n                lineCounters.put(file, restoredLineNumber(pos));\n                LOG.info(\"Discovered new file: {}\", file);\n            }\n        } catch (Exception e) {\n            if (config.isSkipOnError()) {\n                LOG.warn(\"Error during file discovery\", e);\n            } else {\n                throw new RuntimeException(\"File discovery failed\", e);\n            }\n        }\n    }\n\n    private void closeInactiveCursors(long nowMs) {\n        Iterator<Map.Entry<Path, FileTailCursor>> it = activeCursors.entrySet().iterator();\n        while (it.hasNext()) {\n            Map.Entry<Path, FileTailCursor> entry = it.next();\n            Path filePath = entry.getKey();\n            FileTailCursor cursor = entry.getValue();\n            if (nowMs - cursor.lastActivityMs() > config.getCloseInactiveMs()) {\n                LOG.debug(\"Closing inactive cursor: {}\", filePath);\n                clearFileState(filePath, \"inactive timeout\");\n                try {\n                    cursor.close();\n                } catch (Exception ignored) {","sourceCodeStart":254,"sourceCodeEnd":290,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-edge-agent/seatunnel-edge-agent-connector/src/main/java/org/apache/seatunnel/edge/agent/connector/file/FileCollectReader.java#L254-L290","documentation":"FileCollectReader.discoverNewFiles wraps the directory-scan loop in a try/catch. When skipOnError is enabled in the agent file-collect config, any exception during discovery (unreadable directory, listing IO error) is only logged as a warning 'Error during file discovery'; otherwise it escalates to 'File discovery failed'.","triggerScenarios":"poll() -> discoverNewFiles() hits an IO error listing the watch directory: directory deleted or permissions changed mid-run, filesystem errors, or a path that stops being listable while the reader is active.","commonSituations":"Log files being rotated/archived by another process while collecting, the watched directory removed on NFS mounts, or insufficient read permissions after a redeploy running as a different user.","solutions":["Check the logged stack trace to see which path failed listing and why.","Restore read permissions on the watched directory for the agent's OS user.","Verify the watched directory still exists (or recreate it); some watchers auto-recreate, directory scans do not.","If data loss from skipping is unacceptable, set skipOnError=false so the reader fails fast and can be restarted from committed offsets."],"exampleFix":"// before\nFileCollect {\n  path = \"/var/log/app\"\n  skip_on_error = true\n}\n// after: fail fast to surface real discovery problems\nFileCollect {\n  path = \"/var/log/app\"\n  skip_on_error = false\n}","handlingStrategy":"retry","validationCode":"// Pre-flight: ensure the watch directory is readable\nFile dir = config.getWatchPath().toFile();\nif (!dir.isDirectory() || !dir.canRead()) {\n    throw new IllegalStateException(\"Watch path missing or unreadable: \" + dir);\n}","typeGuard":null,"tryCatchPattern":"// If skipOnError=false\ntry {\n    reader.poll();\n} catch (RuntimeException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"File discovery failed\")) {\n        // alert, backoff, and restart the reader from last committed offsets\n    }\n}","preventionTips":["Verify directory permissions for the agent user after every deploy.","Avoid deleting watched directories during collection; rotate files in place.","Set skipOnError=false in environments where losing discovery cycles is unacceptable.","Monitor the agent logs for repeated 'Error during file discovery' warnings."],"tags":["file","io","filesystem","skip-on-error"],"backgroundTag":"file-read-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}