Skipping deletion: Log directory

Error message

Skipping deletion: Log directory '{}' either does not exist or is not a valid directory. Please verify the path and ensure the logs are being written correctly.

What it means

TaskLogManagerService.getLogFiles validates that the configured log directory exists and is a directory before listing job logs. If not, it warns and returns an empty array so no deletion is attempted. This means log cleanup silently skips for a bad/missing path, so old logs accumulate elsewhere or the path is misconfigured.

Solutions

Verify the configured log directory path exists on every cluster node and correct the task-log-manager configuration.
Create the directory (mkdir -p) with proper ownership for the SeaTunnel user.
Check if an external cleanup (cron/tmpfiles) removes the dir and exclude it.
Confirm each worker writes logs to the same configured path to avoid node-specific misses.

Example fix

// before: config points to nonexistent dir
log-dir: /data/seatunnel/job-logs   # directory was never created
// after
mkdir -p /data/seatunnel/job-logs && chown seatunnel:seatunnel /data/seatunnel/job-logs
log-dir: /data/seatunnel/job-logs

Defensive patterns

Strategy: validation

Validate before calling

// validate log dir before configuring cleanup
Path dir = Paths.get(logDir);
if (!Files.isDirectory(dir)) {
    throw new IllegalArgumentException("log-dir does not exist or is not a directory: " + logDir);
}

Type guard

boolean isValidLogDir(String p) {
    File d = new File(p);
    return d.exists() && d.isDirectory();
}

Prevention

Verify the configured log directory exists on every cluster node at startup.
Create and own the directory as the SeaTunnel user before enabling log cleanup.
Exclude the log dir from OS tmp cleaners (tmpwatch/systemd-tmpfiles).
Re-check the path after upgrades that relocate logs.

When it happens

Trigger: clean() calls getLogFiles(jobId, path) where 'path' does not exist or is a file, not a directory — typically a wrong log-dire path in telemetry/task-log-manager configuration or a path deleted externally.

Common situations: Typo in log path config; logs moved to another directory after an upgrade; directory wiped by tmp-cleaners (path under /tmp); running cleanup on a node that never writes logs to that path.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/5d95b7b17eacc45f. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/telemetry/log/TaskLogManagerService.java:73

        if (path == null || jobId <= 0) {
            return;
        }
        String[] logFiles = getLogFiles(jobId, path);
        for (String logFile : logFiles) {
            String logPath = path + "/" + logFile;
            log.info("Cleaning logs for jobId: {} , path : {}", jobId, logPath);
            try {
                Files.delete(Paths.get(logPath));
            } catch (IOException e) {
                log.warn("Failed to delete log file: {}", logPath, e);
            }
        }
    }

    private String[] getLogFiles(long jobId, String path) {
        File logDir = new File(path);
        if (!logDir.exists() || !logDir.isDirectory()) {
            log.warn(
                    "Skipping deletion: Log directory '{}' either does not exist or is not a valid directory. Please verify the path and ensure the logs are being written correctly.",
                    path);
            return new String[0];
        }

        return logDir.list((dir, name) -> name.contains(String.valueOf(jobId)));
    }
}

View on GitHub (pinned to cf67b549a7)