apache/seatunnel · error · IOException

Cannot rename file from

Error message

Cannot rename file from [<oldPath>] to [<newPath>]: both source and target are missing.

What it means

HadoopFileSystemProxy.renameFile renames the sink's in-progress file to its final committed name. When neither the source nor the target path exists on the filesystem, it cannot treat the rename as already-done or as a no-op, so it throws an IOException describing both paths.

Solutions

1. Verify the source temp file exists on the configured filesystem before retrying the commit
2. Ensure the sink filesystem (fs.defaultFS / path scheme) matches the one used when the files were written
3. Restore the missing file from backup or rerun the job from an earlier checkpoint so temp files are regenerated
4. Check storage lifecycle/expiry policies (e.g. S3 lifecycle rules) that may delete temp files

FileSystem fs = FileSystem.get(conf);
if (!fs.exists(oldPath) && !fs.exists(newPath)) {
    throw new IllegalStateException("Rename source and target both missing: " + oldPath + " -> " + newPath);
}

try {
    proxy.renameFile(oldPath, newPath);
} catch (IOException e) {
    if (e.getMessage().contains("both source and target are missing")) {
        // verify filesystem config / restore temp files before retrying commit
    } else {
        throw e;
    }
}

When it happens

Trigger: Commit phase calls renameFile(oldPath, newPath) after the source temp file was deleted (e.g. manually cleaned, expired storage lifecycle rule, or a different cluster/filesystem) and the target was never created — the 'both missing' branch of the existence check.

Common situations: Retrying a commit against a storage where temp files were garbage-collected; pointing sink and recovery state at different HDFS/S3 clusters; manually removing _transaction files between failed and retried jobs; misconfigured checkpoint storage so fileStates reference stale paths.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• CanalJson file does not support this compress type
• check connectivity failed,
• Circular condition chain detected
• Close file output stream
• Close file output stream

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/c461cce4b635f1fb. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/hadoop/HadoopFileSystemProxy.java:153

            @NonNull String oldFilePath,
            @NonNull String newFilePath,
            boolean removeWhenNewFilePathExist)
            throws IOException {
        execute(
                () -> {
                    Path oldPath = new Path(oldFilePath);
                    Path newPath = new Path(newFilePath);

                    if (!fileExist(oldPath.toString())) {
                        if (fileExist(newPath.toString())) {
                            log.info(
                                    "Rename file from [{}] to [{}] already finished in a previous "
                                            + "commit, skip.",
                                    oldPath,
                                    newPath);
                            return Void.class;
                        }
                        throw new IOException(
                                "Cannot rename file from ["
                                        + oldPath
                                        + "] to ["
                                        + newPath
                                        + "]: both source and target are missing.");
                    }

                    if (removeWhenNewFilePathExist) {
                        if (fileExist(newFilePath)) {
                            getFileSystem().delete(newPath, true);
                            log.info("Delete already file: {}", newPath);
                        }
                    }
                    if (!fileExist(newPath.getParent().toString())) {
                        createDir(newPath.getParent().toString());
                    }

                    if (getFileSystem().rename(oldPath, newPath)) {

View on GitHub (pinned to cf67b549a7)