apache/flink · error · IOException

Unable to recover. Rename operation failed

Error message

Unable to recover. Rename operation failed

What it means

Thrown by AzureBlobFsRecoverableDataOutputStream.rename() during recovery when Hadoop's FileSystem.rename(renameTempPath, tempFile) returns false (ABFS reports failure without throwing). The '.rename' sidecar could not be moved back over the temp file, so the recoverable stream cannot be reconstructed.

Source

Thrown at flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AzureBlobFsRecoverableDataOutputStream.java:209

        } catch (IOException e) {
            LOG.error("Unable to recover. Error while deleting the temp file {}", tempFile);
            // unable to recover.
            throw e;
        }
        rename(fs, renameTempPath);
    }

    private void rename(FileSystem fs, Path renameTempPath) throws IOException {
        LOG.info("Renaming the temp rename file {} back to tempFile {}", renameTempPath, tempFile);
        try {
            // Rename by default will overwrite if dest is already found.
            boolean result = fs.rename(renameTempPath, tempFile);
            if (!result) {
                LOG.error(
                        "Unable to recover. Rename operation failed {} to {}",
                        renameTempPath,
                        tempFile);
                throw new IOException("Unable to recover. Rename operation failed");
            } else {
                LOG.info("Rename was successful");
            }
        } catch (IOException e) {
            LOG.error(
                    "Unable to recover. Renaming of tempFile did not happen after truncating {} to {}",
                    renameTempPath,
                    tempFile);
            throw e;
        }
    }

    @Override
    public void sync() throws IOException {
        out.hsync();
    }

    @Override

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Retry job recovery — transient ABFS rename failures usually clear
2. Grant the credential used by the sink write AND delete rights on the container (rename in ABFS needs delete permission)
3. Ensure only one active job attempt recovers the same checkpoint (no zombie JobManager)
4. Check for blob leases on tempFile and release them

try {
    writer.recover(recoverable).close();
} catch (java.io.IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("Rename operation failed")) {
        retryWithBackoff(); // transient ABFS rename failures often clear
    } else { throw e; }
}

When it happens

Trigger: Recovery path where the '.rename' file exists with matching length but the rename call fails: insufficient permissions (SAS token without write/delete), a lease on the destination blob, or transient ABFS errors.

Common situations: Sink configured with a restricted SAS credential or read-only storage key; concurrent recovery attempts from two JobManagers fighting over the same files; transient Azure control-plane failures.

Related errors

• Committing file by rename failed: {} to {}
• Unable to recover the job as the expected {} file is not fou
• Unable to create recoverable outputstream as length of file
• Cannot clean commit: Staging file does not exist.
• The src file {} with length {} does not match the expected l

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/6d0ba17ae433a0f0. Report an issue: GitHub.