apache/flink · critical · IOException

Unable to recover the job as the expected {} file is not fou

Error message

Unable to recover the job as the expected {} file is not found

What it means

Thrown by AzureBlobFsRecoverableDataOutputStream constructor during recovery: the in-progress temp file is missing, and although a rename temp file (tempFile + '.rename') exists, its length does not equal the recoverable offset recorded at persist() time. Because the surviving file cannot reproduce the persisted state, recovery is declared unrecoverable and the job cannot continue from this in-progress part.

Source

Thrown at flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AzureBlobFsRecoverableDataOutputStream.java:94

    AzureBlobFsRecoverableDataOutputStream(FileSystem fs, HadoopFsRecoverable recoverable)
            throws IOException {
        this.fs = checkNotNull(fs);
        this.targetFile = checkNotNull(recoverable.targetFile());
        this.tempFile = checkNotNull(recoverable.tempFile());
        if (!fs.exists(tempFile)) {
            LOG.error("The temp file is not found {}", tempFile);
            // trying the temp rename file.
            Path renameTempPath = new Path(tempFile.toString() + RENAME);
            if (fs.exists(renameTempPath)) {
                LOG.info(
                        "Found the rename file. Probably a case where the rename did not happen {}",
                        renameTempPath);
                if (fs.getFileStatus(renameTempPath).getLen() == recoverable.offset()) {
                    rename(fs, renameTempPath);
                } else {
                    LOG.error(
                            "Unrecoverable error. As the required {} file is not found", tempFile);
                    throw new IOException(
                            "Unable to recover the job as the expected "
                                    + tempFile
                                    + " file is not found");
                }
            } else {
                LOG.error("Unrecoverable error. As the required {} file is not found", tempFile);
                throw new IOException(
                        "Unable to recover the job as the expected "
                                + tempFile
                                + " file is not found");
            }
        } else {
            long len = fs.getFileStatus(tempFile).getLen();
            LOG.info(
                    "The recoverable offset is {} and the file len is {}",
                    recoverable.offset(),
                    len);
            // Happens when we recover from a previously committed offset. Otherwise this is not

View on GitHub (pinned to 2f3c205e92)

Solutions

Discard the broken in-progress state: restart the job from the last completed checkpoint instead of the in-progress part, or from a fresh savepoint
Verify nothing else (lifecycle policies, other jobs, manual cleanup) deletes files under the checkpoint/temp directory in the ABFS container
Confirm the job is not recovering with an old operator UID mapping so that the same temp file paths are reused by two attempts
Inspect the container and compare the '.rename' file's size against the offset reported in the logs

Defensive patterns

Strategy: try-catch

Try / catch

try {
    writer.recover(recoverable).close();
} catch (java.io.IOException e) {
    // unrecoverable in-progress state: fall back to last completed checkpoint
    triggerRestartFromLastCheckpoint();
}

Prevention

Keep in-progress file state young by checkpointing frequently
Never clean temp/pending prefixes in the output bucket manually or by policy
Monitor for duplicate active job attempts writing the same paths

When it happens

Trigger: TaskManager failure followed by recovery of a streaming file sink (RecoverableWriter on abfs/abfss) where the temp file was deleted or never flushed and the '.rename' sidecar has a different length than the recorded offset — e.g. an older persist snapshot being replayed, or external interference with the storage bucket.

Common situations: Recovering from a stale checkpoint/savepoint after the bucket contents were cleaned; multiple jobs writing to the same temp paths; storage-level consistency anomalies after an Azure outage.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/3a66dc7b05faa297. Report an issue: GitHub.