{"record":{"id":"3a66dc7b05faa297","repo":"apache/flink","slug":"unable-to-recover-the-job-as-the-expected-file","errorCode":null,"errorMessage":"Unable to recover the job as the expected {} file is not found","messagePattern":"Unable to recover the job as the expected (.+?) file is not found","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AzureBlobFsRecoverableDataOutputStream.java","lineNumber":94,"sourceCode":"    AzureBlobFsRecoverableDataOutputStream(FileSystem fs, HadoopFsRecoverable recoverable)\n            throws IOException {\n        this.fs = checkNotNull(fs);\n        this.targetFile = checkNotNull(recoverable.targetFile());\n        this.tempFile = checkNotNull(recoverable.tempFile());\n        if (!fs.exists(tempFile)) {\n            LOG.error(\"The temp file is not found {}\", tempFile);\n            // trying the temp rename file.\n            Path renameTempPath = new Path(tempFile.toString() + RENAME);\n            if (fs.exists(renameTempPath)) {\n                LOG.info(\n                        \"Found the rename file. Probably a case where the rename did not happen {}\",\n                        renameTempPath);\n                if (fs.getFileStatus(renameTempPath).getLen() == recoverable.offset()) {\n                    rename(fs, renameTempPath);\n                } else {\n                    LOG.error(\n                            \"Unrecoverable error. As the required {} file is not found\", tempFile);\n                    throw new IOException(\n                            \"Unable to recover the job as the expected \"\n                                    + tempFile\n                                    + \" file is not found\");\n                }\n            } else {\n                LOG.error(\"Unrecoverable error. As the required {} file is not found\", tempFile);\n                throw new IOException(\n                        \"Unable to recover the job as the expected \"\n                                + tempFile\n                                + \" file is not found\");\n            }\n        } else {\n            long len = fs.getFileStatus(tempFile).getLen();\n            LOG.info(\n                    \"The recoverable offset is {} and the file len is {}\",\n                    recoverable.offset(),\n                    len);\n            // Happens when we recover from a previously committed offset. Otherwise this is not","sourceCodeStart":76,"sourceCodeEnd":112,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AzureBlobFsRecoverableDataOutputStream.java#L76-L112","documentation":"Thrown by AzureBlobFsRecoverableDataOutputStream constructor during recovery: the in-progress temp file is missing, and although a rename temp file (tempFile + '.rename') exists, its length does not equal the recoverable offset recorded at persist() time. Because the surviving file cannot reproduce the persisted state, recovery is declared unrecoverable and the job cannot continue from this in-progress part.","triggerScenarios":"TaskManager failure followed by recovery of a streaming file sink (RecoverableWriter on abfs/abfss) where the temp file was deleted or never flushed and the '.rename' sidecar has a different length than the recorded offset — e.g. an older persist snapshot being replayed, or external interference with the storage bucket.","commonSituations":"Recovering from a stale checkpoint/savepoint after the bucket contents were cleaned; multiple jobs writing to the same temp paths; storage-level consistency anomalies after an Azure outage.","solutions":["Discard the broken in-progress state: restart the job from the last completed checkpoint instead of the in-progress part, or from a fresh savepoint","Verify nothing else (lifecycle policies, other jobs, manual cleanup) deletes files under the checkpoint/temp directory in the ABFS container","Confirm the job is not recovering with an old operator UID mapping so that the same temp file paths are reused by two attempts","Inspect the container and compare the '.rename' file's size against the offset reported in the logs"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    writer.recover(recoverable).close();\n} catch (java.io.IOException e) {\n    // unrecoverable in-progress state: fall back to last completed checkpoint\n    triggerRestartFromLastCheckpoint();\n}","preventionTips":["Keep in-progress file state young by checkpointing frequently","Never clean temp/pending prefixes in the output bucket manually or by policy","Monitor for duplicate active job attempts writing the same paths"],"tags":["azure","abfs","recovery","file-sink","io"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}