{"record":{"id":"43c80c8507ec40a1","repo":"apache/flink","slug":"incomplete-tail-file-does-not-exist","errorCode":null,"errorMessage":"Incomplete-tail file does not exist: {}","messagePattern":"Incomplete-tail file does not exist: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3RecoverableFsDataOutputStream.java","lineNumber":121,"sourceCode":"        this.uploadId = uploadId;\n        this.localTmpDir = localTmpDir;\n        this.minPartSize = minPartSize;\n        this.completedParts = new ArrayList<>(existingParts);\n        this.numBytesInParts = numBytesInParts;\n        this.nextPartNumber = existingParts.size() + 1;\n        this.currentPartSize = 0;\n        this.closed = false;\n\n        if (incompleteTailFile != null) {\n            resumeFromIncompleteTail(incompleteTailFile);\n        } else {\n            createNewTempFile();\n        }\n    }\n\n    private void resumeFromIncompleteTail(File tailFile) throws IOException {\n        if (!tailFile.exists()) {\n            throw new IOException(\"Incomplete-tail file does not exist: \" + tailFile);\n        }\n        currentTempFile = tailFile;\n        currentPartSize = tailFile.length();\n        // Append mode so subsequent writes land after the recovered bytes.\n        currentFileStream = new FileOutputStream(currentTempFile, true);\n        currentOutputStream = new BufferedOutputStream(currentFileStream, BUFFER_SIZE);\n    }\n\n    private void createNewTempFile() throws IOException {\n        File tmpDir = new File(localTmpDir);\n        Files.createDirectories(tmpDir.toPath());\n\n        currentTempFile = new File(tmpDir, \"s3-part-\" + UUID.randomUUID());\n        currentFileStream = new FileOutputStream(currentTempFile);\n        currentOutputStream = new BufferedOutputStream(currentFileStream, BUFFER_SIZE);\n        currentPartSize = 0;\n    }\n","sourceCodeStart":103,"sourceCodeEnd":139,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/writer/NativeS3RecoverableFsDataOutputStream.java#L103-L139","documentation":"Thrown by NativeS3RecoverableFsDataOutputStream.resumeFromIncompleteTail when the local file passed to resume the stream does not exist on disk. The constructor expects the previously downloaded tail file to be present; its absence means local state was lost between download and stream construction.","triggerScenarios":"Constructing the recoverable output stream with a non-null incompleteTailFile (typically produced by NativeS3RecoverableWriter.downloadIncompleteTail) whose path no longer resolves to an existing file — e.g. localTmpDir was wiped, the file was deleted by another process, or the path was built for a different node after a TaskManager failover.","commonSituations":"TaskManager restart where local tmp dirs (/tmp) are cleaned between attempts; localTmpDir on ephemeral container storage; recovery happening on a different physical machine than where the tail was downloaded; antivirus/cleanup jobs deleting files in tmp; disk-full followed by cleanup scripts.","solutions":["Recover on the same TaskManager/local directory that holds the tail file, or re-download the incomplete tail via NativeS3RecoverableWriter.recover() which re-fetches from S3 instead of reusing a stale local path.","Configure localTmpDir on durable storage that survives task restarts (not OS-managed /tmp with tmpfs/cleanup).","Check that no external process (systemd-tmpfiles, container reaper) purges the tmp dir during job lifetime.","If the local file is unrecoverable, restart from the checkpoint using the standard recover() path rather than manually constructing the stream."],"exampleFix":"// before\nnew NativeS3RecoverableFsDataOutputStream(..., tailFile); // file may be gone\n\n// after\nif (!tailFile.exists()) {\n    // re-download from S3 via the writer instead of assuming the local copy\n    tailFile = writerRedownloadPath; // NativeS3RecoverableWriter.recover(recoverable)\n}","handlingStrategy":"validation","validationCode":"if (incompleteTailFile != null && !incompleteTailFile.exists()) {\n    // re-download via NativeS3RecoverableWriter.recover(recoverable) instead of\n    // constructing the stream with the stale local path\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Put localTmpDir on storage that survives task restarts.","Always prefer writer.recover() over manually wiring tail files.","Never assume the tail file survives a TaskManager failover to another node."],"tags":["s3","recovery","local-state","filesystem","io"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}