apache/flink · critical · IOException

Unable to create recoverable outputstream as length of file

Error message

Unable to create recoverable outputstream as length of file {} is less than recoverable offset {}

What it means

Thrown by AzureBlobFsRecoverableDataOutputStream during recovery when the temp file exists but its length is LESS than the recoverable offset from the last persist(). That means part of the acknowledged data never made it to storage (truncated/uncommitted blocks), so appending from the offset would produce a corrupt file; the writer refuses to continue.

Source

Thrown at flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AzureBlobFsRecoverableDataOutputStream.java:121

                                + tempFile
                                + " file is not found");
            }
        } else {
            long len = fs.getFileStatus(tempFile).getLen();
            LOG.info(
                    "The recoverable offset is {} and the file len is {}",
                    recoverable.offset(),
                    len);
            // Happens when we recover from a previously committed offset. Otherwise this is not
            // really needed
            if (len > recoverable.offset()) {
                truncate(fs, recoverable);
            } else if (len < recoverable.offset()) {
                LOG.error(
                        "Temp file length {} is less than the expected recoverable offset {}",
                        len,
                        recoverable.offset());
                throw new IOException(
                        "Unable to create recoverable outputstream as length of file "
                                + len
                                + " is less than "
                                + "recoverable offset "
                                + recoverable.offset());
            }
        }
        out = fs.append(tempFile);
        if (out.getPos() == 0) {
            // In ABFS when we try to append we don't account for the initial file size like we do
            // in DFS.
            // So we explicitly store this and when we do a persist call we make use of it.
            // This we have raised a bug in ABFS hadoop driver side. Once fixed this will not be
            // needed. So it should be ok to put this in side the 'if' check.
            initialFileSize = fs.getFileStatus(tempFile).getLen();
        }
        LOG.debug("Created a new OS for appending {}", tempFile);
    }

View on GitHub (pinned to 2f3c205e92)

Solutions

Fall back to the last completed checkpoint and let the sink rewrite the file part
Upgrade flink-azure-fs-hadoop and the Hadoop ABFS client to a version with stable append/flush semantics
Report with the two numbers from the log line ('Temp file length {} is less than the expected recoverable offset {}') — it indicates data actually lost between persist and recovery
Reduce the sink in-progress state age (more frequent checkpoints) so less data is at risk

Defensive patterns

Strategy: try-catch

Try / catch

try {
    writer.recover(recoverable);
} catch (java.io.IOException e) {
    // offset > stored length means data loss: do not append; re-emit from checkpoint
    handleUnrecoverableInProgressState(e);
}

Prevention

Checkpoint often to bound the amount of in-flight uncommitted data
Keep flink-azure-fs-hadoop and Hadoop ABFS client versions aligned and current
Avoid killing TaskManagers with SIGKILL during heavy append phases when possible

When it happens

Trigger: Recovery where Azure Blob did not retain all uncommitted appended data: append-block timeouts, server-side truncation, or a mismatch between the offset recorded at persist() and what ABFS actually committed.

Common situations: ABFS append operations that failed silently before persist; recovering across a storage outage; Hadoop ABFS client version differences in flush/append semantics.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/850eee208856d8c8. Report an issue: GitHub.