apache/flink · error · IOException

The src file {} with length {} does not match the expected l

Error message

The src file {} with length {} does not match the expected length {}

What it means

Thrown by the AzureBlobFsRecoverableDataOutputStream Committer during commit() when the staging file exists but its length differs from the expected length recorded in the recoverable (the offset at persist time). A length mismatch means the staging file was modified (appended to or truncated) after the recoverable was created, so renaming it to the target would commit wrong data.

Source

Thrown at flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AzureBlobFsRecoverableDataOutputStream.java:275

            try {
                srcStatus = fs.getFileStatus(src);
            } catch (FileNotFoundException fnfe) {
                // srcStatus will be null
            } catch (IOException e) {
                throw new IOException("Cannot clean commit: Staging file does not exist.");
            }
            if (srcStatus != null) {
                LOG.debug(
                        "The srcStatus is {} and exp length is {}",
                        srcStatus.getLen(),
                        expectedLength);
                if (srcStatus.getLen() != expectedLength) {
                    LOG.error(
                            "The src file {} with length {} does not match the expected length {}",
                            src,
                            srcStatus.getLen(),
                            expectedLength);
                    throw new IOException(
                            "The src file "
                                    + src
                                    + " with length "
                                    + srcStatus.getLen()
                                    + " "
                                    + "does not match the expected length "
                                    + expectedLength);
                }
                try {
                    fs.rename(src, dest);
                } catch (IOException e) {
                    throw new IOException(
                            "Committing file by rename failed: " + src + " to " + dest, e);
                }
            } else if (!fs.exists(dest)) {
                // neither exists - that can be a sign of
                //   - (1) a serious problem (file system loss of data)
                //   - (2) a recovery of a savepoint that is some time old and the users

View on GitHub (pinned to 2f3c205e92)

Solutions

Ensure exactly one active attempt per operator instance (proper failure detection, no duplicate JobManagers)
Give each sink/attempt unique output or temp-dir prefixes to avoid shared temp files
Restart cleanly from the last consistent checkpoint so pending files match their recoverables
Compare the three values in the log line (src path, actual length, expected length) to identify which attempt wrote the extra/missing bytes

Defensive patterns

Strategy: validation

Validate before calling

// before committing, verify staging length matches the recoverable offset
org.apache.hadoop.fs.FileStatus st = fs.getFileStatus(recoverable.tempFile());
if (st.getLen() != recoverable.offset()) {
    // another attempt modified the file: do not commit, alert
}

Try / catch

try {
    committer.commit();
} catch (java.io.IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("does not match the expected length")) {
        alertOnConcurrentModification();
    } else { throw e; }
}

Prevention

Ensure unique temp file names per job attempt (unique-part-dir / attempt counters)
Prevent zombie TaskManagers from writing while a new attempt recovers
Never point two running jobs at the same output checkpoint prefix

When it happens

Trigger: commit() after task recovery where another attempt of the same operator already appended to the same temp file, or where the file was truncated by a previous failed recovery attempt; also when the same checkpoint state is committed twice with intervening writes.

Common situations: Two job attempts running concurrently against one checkpoint (zombie TaskManager); inconsistent output-path configuration so two sinks share temp file names; recovery racing with cleanup.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/08937e7edae23632. Report an issue: GitHub.