apache/flink · critical · IOException

Unrecoverable exception while trying to recover {}

Error message

Unrecoverable exception while trying to recover {}

What it means

Thrown by the AzureBlobFsRecoverableDataOutputStream Committer when, at commit time, neither the staging (temp) file nor the final target file exists. The code comments flag exactly two possible meanings: (1) real file-system data loss, or (2) recovering from an old savepoint whose output files were removed afterwards. Commit is impossible either way.

Source

Thrown at flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AzureBlobFsRecoverableDataOutputStream.java:295

                                    + src
                                    + " with length "
                                    + srcStatus.getLen()
                                    + " "
                                    + "does not match the expected length "
                                    + expectedLength);
                }
                try {
                    fs.rename(src, dest);
                } catch (IOException e) {
                    throw new IOException(
                            "Committing file by rename failed: " + src + " to " + dest, e);
                }
            } else if (!fs.exists(dest)) {
                // neither exists - that can be a sign of
                //   - (1) a serious problem (file system loss of data)
                //   - (2) a recovery of a savepoint that is some time old and the users
                //         removed the files in the meantime.
                throw new IOException(
                        "Unrecoverable exception while trying to recover "
                                + recoverable.tempFile());
            }
        }

        @Override
        public void commitAfterRecovery() throws IOException {
            commit();
        }

        @Override
        public RecoverableWriter.CommitRecoverable getRecoverable() {
            return recoverable;
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

If files were intentionally deleted, accept the loss and restart with a fresh sink state (new checkpoint) rather than the stale savepoint
Verify the target path/container still holds the expected part files before resuming an old savepoint
Protect output directories from lifecycle policies and manual cleanup
For suspected data loss, engage Azure support with the timestamps from the task log

Defensive patterns

Strategy: try-catch

Validate before calling

// before resuming an old savepoint, verify referenced files still exist
if (!fs.exists(recoverable.tempFile()) && !fs.exists(recoverable.targetFile())) {
    // files already gone: plan a fresh run instead of a doomed recovery
}

Try / catch

try {
    committer.commitAfterRecovery();
} catch (java.io.IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("Unrecoverable")) {
        restartFreshWithoutStaleSavepoint();
    } else { throw e; }
}

Prevention

Archive or preserve part files for the lifetime of any savepoint you may resume
Document retention windows for output buckets
Dry-run savepoint resumes in a test container against the real bucket

When it happens

Trigger: commitAfterRecovery() on a savepoint taken long ago, after the bucket contents were cleaned; or genuine ABFS data loss where both the temp file and an already-committed target vanished.

Common situations: Resuming an old savepoint for a file sink whose part files were archived/deleted by policy or manually; storage account corrupted or misconfigured after the checkpoint was taken.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/ec51cdfe8608ff06. Report an issue: GitHub.