Post-sync backup staging disappeared before verification; op

Error message

Post-sync backup staging disappeared before verification; operation will be retried: splitId={}, source={}, staging={}, checkpointId={}

What it means

After the enumerator renames the source into the staging path, it re-checks the staging file before verification. This WARN fires when that re-check finds nothing: the staged file disappeared between the rename and the verification. The operation is retried (FAILED_RETRYABLE). It signals an external actor or FS inconsistency removing the staged file.

Source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/ContinuousMultipleTableFileSourceSplitEnumerator.java:912

            return OpCommitResult.FAILED_RETRYABLE;
        }

        if (stagingStatus == null) {
            try {
                ctx.sourceFs.renameFile(op.getSourcePath(), stagingPath, false);
            } catch (Exception e) {
                log.warn(
                        "Post-sync backup: rename-to-staging failed, will retry: source={}, staging={}",
                        maskUriUserInfo(op.getSourcePath()),
                        maskUriUserInfo(stagingPath),
                        e);
                return OpCommitResult.FAILED_RETRYABLE;
            }
            stagingStatus = getFileStatusIfPresent(ctx.sourceFs, stagingPath);
        }

        if (stagingStatus == null) {
            log.warn(
                    "Post-sync backup staging disappeared before verification; operation will be retried: "
                            + "splitId={}, source={}, staging={}, checkpointId={}",
                    op.getSplitId(),
                    maskUriUserInfo(op.getSourcePath()),
                    maskUriUserInfo(stagingPath),
                    checkpointId);
            return OpCommitResult.FAILED_RETRYABLE;
        }

        if (!isOperationContentMatched(ctx, op, stagingPath, stagingStatus)) {
            return handleStaleStagedOperation(
                    ctx, op, checkpointId, stagingPath, "backup", stagingStatus);
        }

        if (!isSinkTargetCommitted(ctx, op, checkpointId, stagingPath)) {
            return handleRetryableStagedOperation(
                    ctx, op, checkpointId, stagingPath, "backup", stagingStatus);
        }

View on GitHub (pinned to cf67b549a7)

Solutions

Ensure only one job instance runs against the same source/staging paths; kill duplicate pipelines
Exclude the staging directory from external cleanup/TTL jobs
Use a job-unique staging path (buildBackupStagingPath input) so concurrent runs never collide
If using an eventually-consistent object store, increase the retry interval so listings settle

Example fix

null

Defensive patterns

Strategy: retry

Validate before calling

// After a successful rename, verify the staged file before proceeding in your own tooling
if (!fs.exists(stagingPath)) {
    throw new IOException("Staged file missing right after rename: " + stagingPath);
}

Try / catch

try {
    verifyStaged(stagingPath);
} catch (IOException e) {
    // transient disappearance: backoff, re-list, retry
}

Prevention

Do not run cleanup jobs against the staging directory
Use job-unique staging paths to avoid cross-pipeline collisions
On eventually-consistent stores, allow settle time before verification

When it happens

Trigger: renameFile(source -> staging) reports success, but the immediately following getFileStatusIfPresent(ctx.sourceFs, stagingPath) returns null.

Common situations: A cleanup/TTL job or another pipeline instance deleting files under the staging directory; concurrent job instances using the same staging path; object-store listing inconsistency right after rename (S3/OSS eventual consistency); manual operator intervention.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/54ce014aefffa32c. Report an issue: GitHub.