neondatabase/neon · error

uploading cancelled

Error message

uploading cancelled

What it means

endpoint_storage wraps its S3 upload in a retry helper guarded by a CancellationToken. When the retry helper returns None — the cancellation token fired (service shutdown) or the attempt timed out terminally — the code substitutes this 'uploading cancelled' error via unwrap_or. It indicates the operation did not complete because it was cancelled, not that S3 returned an error.

Source

Thrown at endpoint_storage/src/app.rs:115

    let cancel = state.cancel.clone();
    let fun = async || {
        let stream = bytes_to_stream(bytes.clone());
        state
            .storage
            .upload(stream, request_len, &path, None, &cancel)
            .await
    };
    retry(
        fun,
        TimeoutOrCancel::caused_by_cancel,
        WARN_THRESHOLD,
        MAX_RETRIES,
        "uploading",
        &cancel,
    )
    .await
    .unwrap_or(Err(anyhow!("uploading cancelled")))
    .map_err(|e| internal_error(e, path, "reading response"))?;
    Ok(ok())
}

async fn delete(S3Path { path }: S3Path, state: State) -> Result {
    info!(%path, "deleting");
    let cancel = state.cancel.clone();
    retry(
        async || state.storage.delete(&path, &cancel).await,
        TimeoutOrCancel::caused_by_cancel,
        WARN_THRESHOLD,
        MAX_RETRIES,
        "deleting",
        &cancel,
    )
    .await
    .unwrap_or(Err(anyhow!("deleting cancelled")))
    .map_err(|e| internal_error(e, path, "deleting"))?;

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Retry the upload once the service is back up — cancellation leaves no partial committed state to clean up
2. Check service logs for shutdown signals (SIGTERM, cancel token) that coincide with the failure
3. If this recurs outside shutdowns, investigate remote-storage latency/timeouts exhausting the retry budget

Example fix

# before: uploading during a restart
curl -T big-file.bin https://endpoint-storage/...
# after: wait for the service to be ready, then retry
curl --retry 3 --retry-all-errors -T big-file.bin https://endpoint-storage/...

// Client side: distinguish cancellation (500 with 'cancelled') from S3 errors, retry when healthy
if status.is_server_error() && text.contains("cancelled") {
    // operation was aborted mid-flight; safe to retry once the service reports ready
    backoff_and_retry().await?;
} else if status.is_server_error() {
    // genuine storage error: inspect 5xx body, likely not transient
}

When it happens

Trigger: The service's global cancel token fires while an upload retry loop is in progress (shutdown, SIGTERM), or the upload exhausts its retry budget in a cancelled/timed-out state. The 500 response then wraps 'uploading cancelled'.

Common situations: Rolling restarts or pod eviction while uploads are in flight; Kubernetes liveness failures killing the process mid-upload; clients uploading when the service is shutting down and reading a 500 instead of a clean connection close.

Related errors

• deleting cancelled
• deleting prefix cancelled
• File size {request_len} exceeds max {max_len}
• Exhausted all attempts to retrieve the config from the contr
• Read back file doesn't match original

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/927055807d838a2c. Report an issue: GitHub.