neondatabase/neon · error

deleting cancelled

Error message

deleting cancelled

What it means

endpoint_storage wraps its S3 delete in a retry helper guarded by a CancellationToken. When the helper returns None — the token fired or the attempt ended cancelled — the code substitutes 'deleting cancelled' via unwrap_or. The delete did not complete because it was cancelled, not because S3 reported an error.

Source

Thrown at endpoint_storage/src/app.rs:132

    .await
    .unwrap_or(Err(anyhow!("uploading cancelled")))
    .map_err(|e| internal_error(e, path, "reading response"))?;
    Ok(ok())
}

async fn delete(S3Path { path }: S3Path, state: State) -> Result {
    info!(%path, "deleting");
    let cancel = state.cancel.clone();
    retry(
        async || state.storage.delete(&path, &cancel).await,
        TimeoutOrCancel::caused_by_cancel,
        WARN_THRESHOLD,
        MAX_RETRIES,
        "deleting",
        &cancel,
    )
    .await
    .unwrap_or(Err(anyhow!("deleting cancelled")))
    .map_err(|e| internal_error(e, path, "deleting"))?;
    Ok(ok())
}

async fn delete_prefix(PrefixS3Path { path }: PrefixS3Path, state: State) -> Result {
    info!(%path, "deleting prefix");
    let cancel = state.cancel.clone();
    retry(
        async || state.storage.delete_prefix(&path, &cancel).await,
        TimeoutOrCancel::caused_by_cancel,
        WARN_THRESHOLD,
        MAX_RETRIES,
        "deleting prefix",
        &cancel,
    )
    .await
    .unwrap_or(Err(anyhow!("deleting prefix cancelled")))
    .map_err(|e| internal_error(e, path, "deleting prefix"))?;

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Retry the DELETE after the service is healthy again — deletes are idempotent
2. Correlate with service logs to confirm a shutdown/cancel event at that timestamp
3. For recurring non-shutdown cases, check remote-storage timeouts and the retry budget

Example fix

# before
curl -X DELETE https://endpoint-storage/path/obj
# after: retry through restarts
curl --retry 3 --retry-all-errors -X DELETE https://endpoint-storage/path/obj

// Deletes are idempotent: on a 500 whose body mentions 'cancelled', retry with backoff
loop {
    let resp = client.delete(url).await?;
    if resp.status().is_success() { break; }
    if resp.status().is_server_error() && body_says_cancelled(&resp).await && attempts < MAX {
        sleep(backoff(attempts)); attempts += 1; continue;
    }
    return Err(non_retryable(resp));
}

When it happens

Trigger: Issuing a DELETE to the endpoint_storage HTTP endpoint while the service's cancel token fires (shutdown/SIGTERM) or the retry loop terminates in a cancelled state; the 500 response wraps this message.

Common situations: Deletion requests racing a rolling restart or pod termination; automation deleting objects at the same moment the service drains connections.

Related errors

• uploading cancelled
• deleting prefix cancelled
• Exhausted all attempts to retrieve the config from the contr
• File size {request_len} exceeds max {max_len}
• Read back file doesn't match original

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/0fedf89418234dde. Report an issue: GitHub.