neondatabase/neon · error

deleting prefix cancelled

Error message

deleting prefix cancelled

What it means

endpoint_storage wraps its S3 delete_prefix in a retry helper guarded by a CancellationToken. When the helper returns None — the token fired or the loop ended cancelled — the code substitutes 'deleting prefix cancelled' via unwrap_or. The prefix deletion did not complete because it was cancelled.

Source

Thrown at endpoint_storage/src/app.rs:149

    .await
    .unwrap_or(Err(anyhow!("deleting cancelled")))
    .map_err(|e| internal_error(e, path, "deleting"))?;
    Ok(ok())
}

async fn delete_prefix(PrefixS3Path { path }: PrefixS3Path, state: State) -> Result {
    info!(%path, "deleting prefix");
    let cancel = state.cancel.clone();
    retry(
        async || state.storage.delete_prefix(&path, &cancel).await,
        TimeoutOrCancel::caused_by_cancel,
        WARN_THRESHOLD,
        MAX_RETRIES,
        "deleting prefix",
        &cancel,
    )
    .await
    .unwrap_or(Err(anyhow!("deleting prefix cancelled")))
    .map_err(|e| internal_error(e, path, "deleting prefix"))?;
    Ok(ok())
}

pub async fn check_storage_permissions(
    client: &GenericRemoteStorage,
    cancel: CancellationToken,
) -> anyhow::Result<()> {
    info!("storage permissions check");

    // as_nanos() as multiple instances proxying same bucket may be started at once
    let now = SystemTime::now()
        .duration_since(UNIX_EPOCH)?
        .as_nanos()
        .to_string();

    let path = RemotePath::from_string(&format!("write_access_{now}"))?;
    info!(%path, "uploading");

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Re-issue the prefix deletion once the service is back — delete_prefix is idempotent
2. Raise the termination grace period or stop sending prefix deletes during drain windows
3. Investigate remote-storage latency if cancellations happen without shutdowns

Example fix

# before: fire-and-forget during restarts
curl -X DELETE https://endpoint-storage/prefix/path/
# after: retry until it completes
curl --retry 5 --retry-all-errors -X DELETE https://endpoint-storage/prefix/path/

// Prefix deletes are idempotent; retry 'cancelled' 500s until the service is stable
let mut attempts = 0;
while attempts < MAX_ATTEMPTS {
    match send_prefix_delete(url).await {
        Ok(_) => break,
        Err(e) if is_cancelled_500(&e) => { attempts += 1; tokio::time::sleep(backoff).await; }
        Err(e) => return Err(e),
    }
}

When it happens

Trigger: Issuing a prefix-delete request to endpoint_storage while the service shuts down (cancel token fired) or the retry loop terminates cancelled; the 500 response wraps this message.

Common situations: Bulk cleanup requests racing a deployment restart; long-running prefix deletions that outlive a pod's termination grace period.

Related errors

• uploading cancelled
• deleting cancelled
• Exhausted all attempts to retrieve the config from the contr
• File size {request_len} exceeds max {max_len}
• Read back file doesn't match original

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/0ad9d3a3e36312eb. Report an issue: GitHub.