{"record":{"id":"0fedf89418234dde","repo":"neondatabase/neon","slug":"deleting-cancelled","errorCode":null,"errorMessage":"deleting cancelled","messagePattern":"deleting cancelled","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"endpoint_storage/src/app.rs","lineNumber":132,"sourceCode":"    .await\n    .unwrap_or(Err(anyhow!(\"uploading cancelled\")))\n    .map_err(|e| internal_error(e, path, \"reading response\"))?;\n    Ok(ok())\n}\n\nasync fn delete(S3Path { path }: S3Path, state: State) -> Result {\n    info!(%path, \"deleting\");\n    let cancel = state.cancel.clone();\n    retry(\n        async || state.storage.delete(&path, &cancel).await,\n        TimeoutOrCancel::caused_by_cancel,\n        WARN_THRESHOLD,\n        MAX_RETRIES,\n        \"deleting\",\n        &cancel,\n    )\n    .await\n    .unwrap_or(Err(anyhow!(\"deleting cancelled\")))\n    .map_err(|e| internal_error(e, path, \"deleting\"))?;\n    Ok(ok())\n}\n\nasync fn delete_prefix(PrefixS3Path { path }: PrefixS3Path, state: State) -> Result {\n    info!(%path, \"deleting prefix\");\n    let cancel = state.cancel.clone();\n    retry(\n        async || state.storage.delete_prefix(&path, &cancel).await,\n        TimeoutOrCancel::caused_by_cancel,\n        WARN_THRESHOLD,\n        MAX_RETRIES,\n        \"deleting prefix\",\n        &cancel,\n    )\n    .await\n    .unwrap_or(Err(anyhow!(\"deleting prefix cancelled\")))\n    .map_err(|e| internal_error(e, path, \"deleting prefix\"))?;","sourceCodeStart":114,"sourceCodeEnd":150,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/endpoint_storage/src/app.rs#L114-L150","documentation":"endpoint_storage wraps its S3 delete in a retry helper guarded by a CancellationToken. When the helper returns None — the token fired or the attempt ended cancelled — the code substitutes 'deleting cancelled' via unwrap_or. The delete did not complete because it was cancelled, not because S3 reported an error.","triggerScenarios":"Issuing a DELETE to the endpoint_storage HTTP endpoint while the service's cancel token fires (shutdown/SIGTERM) or the retry loop terminates in a cancelled state; the 500 response wraps this message.","commonSituations":"Deletion requests racing a rolling restart or pod termination; automation deleting objects at the same moment the service drains connections.","solutions":["Retry the DELETE after the service is healthy again — deletes are idempotent","Correlate with service logs to confirm a shutdown/cancel event at that timestamp","For recurring non-shutdown cases, check remote-storage timeouts and the retry budget"],"exampleFix":"# before\ncurl -X DELETE https://endpoint-storage/path/obj\n# after: retry through restarts\ncurl --retry 3 --retry-all-errors -X DELETE https://endpoint-storage/path/obj","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Deletes are idempotent: on a 500 whose body mentions 'cancelled', retry with backoff\nloop {\n    let resp = client.delete(url).await?;\n    if resp.status().is_success() { break; }\n    if resp.status().is_server_error() && body_says_cancelled(&resp).await && attempts < MAX {\n        sleep(backoff(attempts)); attempts += 1; continue;\n    }\n    return Err(non_retryable(resp));\n}","preventionTips":["Treat DELETE as idempotent and always wire retries for it","Quiesce deletion traffic during deployments instead of racing the shutdown path"],"tags":["endpoint-storage","cancellation","delete","retry","rust"],"backgroundTag":"operation-cancelled","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}