{"record":{"id":"0ad9d3a3e36312eb","repo":"neondatabase/neon","slug":"deleting-prefix-cancelled","errorCode":null,"errorMessage":"deleting prefix cancelled","messagePattern":"deleting prefix cancelled","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"endpoint_storage/src/app.rs","lineNumber":149,"sourceCode":"    .await\n    .unwrap_or(Err(anyhow!(\"deleting cancelled\")))\n    .map_err(|e| internal_error(e, path, \"deleting\"))?;\n    Ok(ok())\n}\n\nasync fn delete_prefix(PrefixS3Path { path }: PrefixS3Path, state: State) -> Result {\n    info!(%path, \"deleting prefix\");\n    let cancel = state.cancel.clone();\n    retry(\n        async || state.storage.delete_prefix(&path, &cancel).await,\n        TimeoutOrCancel::caused_by_cancel,\n        WARN_THRESHOLD,\n        MAX_RETRIES,\n        \"deleting prefix\",\n        &cancel,\n    )\n    .await\n    .unwrap_or(Err(anyhow!(\"deleting prefix cancelled\")))\n    .map_err(|e| internal_error(e, path, \"deleting prefix\"))?;\n    Ok(ok())\n}\n\npub async fn check_storage_permissions(\n    client: &GenericRemoteStorage,\n    cancel: CancellationToken,\n) -> anyhow::Result<()> {\n    info!(\"storage permissions check\");\n\n    // as_nanos() as multiple instances proxying same bucket may be started at once\n    let now = SystemTime::now()\n        .duration_since(UNIX_EPOCH)?\n        .as_nanos()\n        .to_string();\n\n    let path = RemotePath::from_string(&format!(\"write_access_{now}\"))?;\n    info!(%path, \"uploading\");","sourceCodeStart":131,"sourceCodeEnd":167,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/endpoint_storage/src/app.rs#L131-L167","documentation":"endpoint_storage wraps its S3 delete_prefix in a retry helper guarded by a CancellationToken. When the helper returns None — the token fired or the loop ended cancelled — the code substitutes 'deleting prefix cancelled' via unwrap_or. The prefix deletion did not complete because it was cancelled.","triggerScenarios":"Issuing a prefix-delete request to endpoint_storage while the service shuts down (cancel token fired) or the retry loop terminates cancelled; the 500 response wraps this message.","commonSituations":"Bulk cleanup requests racing a deployment restart; long-running prefix deletions that outlive a pod's termination grace period.","solutions":["Re-issue the prefix deletion once the service is back — delete_prefix is idempotent","Raise the termination grace period or stop sending prefix deletes during drain windows","Investigate remote-storage latency if cancellations happen without shutdowns"],"exampleFix":"# before: fire-and-forget during restarts\ncurl -X DELETE https://endpoint-storage/prefix/path/\n# after: retry until it completes\ncurl --retry 5 --retry-all-errors -X DELETE https://endpoint-storage/prefix/path/","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Prefix deletes are idempotent; retry 'cancelled' 500s until the service is stable\nlet mut attempts = 0;\nwhile attempts < MAX_ATTEMPTS {\n    match send_prefix_delete(url).await {\n        Ok(_) => break,\n        Err(e) if is_cancelled_500(&e) => { attempts += 1; tokio::time::sleep(backoff).await; }\n        Err(e) => return Err(e),\n    }\n}","preventionTips":["Schedule bulk prefix deletions outside deploy/drain windows","Give the pod enough termination grace period for in-flight prefix deletions to finish"],"tags":["endpoint-storage","cancellation","delete-prefix","retry","rust"],"backgroundTag":"operation-cancelled","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}