{"record":{"id":"927055807d838a2c","repo":"neondatabase/neon","slug":"uploading-cancelled","errorCode":null,"errorMessage":"uploading cancelled","messagePattern":"uploading cancelled","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"endpoint_storage/src/app.rs","lineNumber":115,"sourceCode":"\n    let cancel = state.cancel.clone();\n    let fun = async || {\n        let stream = bytes_to_stream(bytes.clone());\n        state\n            .storage\n            .upload(stream, request_len, &path, None, &cancel)\n            .await\n    };\n    retry(\n        fun,\n        TimeoutOrCancel::caused_by_cancel,\n        WARN_THRESHOLD,\n        MAX_RETRIES,\n        \"uploading\",\n        &cancel,\n    )\n    .await\n    .unwrap_or(Err(anyhow!(\"uploading cancelled\")))\n    .map_err(|e| internal_error(e, path, \"reading response\"))?;\n    Ok(ok())\n}\n\nasync fn delete(S3Path { path }: S3Path, state: State) -> Result {\n    info!(%path, \"deleting\");\n    let cancel = state.cancel.clone();\n    retry(\n        async || state.storage.delete(&path, &cancel).await,\n        TimeoutOrCancel::caused_by_cancel,\n        WARN_THRESHOLD,\n        MAX_RETRIES,\n        \"deleting\",\n        &cancel,\n    )\n    .await\n    .unwrap_or(Err(anyhow!(\"deleting cancelled\")))\n    .map_err(|e| internal_error(e, path, \"deleting\"))?;","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/endpoint_storage/src/app.rs#L97-L133","documentation":"endpoint_storage wraps its S3 upload in a retry helper guarded by a CancellationToken. When the retry helper returns None — the cancellation token fired (service shutdown) or the attempt timed out terminally — the code substitutes this 'uploading cancelled' error via unwrap_or. It indicates the operation did not complete because it was cancelled, not that S3 returned an error.","triggerScenarios":"The service's global cancel token fires while an upload retry loop is in progress (shutdown, SIGTERM), or the upload exhausts its retry budget in a cancelled/timed-out state. The 500 response then wraps 'uploading cancelled'.","commonSituations":"Rolling restarts or pod eviction while uploads are in flight; Kubernetes liveness failures killing the process mid-upload; clients uploading when the service is shutting down and reading a 500 instead of a clean connection close.","solutions":["Retry the upload once the service is back up — cancellation leaves no partial committed state to clean up","Check service logs for shutdown signals (SIGTERM, cancel token) that coincide with the failure","If this recurs outside shutdowns, investigate remote-storage latency/timeouts exhausting the retry budget"],"exampleFix":"# before: uploading during a restart\ncurl -T big-file.bin https://endpoint-storage/...\n# after: wait for the service to be ready, then retry\ncurl --retry 3 --retry-all-errors -T big-file.bin https://endpoint-storage/...","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Client side: distinguish cancellation (500 with 'cancelled') from S3 errors, retry when healthy\nif status.is_server_error() && text.contains(\"cancelled\") {\n    // operation was aborted mid-flight; safe to retry once the service reports ready\n    backoff_and_retry().await?;\n} else if status.is_server_error() {\n    // genuine storage error: inspect 5xx body, likely not transient\n}","preventionTips":["Use readiness probes: only upload when endpoint_storage reports ready, avoiding drain windows","Make uploads retryable (idempotent PUT to a deterministic path) so cancellation is recoverable","Watch service logs for cancel-token firing to distinguish shutdowns from storage faults"],"tags":["endpoint-storage","cancellation","upload","retry","rust"],"backgroundTag":"operation-cancelled","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}