{"record":{"id":"473f473ee7527775","repo":"neondatabase/neon","slug":"failed-to-upload-all-blocks","errorCode":null,"errorMessage":"Failed to upload all blocks {:#?}","messagePattern":"Failed to upload all blocks (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"libs/remote_storage/src/azure_blob.rs","lineNumber":709,"sourceCode":"                remaining_bytes -= block_size;\n                start_bytes += block_size as u64;\n\n                block_list\n                    .blocks\n                    .push(BlobBlockType::Uncommitted(encoded_block_id.to_vec().into()));\n            }\n\n            tracing::debug!(\n                \"azure put blocks {} total MB: {} chunk size MB: {}\",\n                block_list_count,\n                data_size_bytes / 1024 / 1024,\n                put_block_size / 1024 / 1024\n            );\n            // Wait for all blocks to be uploaded.\n            let upload_results = futures::future::try_join_all(upload_futures).await;\n            if upload_results.is_err() {\n                return Err(anyhow::anyhow!(format!(\n                    \"Failed to upload all blocks {:#?}\",\n                    upload_results.unwrap_err()\n                )));\n            }\n\n            // Commit the blocks.\n            let mut builder = blob_client.put_block_list(block_list);\n            if !metadata_map.0.is_empty() {\n                builder = builder.metadata(to_azure_metadata(metadata_map));\n            }\n            let fut = builder.into_future();\n            let fut = tokio::time::timeout(self.timeout, fut);\n            let result = fut.await;\n            tracing::debug!(\"azure put block list response {:#?}\", result);\n\n            match result {\n                Ok(Ok(_response)) => Ok(()),\n                Ok(Err(azure)) => Err(azure.into()),\n                Err(_timeout) => Err(TimeoutOrCancel::Timeout.into()),","sourceCodeStart":691,"sourceCodeEnd":727,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/libs/remote_storage/src/azure_blob.rs#L691-L727","documentation":"Thrown by the Azure Blob backend's upload() after a large object is split into blocks with Put Block and futures::future::try_join_all over the spawned block-upload tasks fails. At least one block request returned Err (network error, auth error, or the per-block tokio::time::timeout converting to an azure_core Io/TimedOut error), or a spawned task panicked (JoinError); the first failure is embedded in the message.","triggerScenarios":"Uploading an object large enough to take the multi-block path when any single put_block request errors or times out, or when a block task panics (e.g. File::open/seek fails because the source file vanished). try_join_all short-circuits on the first Err and the whole upload fails before put_block_list commits.","commonSituations":"Flaky or slow networks during multi-hundred-MB uploads; storage credentials (account key/SAS) expiring mid-upload; Azure throttling (429/503) when many blocks upload concurrently; per-request timeout too small for the configured put_block_size.","solutions":["Retry the whole upload: uncommitted blocks simply expire, so re-uploading is safe and usually succeeds on transient faults","Read the embedded error: an azure_core::Error with ErrorKind::Io / 'Operation timed out' means the per-block timeout is too small — increase timeout or reduce put_block_size","Check Azure storage metrics for throttling (429/503) and lower block count/concurrency or block size","Verify credentials are valid for the full upload duration (SAS expiry, key rotation)","A JoinError means a block task panicked — check the source file still exists and is readable at the path logged by 'azure put block' debug lines"],"exampleFix":"// before: single-shot, fails hard on any transient block failure\nremote_storage.upload(\n    UploadOpts::from(path),\n    &BytesSource::read_from_file(&file, data_size_bytes),\n    &cancel,\n).await?;\n\n// after: retry with backoff — block uploads are idempotent\nlet upload = || async {\n    remote_storage\n        .upload(\n            UploadOpts::from(path.clone()),\n            &BytesSource::read_from_file(&file, data_size_bytes),\n            &cancel,\n        )\n        .await\n};\nbackoff::retry(upload, is_not_permanent, warn_threshold, max_retries, \"azure upload\", &cancel).await?;","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// In Rust: match the upload result, classify transient vs permanent, retry with backoff.\nconst MAX_ATTEMPTS: u32 = 5;\nasync fn upload_with_retry(storage: &Arc<GenericRemoteStorage>, data: &str, size: u64, cancel: &CancellationToken) -> anyhow::Result<()> {\n    let mut attempt = 0;\n    loop {\n        attempt += 1;\n        match storage.upload(UploadOpts::from(path.clone()), &BytesSource::read_from_string(data), cancel).await {\n            Ok(()) => return Ok(()),\n            Err(e) if attempt < MAX_ATTEMPTS && is_transient(&e) => {\n                tracing::warn!(\"azure block upload failed (attempt {attempt}): {e:#}\");\n                tokio::time::sleep(Duration::from_millis(200 * 2u64.pow(attempt - 1))).await;\n            }\n            Err(e) => return Err(e),\n        }\n    }\n}\nfn is_transient(e: &anyhow::Error) -> bool {\n    let msg = format!(\"{e:#}\");\n    msg.contains(\"Failed to upload all blocks\")\n        && !msg.contains(\"AuthorizationFailure\") // permanent: do not retry\n        && !msg.contains(\"authentication\")\n}","preventionTips":["Size put_block_size and the request timeout together: each block must transfer within one timeout window","Monitor per-upload failure rates; a rising rate usually precedes throttling (429/503) — lower concurrency then","Keep the source file immutable and present for the whole upload; block tasks re-open and seek it","Wrap every large-object upload in bounded backoff retry rather than failing the caller on first transient error"],"tags":["azure","blob-storage","multipart-upload","rust","network"],"backgroundTag":"multipart-upload-failed","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}