neondatabase/neon · error

Failed to upload all blocks {:#?}

Error message

Failed to upload all blocks {:#?}

What it means

Thrown by the Azure Blob backend's upload() after a large object is split into blocks with Put Block and futures::future::try_join_all over the spawned block-upload tasks fails. At least one block request returned Err (network error, auth error, or the per-block tokio::time::timeout converting to an azure_core Io/TimedOut error), or a spawned task panicked (JoinError); the first failure is embedded in the message.

Source

Thrown at libs/remote_storage/src/azure_blob.rs:709

                remaining_bytes -= block_size;
                start_bytes += block_size as u64;

                block_list
                    .blocks
                    .push(BlobBlockType::Uncommitted(encoded_block_id.to_vec().into()));
            }

            tracing::debug!(
                "azure put blocks {} total MB: {} chunk size MB: {}",
                block_list_count,
                data_size_bytes / 1024 / 1024,
                put_block_size / 1024 / 1024
            );
            // Wait for all blocks to be uploaded.
            let upload_results = futures::future::try_join_all(upload_futures).await;
            if upload_results.is_err() {
                return Err(anyhow::anyhow!(format!(
                    "Failed to upload all blocks {:#?}",
                    upload_results.unwrap_err()
                )));
            }

            // Commit the blocks.
            let mut builder = blob_client.put_block_list(block_list);
            if !metadata_map.0.is_empty() {
                builder = builder.metadata(to_azure_metadata(metadata_map));
            }
            let fut = builder.into_future();
            let fut = tokio::time::timeout(self.timeout, fut);
            let result = fut.await;
            tracing::debug!("azure put block list response {:#?}", result);

            match result {
                Ok(Ok(_response)) => Ok(()),
                Ok(Err(azure)) => Err(azure.into()),
                Err(_timeout) => Err(TimeoutOrCancel::Timeout.into()),

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Retry the whole upload: uncommitted blocks simply expire, so re-uploading is safe and usually succeeds on transient faults
2. Read the embedded error: an azure_core::Error with ErrorKind::Io / 'Operation timed out' means the per-block timeout is too small — increase timeout or reduce put_block_size
3. Check Azure storage metrics for throttling (429/503) and lower block count/concurrency or block size
4. Verify credentials are valid for the full upload duration (SAS expiry, key rotation)
5. A JoinError means a block task panicked — check the source file still exists and is readable at the path logged by 'azure put block' debug lines

Example fix

// before: single-shot, fails hard on any transient block failure
remote_storage.upload(
    UploadOpts::from(path),
    &BytesSource::read_from_file(&file, data_size_bytes),
    &cancel,
).await?;

// after: retry with backoff — block uploads are idempotent
let upload = || async {
    remote_storage
        .upload(
            UploadOpts::from(path.clone()),
            &BytesSource::read_from_file(&file, data_size_bytes),
            &cancel,
        )
        .await
};
backoff::retry(upload, is_not_permanent, warn_threshold, max_retries, "azure upload", &cancel).await?;

// In Rust: match the upload result, classify transient vs permanent, retry with backoff.
const MAX_ATTEMPTS: u32 = 5;
async fn upload_with_retry(storage: &Arc<GenericRemoteStorage>, data: &str, size: u64, cancel: &CancellationToken) -> anyhow::Result<()> {
    let mut attempt = 0;
    loop {
        attempt += 1;
        match storage.upload(UploadOpts::from(path.clone()), &BytesSource::read_from_string(data), cancel).await {
            Ok(()) => return Ok(()),
            Err(e) if attempt < MAX_ATTEMPTS && is_transient(&e) => {
                tracing::warn!("azure block upload failed (attempt {attempt}): {e:#}");
                tokio::time::sleep(Duration::from_millis(200 * 2u64.pow(attempt - 1))).await;
            }
            Err(e) => return Err(e),
        }
    }
}
fn is_transient(e: &anyhow::Error) -> bool {
    let msg = format!("{e:#}");
    msg.contains("Failed to upload all blocks")
        && !msg.contains("AuthorizationFailure") // permanent: do not retry
        && !msg.contains("authentication")
}

When it happens

Trigger: Uploading an object large enough to take the multi-block path when any single put_block request errors or times out, or when a block task panics (e.g. File::open/seek fails because the source file vanished). try_join_all short-circuits on the first Err and the whole upload fails before put_block_list commits.

Common situations: Flaky or slow networks during multi-hundred-MB uploads; storage credentials (account key/SAS) expiring mid-upload; Azure throttling (429/503) when many blocks upload concurrently; per-request timeout too small for the configured put_block_size.

Related errors

• error downloading extension {:?}: {:?}
• max_keys_per_list_response can't be 0
• Azure GET response contained no response body
• Received abort for copy from {from} to {to}.
• Received failure response for copy from {from} to {to}.

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/473f473ee7527775. Report an issue: GitHub.