What causes "Received abort for copy from {from} to {to}." in neondatabase/neon?

Serialize writers: ensure only one upload/copy targets the destination key at a time

Received abort for copy from {from} to {to}.

Error message

Received abort for copy from {from} to {to}.

What it means

Azure's Copy Blob is asynchronous: the backend polls get_properties once per second and matches the reported copy_status. CopyStatus::Aborted means Azure terminated the pending copy — most commonly because the destination blob was modified or overwritten while the copy was pending, the copy was explicitly aborted, or the source became unavailable.

Source

Thrown at libs/remote_storage/src/azure_blob.rs:903

        let op = async {
            let blob_client = self.client.blob_client(self.relative_path_to_name(to));

            let source_url = format!(
                "{}/{}",
                self.client.url()?,
                self.relative_path_to_name(from)
            );

            let builder = blob_client.copy(Url::from_str(&source_url)?);
            let copy = builder.into_future();

            let result = copy.await?;

            copy_status = Some(result.copy_status);
            loop {
                match copy_status.as_ref().expect("we always set it to Some") {
                    CopyStatus::Aborted => {
                        anyhow::bail!("Received abort for copy from {from} to {to}.");
                    }
                    CopyStatus::Failed => {
                        anyhow::bail!("Received failure response for copy from {from} to {to}.");
                    }
                    CopyStatus::Success => return Ok(()),
                    CopyStatus::Pending => (),
                }
                // The copy is taking longer. Waiting a second and then re-trying.
                // TODO estimate time based on copy_progress and adjust time based on that
                tokio::time::sleep(Duration::from_millis(1000)).await;
                let properties = blob_client.get_properties().into_future().await?;
                let Some(status) = properties.blob.properties.copy_status else {
                    tracing::warn!("copy_status for copy is None!, from={from}, to={to}");
                    return Ok(());
                };
                copy_status = Some(status);
            }
        };

View on GitHub (pinned to 8f60b04da4)

Solutions

Serialize writers: ensure only one upload/copy targets the destination key at a time
Retry the copy once the conflicting writer has finished
Verify the source blob still exists immediately before issuing the copy
For small objects use download+upload instead of server-side copy to avoid long-pending async copies

Example fix

// before: concurrent writers can abort the pending copy
let _ = storage.copy(&from, &to, &cancel).await;

// after: single-writer per destination via a keyed lock
let _guard = key_locks.lock(to.clone()).await;
storage.copy(&from, &to, &cancel).await?;

Defensive patterns

Strategy: retry

Try / catch

// Catch, distinguish abort from other failures, re-issue the copy after the contended writer settles.
match storage.copy(&from, &to, &cancel).await {
    Ok(()) => {}
    Err(e) if format!("{e:#}").contains("Received abort for copy") => {
        tracing::warn!("copy aborted (contended destination), retrying: {from} -> {to}");
        tokio::time::sleep(Duration::from_secs(1)).await;
        storage.copy(&from, &to, &cancel).await?;
    }
    Err(e) => return Err(e),
}

Prevention

Serialize all writers (uploads and copies) per destination key with a keyed lock or queue
Never run two copies to the same destination concurrently, including across retry logic
Keep source objects alive (defer GC/retention deletion) until in-flight copies complete

When it happens

Trigger: Calling RemoteStorage::copy on Azure while another writer overwrites/puts the destination blob during the pending copy; another actor issues Abort Copy Blob; the source blob is deleted mid-copy; two concurrent copies race to the same destination key.

Common situations: Concurrent upload and copy targeting the same destination path; retry storms where a retried copy races the previous attempt; GC deleting source timelines while copies are in flight.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/6fe2c5e634c79250. Report an issue: GitHub.