Boundary string from 'multipart/related' HTTP upload occurre

Error message

Boundary string from 'multipart/related' HTTP upload occurred in payload

What it means

After an HTTP-successful multipart upload the backend verifies the size GCS reports for the stored object equals the number of bytes it declared it would stream (fs_size). A mismatch trips this integrity check; the code attributes it to the random multipart boundary string occurring inside the payload, which would make the server split the body at the wrong offset and store wrong bytes. Marked 'very unlikely' — but any size disagreement (fs_size computed wrong up front, short/truncated stream) also lands here.

Source

Thrown at libs/remote_storage/src/gcs_bucket.rs:412

        match res {
            Ok(res) => {
                if !res.status().is_success() {
                    match res.status() {
                        _ => Err(anyhow::anyhow!("GCS PUT error \n\t {:?}", res)),
                    }
                } else {
                    let body = res
                        .text()
                        .await
                        .map_err(|e: reqwest::Error| DownloadError::Other(e.into()))?;

                    let resp: GCSObject = serde_json::from_str(&body)
                        .map_err(|e: serde_json::Error| DownloadError::Other(e.into()))?;

                    if !resp.size.is_some_and(|s| s == fs_size as i64) {
                        // very unlikely
                        return Err(anyhow::anyhow!(
                            "Boundary string from 'multipart/related' HTTP upload occurred in payload"
                        ));
                    };

                    Ok(())
                }
            }
            Err(reqw) => Err(reqw.into()),
        }
    }

    async fn delete_oids(
        &self,
        delete_objects: &[String],
        cancel: &CancellationToken,
        _permit: &tokio::sync::SemaphorePermit<'_>,
    ) -> anyhow::Result<()> {
        let kind = RequestKind::Delete;
        let mut cancel = std::pin::pin!(cancel.cancelled());

View on GitHub (pinned to 8f60b04da4)

Solutions

Retry the upload — a new attempt generates a new boundary and re-transfers the bytes
Verify the fs_size passed to upload matches the exact byte count the stream will produce (stat the file right before streaming, or compute from the source of truth)
Hold the source file immutable (no truncation/append) between size computation and upload completion
If genuinely reproducible, switch to a resumable upload session to avoid multipart framing entirely

Example fix

// before: size computed once, possibly stale by upload time
let fs_size = file.metadata().await?.len();
storage.upload(&data, fs_size, &cancel).await?;

// after: freeze the byte range being uploaded (or re-stat under a lock)
let fs_size = {
    let f = File::open(&path).await?;
    let meta = f.metadata().await?;
    // upload from an open handle so size and content cannot diverge
    f.into_std().await.into()
};

Defensive patterns

Strategy: retry

Validate before calling

// Ensure the declared size matches the stream before uploading.
fn assert_size_matches(source: &std::fs::File, declared_len: u64) -> anyhow::Result<()> {
    let actual = source.metadata()?.len();
    anyhow::ensure!(actual == declared_len,
        "size mismatch: declared {declared_len}, stream holds {actual}");
    Ok(())
}

Try / catch

// Integrity-check failure: retry uploads (new boundary each attempt), escalate if persistent.
for attempt in 1..=3 {
    match storage.upload(&data, fs_size, &cancel).await {
        Ok(()) => break,
        Err(e) if format!("{e:#}").contains("Boundary string") && attempt < 3 => {
            tracing::warn!("upload integrity check tripped (attempt {attempt}); retrying");
            tokio::time::sleep(Duration::from_millis(250)).await;
        }
        Err(e) => return Err(e),
    }
}

Prevention

Compute fs_size from the same open handle that streams the bytes, never from a separately-stat'ed path
Keep source files immutable during upload (no truncate/append between size check and transfer)
Never ignore this error — it guards against silently storing corrupted data

When it happens

Trigger: Uploading a payload that happens to contain the generated multipart boundary string; or passing an fs_size that disagrees with the actual byte count of the stream (bug in size accounting, short read from the source stream, concurrent file truncation while uploading).

Common situations: Almost never a real boundary collision; more plausibly a caller streaming a file that changed size between stat and upload, or a computed size off by framing overhead.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/ed665e7d6ed84057. Report an issue: GitHub.