{"record":{"id":"ed665e7d6ed84057","repo":"neondatabase/neon","slug":"boundary-string-from-multipart-related-http-uplo","errorCode":null,"errorMessage":"Boundary string from 'multipart/related' HTTP upload occurred in payload","messagePattern":"Boundary string from 'multipart/related' HTTP upload occurred in payload","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"libs/remote_storage/src/gcs_bucket.rs","lineNumber":412,"sourceCode":"        match res {\n            Ok(res) => {\n                if !res.status().is_success() {\n                    match res.status() {\n                        _ => Err(anyhow::anyhow!(\"GCS PUT error \\n\\t {:?}\", res)),\n                    }\n                } else {\n                    let body = res\n                        .text()\n                        .await\n                        .map_err(|e: reqwest::Error| DownloadError::Other(e.into()))?;\n\n                    let resp: GCSObject = serde_json::from_str(&body)\n                        .map_err(|e: serde_json::Error| DownloadError::Other(e.into()))?;\n\n                    if !resp.size.is_some_and(|s| s == fs_size as i64) {\n                        // very unlikely\n                        return Err(anyhow::anyhow!(\n                            \"Boundary string from 'multipart/related' HTTP upload occurred in payload\"\n                        ));\n                    };\n\n                    Ok(())\n                }\n            }\n            Err(reqw) => Err(reqw.into()),\n        }\n    }\n\n    async fn delete_oids(\n        &self,\n        delete_objects: &[String],\n        cancel: &CancellationToken,\n        _permit: &tokio::sync::SemaphorePermit<'_>,\n    ) -> anyhow::Result<()> {\n        let kind = RequestKind::Delete;\n        let mut cancel = std::pin::pin!(cancel.cancelled());","sourceCodeStart":394,"sourceCodeEnd":430,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/libs/remote_storage/src/gcs_bucket.rs#L394-L430","documentation":"After an HTTP-successful multipart upload the backend verifies the size GCS reports for the stored object equals the number of bytes it declared it would stream (fs_size). A mismatch trips this integrity check; the code attributes it to the random multipart boundary string occurring inside the payload, which would make the server split the body at the wrong offset and store wrong bytes. Marked 'very unlikely' — but any size disagreement (fs_size computed wrong up front, short/truncated stream) also lands here.","triggerScenarios":"Uploading a payload that happens to contain the generated multipart boundary string; or passing an fs_size that disagrees with the actual byte count of the stream (bug in size accounting, short read from the source stream, concurrent file truncation while uploading).","commonSituations":"Almost never a real boundary collision; more plausibly a caller streaming a file that changed size between stat and upload, or a computed size off by framing overhead.","solutions":["Retry the upload — a new attempt generates a new boundary and re-transfers the bytes","Verify the fs_size passed to upload matches the exact byte count the stream will produce (stat the file right before streaming, or compute from the source of truth)","Hold the source file immutable (no truncation/append) between size computation and upload completion","If genuinely reproducible, switch to a resumable upload session to avoid multipart framing entirely"],"exampleFix":"// before: size computed once, possibly stale by upload time\nlet fs_size = file.metadata().await?.len();\nstorage.upload(&data, fs_size, &cancel).await?;\n\n// after: freeze the byte range being uploaded (or re-stat under a lock)\nlet fs_size = {\n    let f = File::open(&path).await?;\n    let meta = f.metadata().await?;\n    // upload from an open handle so size and content cannot diverge\n    f.into_std().await.into()\n};","handlingStrategy":"retry","validationCode":"// Ensure the declared size matches the stream before uploading.\nfn assert_size_matches(source: &std::fs::File, declared_len: u64) -> anyhow::Result<()> {\n    let actual = source.metadata()?.len();\n    anyhow::ensure!(actual == declared_len,\n        \"size mismatch: declared {declared_len}, stream holds {actual}\");\n    Ok(())\n}","typeGuard":null,"tryCatchPattern":"// Integrity-check failure: retry uploads (new boundary each attempt), escalate if persistent.\nfor attempt in 1..=3 {\n    match storage.upload(&data, fs_size, &cancel).await {\n        Ok(()) => break,\n        Err(e) if format!(\"{e:#}\").contains(\"Boundary string\") && attempt < 3 => {\n            tracing::warn!(\"upload integrity check tripped (attempt {attempt}); retrying\");\n            tokio::time::sleep(Duration::from_millis(250)).await;\n        }\n        Err(e) => return Err(e),\n    }\n}","preventionTips":["Compute fs_size from the same open handle that streams the bytes, never from a separately-stat'ed path","Keep source files immutable during upload (no truncate/append between size check and transfer)","Never ignore this error — it guards against silently storing corrupted data"],"tags":["gcs","google-cloud-storage","upload","data-integrity","multipart"],"backgroundTag":"upload-size-mismatch","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}