{"record":{"id":"9180f0683c69d4c8","repo":"jdx/mise","slug":"remote-cache-blob-pack-returned-a-duplicate-digest","errorCode":null,"errorMessage":"remote cache blob pack returned a duplicate digest","messagePattern":"remote cache blob pack returned a duplicate digest","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/mise-cache-core/src/lib.rs","lineNumber":933,"sourceCode":"                BlobPackHasher::Blake3(Box::new(blake3::Hasher::new())),\n            ),\n            2 => (\"sha256\", BlobPackHasher::Sha256(sha2::Sha256::new())),\n            _ => bail!(\"remote cache blob pack has an invalid digest algorithm\"),\n        };\n        let mut hash = [0_u8; 32];\n        reader.read_exact(&mut hash).await?;\n        let mut size = [0_u8; 8];\n        reader.read_exact(&mut size).await?;\n        let digest = CacheDigest {\n            algorithm: algorithm.into(),\n            hash: hex::encode(hash),\n            size: u64::from_be_bytes(size),\n        };\n        if !requested.contains(&digest) {\n            bail!(\"remote cache blob pack returned an unrequested digest\");\n        }\n        if !seen.insert(digest.clone()) {\n            bail!(\"remote cache blob pack returned a duplicate digest\");\n        }\n        framed_bytes = framed_bytes\n            .checked_add(BLOB_PACK_HEADER_BYTES)\n            .and_then(|bytes| bytes.checked_add(digest.size))\n            .ok_or_else(|| eyre!(\"remote cache blob pack is too large\"))?;\n        payload_bytes = payload_bytes\n            .checked_add(digest.size)\n            .ok_or_else(|| eyre!(\"remote cache blob pack payload is too large\"))?;\n\n        let path = directory.path().join(blobs.len().to_string());\n        let mut output = tokio::fs::File::create(&path).await?;\n        let mut remaining = digest.size;\n        let mut buffer = [0_u8; 64 * 1024];\n        while remaining > 0 {\n            let limit = usize::try_from(remaining.min(buffer.len() as u64)).unwrap();\n            let count = reader.read(&mut buffer[..limit]).await?;\n            if count == 0 {\n                bail!(\"remote cache blob pack ended before a blob was complete\");","sourceCodeStart":915,"sourceCodeEnd":951,"githubUrl":"https://github.com/jdx/mise/blob/6f52dcdf99e282ef7a7db68c81301fa4618d0f79/crates/mise-cache-core/src/lib.rs#L915-L951","documentation":"While decoding a blob pack, mise-cache-core keeps a `seen` set of digests already streamed; each digest may appear at most once per pack. A second occurrence of the same (algorithm, hash, size) triple aborts decoding. The guard exists because the local CAS path for a digest is unique, and the pack's response metadata stats would double-count a duplicate. The client already deduplicates its request chunks (see `blob_pack_chunk`), so a duplicate can only originate server-side.","triggerScenarios":"A blob pack response that streams the same digest twice: a server that concatenates request chunks without deduplicating, emits a shared blob once per chunk it belongs to, or merges several concurrent requests into one pack. Triggered during any remote cache blob-pack download when the server's pack writer does not dedupe.","commonSituations":"Custom cache servers that fan out multi-chunk requests and merge naively; server bugs where a blob satisfying several requested digests is emitted repeatedly; version skew in pack assembly logic.","solutions":["Fix the server to deduplicate the digest list before writing the pack (collect into a BTreeSet/HashSet keyed by the full digest).","Reproduce with a two-digest request where both map to overlapping chunks and hexdump the response to confirm which digest repeats.","Check that server-side chunk merging unions chunks without re-emitting shared blobs.","If the server cannot be fixed quickly, make the pack endpoint return 404/405/501 so the client degrades to per-blob downloads."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"let blobs = match remote_cache.download(staging_dir).await {\n    Ok(pack) => pack.blobs,\n    Err(err) if err.to_string().contains(\"blob pack returned a duplicate\") => {\n        warn!(\"server sent a duplicate digest: {err}\");\n        Vec::new() // fall back to per-blob GET / local recompute\n    }\n    Err(err) => return Err(err),\n};","preventionTips":["Run a pack-writer test with overlapping chunks on any custom cache server.","Deduplicate the request digest list server-side before assembling the pack."],"tags":["remote-cache","blob-pack","duplicate-entries","protocol-violation","rust"],"backgroundTag":"server-protocol-violation","analyzedSha":"6f52dcdf99e282ef7a7db68c81301fa4618d0f79","analyzedAt":"2026-08-22T10:14:23.840Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}