jdx/mise · error

remote cache blob pack returned a duplicate digest

Error message

remote cache blob pack returned a duplicate digest

What it means

While decoding a blob pack, mise-cache-core keeps a `seen` set of digests already streamed; each digest may appear at most once per pack. A second occurrence of the same (algorithm, hash, size) triple aborts decoding. The guard exists because the local CAS path for a digest is unique, and the pack's response metadata stats would double-count a duplicate. The client already deduplicates its request chunks (see `blob_pack_chunk`), so a duplicate can only originate server-side.

Source

Thrown at crates/mise-cache-core/src/lib.rs:933

                BlobPackHasher::Blake3(Box::new(blake3::Hasher::new())),
            ),
            2 => ("sha256", BlobPackHasher::Sha256(sha2::Sha256::new())),
            _ => bail!("remote cache blob pack has an invalid digest algorithm"),
        };
        let mut hash = [0_u8; 32];
        reader.read_exact(&mut hash).await?;
        let mut size = [0_u8; 8];
        reader.read_exact(&mut size).await?;
        let digest = CacheDigest {
            algorithm: algorithm.into(),
            hash: hex::encode(hash),
            size: u64::from_be_bytes(size),
        };
        if !requested.contains(&digest) {
            bail!("remote cache blob pack returned an unrequested digest");
        }
        if !seen.insert(digest.clone()) {
            bail!("remote cache blob pack returned a duplicate digest");
        }
        framed_bytes = framed_bytes
            .checked_add(BLOB_PACK_HEADER_BYTES)
            .and_then(|bytes| bytes.checked_add(digest.size))
            .ok_or_else(|| eyre!("remote cache blob pack is too large"))?;
        payload_bytes = payload_bytes
            .checked_add(digest.size)
            .ok_or_else(|| eyre!("remote cache blob pack payload is too large"))?;

        let path = directory.path().join(blobs.len().to_string());
        let mut output = tokio::fs::File::create(&path).await?;
        let mut remaining = digest.size;
        let mut buffer = [0_u8; 64 * 1024];
        while remaining > 0 {
            let limit = usize::try_from(remaining.min(buffer.len() as u64)).unwrap();
            let count = reader.read(&mut buffer[..limit]).await?;
            if count == 0 {
                bail!("remote cache blob pack ended before a blob was complete");

View on GitHub (pinned to 6f52dcdf99)

Solutions

  1. Fix the server to deduplicate the digest list before writing the pack (collect into a BTreeSet/HashSet keyed by the full digest).
  2. Reproduce with a two-digest request where both map to overlapping chunks and hexdump the response to confirm which digest repeats.
  3. Check that server-side chunk merging unions chunks without re-emitting shared blobs.
  4. If the server cannot be fixed quickly, make the pack endpoint return 404/405/501 so the client degrades to per-blob downloads.
Defensive patterns

Strategy: try-catch

Try / catch

let blobs = match remote_cache.download(staging_dir).await {
    Ok(pack) => pack.blobs,
    Err(err) if err.to_string().contains("blob pack returned a duplicate") => {
        warn!("server sent a duplicate digest: {err}");
        Vec::new() // fall back to per-blob GET / local recompute
    }
    Err(err) => return Err(err),
};

Prevention

When it happens

Trigger: A blob pack response that streams the same digest twice: a server that concatenates request chunks without deduplicating, emits a shared blob once per chunk it belongs to, or merges several concurrent requests into one pack. Triggered during any remote cache blob-pack download when the server's pack writer does not dedupe.

Common situations: Custom cache servers that fan out multi-chunk requests and merge naively; server bugs where a blob satisfying several requested digests is emitted repeatedly; version skew in pack assembly logic.

Related errors

AI-assisted analysis of jdx/mise@6f52dcdf99 (2026-08-22). Data as JSON: /api/errors/9180f0683c69d4c8. Report an issue: GitHub.