astrid-runtime/astrid · error

retained bytes exceed logical bytes

Error message

retained bytes exceed logical bytes

What it means

deduplication computes saved bytes as logical_bytes minus retained_bytes; it bails if retained bytes exceed logical bytes, which is mathematically impossible for a correct corpus measurement. This is a sanity invariant protecting the Deduplication result.

Solutions

  1. Audit how logical_bytes and retained_bytes are accumulated so every retained byte is counted exactly once within the same logical corpus.
  2. Recompute metrics from the complete corpus in a single pass.
  3. Check that no representation records were added from a different/overlapping corpus.

Example fix

// before
// logical bytes from 3 of 4 files, retained bytes from all 4
// after
// visit all inputs in one pass so logical and retained totals cover the same corpus
Defensive patterns

Strategy: validation

Validate before calling

if retained_bytes > logical_bytes { return Err(anyhow!("retained {} > logical {}", retained_bytes, logical_bytes)); }

Try / catch

match summary() {
    Err(e) if e.to_string().contains("retained bytes exceed") => recompute_from_full_corpus(),
    other => other,
}

Prevention

When it happens

Trigger: Calling the Measurements summary with retained_bytes > logical_bytes — e.g. summing retained representations across overlapping records double-counted, or logical byte totals undercounted because some files were skipped.

Common situations: Measuring a subset of the corpus but summing retained bytes over the full set; mixing metrics from multiple runs; counting compressed/replicated bytes as both retained and additional logical bytes.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of astrid-runtime/astrid@affd8760f4 (2026-09-09). Data as JSON: /api/errors/1fefa9de69dd083f. Report an issue: GitHub.

Appendix: source

Thrown at crates/astrid-storage-chunker-evidence/src/metrics.rs:183

    let length = u64::try_from(bytes.len())?;
    let identity = *blake3::hash(bytes).as_bytes();
    match identities.entry(identity) {
        std::collections::hash_map::Entry::Vacant(entry) => {
            entry.insert(length);
            *unique_bytes = checked_add(*unique_bytes, length, "unique byte count")?;
        },
        std::collections::hash_map::Entry::Occupied(entry) => {
            if *entry.get() != length {
                bail!("BLAKE3 evidence collision with inconsistent lengths");
            }
        },
    }
    Ok(())
}

fn deduplication(logical_bytes: u64, retained_bytes: u64) -> Result<Deduplication> {
    if retained_bytes > logical_bytes {
        bail!("retained bytes exceed logical bytes");
    }
    let saved_bytes = logical_bytes
        .checked_sub(retained_bytes)
        .ok_or_else(|| anyhow::anyhow!("retained bytes exceed logical bytes"))?;
    Ok(Deduplication {
        retained_bytes,
        saved_bytes,
        retained_basis_points: basis_points(retained_bytes, logical_bytes)?,
        saved_basis_points: basis_points(saved_bytes, logical_bytes)?,
    })
}

/// Returns the ratio in basis points, rounded down.
///
/// Retained and saved ratios are deliberately calculated independently with
/// this same rule. Deriving one as the complement of the other would round one
/// side up whenever the exact ratio is fractional.
fn basis_points(part: u64, total: u64) -> Result<u64> {

View on GitHub (pinned to affd8760f4)