nautechsystems/nautilus_trader · error

canonical backtest result bytes do not use the canonical enc

Error message

canonical backtest result bytes do not use the canonical encoding

What it means

After semantic canonicalization checks pass, `from_slice` re-serializes the normalized document and requires the resulting bytes to be byte-identical to the input. This is the strictest check: any difference in whitespace, key order, or number/text encoding from the canonical encoder fails with this message. It guarantees hashes/deduplication of result bytes are stable.

Source

Thrown at crates/backtest/src/result.rs:198

    ///
    /// Inner records retain their NautilusTrader model serialization and are compared as semantic
    /// content. Producers must use the version 1 writer rather than construct records independently.
    ///
    /// # Errors
    ///
    /// Returns an error if the bytes are not a canonical version 1 result.
    pub fn from_slice(bytes: &[u8]) -> anyhow::Result<Self> {
        let document: Value = serde_json::from_slice(bytes)
            .map_err(|e| anyhow::anyhow!("invalid canonical backtest result JSON: {e}"))?;
        validate_document(&document)?;
        let mut normalized = document.clone();
        canonicalize_document(&mut normalized)?;
        anyhow::ensure!(
            normalized == document,
            "canonical backtest result violates the version 1 encoding rules"
        );
        let canonical = serde_json::to_vec(&normalized)?;
        anyhow::ensure!(
            canonical == bytes,
            "canonical backtest result bytes do not use the canonical encoding"
        );
        Ok(Self { document })
    }

    /// Returns the canonical compact UTF-8 JSON bytes without trailing data.
    ///
    /// # Errors
    ///
    /// Returns an error if the in-memory document cannot be serialized.
    pub fn to_bytes(&self) -> anyhow::Result<Vec<u8>> {
        Ok(serde_json::to_vec(&self.document)?)
    }

    /// Returns `blake3:` followed by the 32-byte BLAKE3 digest as 64 lowercase hex digits.
    ///
    /// # Errors

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Use the exact bytes emitted by the canonical writer (do not pretty-print, reformat, or re-key the JSON)
  2. If you only have a reformatted copy, parse it, canonicalize via the library path (from_state or the canonical writer), and write canonical bytes first
  3. Compare `serde_json::to_vec(&normalized)` with your bytes to locate the encoding difference
  4. Validate with `jq -c .` that round-tripping to compact form matches — if it still differs, the issue is number/string encoding

Example fix

// before
let pretty = serde_json::to_vec_pretty(&result)?;
BacktestResult::from_slice(&pretty)?;
// after
let canonical = serde_json::to_vec(&result)?; // compact, canonical ordering
BacktestResult::from_slice(&canonical)?;
Defensive patterns

Strategy: validation

Validate before calling

import json
def roundtrip_equal(path):
    raw = open(path, 'rb').read()
    return json.dumps(json.loads(raw), separators=(',', ':'), ensure_ascii=False).encode() == raw

Try / catch

match BacktestResult::from_slice(&bytes) {
    Err(e) if e.to_string().contains("bytes do not use the canonical encoding") => {
        eprintln!("re-encode canonically before loading");
    }
    Err(e) => return Err(e),
    Ok(r) => r,
}

Prevention

When it happens

Trigger: Calling `from_slice` with pretty-printed JSON, JSON with different key ordering, alternate whitespace, differently formatted numbers, or escaped-vs-unescaped unicode — anything not produced by the library's canonical `serde_json::to_vec` of the normalized document.

Common situations: Pretty-printing a result file for readability then trying to load it back; reserializing with `jq` or Python's json (different spacing/key order); concatenating or trimming bytes.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/474e2c710681e9ab. Report an issue: GitHub.