What causes "Failed to deserialize msgpack `payload`: {e}" in nautechsystems/nautilus_trader?

Verify the encoding parameter matches the encoding used at write time (check the entry's `encoding` header).

Failed to deserialize msgpack `payload`: {e}

Error message

Failed to deserialize msgpack `payload`: {e}

What it means

deserialize_payload decodes raw bytes back into T using rmp_serde::from_slice (MsgPack) or serde_json::from_slice (Json). This error wraps rmp_serde decode failures: the bytes are not valid msgpack or do not match the shape of T. SBE and Capnp are explicitly rejected before this branch.

Source

Thrown at crates/infrastructure/src/redis/queries.rs:115

            }
            SerializationEncoding::Capnp => {
                anyhow::bail!("Cap'n Proto encoding is not supported for Redis cache payloads")
            }
        }
    }

    /// Deserializes the given byte slice `payload` into type `T` using the specified `encoding`.
    ///
    /// # Errors
    ///
    /// Returns an error if deserialization from the chosen encoding fails or converting to the target type fails.
    pub fn deserialize_payload<T: DeserializeOwned>(
        encoding: SerializationEncoding,
        payload: &[u8],
    ) -> anyhow::Result<T> {
        let mut value = match encoding {
            SerializationEncoding::MsgPack => rmp_serde::from_slice(payload)
                .map_err(|e| anyhow::anyhow!("Failed to deserialize msgpack `payload`: {e}"))?,
            SerializationEncoding::Json => serde_json::from_slice(payload)
                .map_err(|e| anyhow::anyhow!("Failed to deserialize json `payload`: {e}"))?,
            SerializationEncoding::Sbe => {
                anyhow::bail!("SBE encoding is not supported for Redis cache payloads")
            }
            SerializationEncoding::Capnp => {
                anyhow::bail!("Cap'n Proto encoding is not supported for Redis cache payloads")
            }
        };

        convert_timestamp_strings(&mut value);

        serde_json::from_value(value)
            .map_err(|e| anyhow::anyhow!("Failed to convert value to target type: {e}"))
    }

    /// Scans Redis for keys matching the given `pattern`.
    ///

View on GitHub (pinned to 18893faf8b)

Solutions

Verify the encoding parameter matches the encoding used at write time (check the entry's `encoding` header).
Ensure the target type T matches the struct that was serialized (same crate version on reader and writer).
Inspect the raw bytes with rmp_serde::from_slice::<rmpv::Value> to diagnose the mismatch.
Re-write the affected cache entries after aligning versions.

Example fix

// before: mismatched encoding
let v: MyType = deserialize_payload(SerializationEncoding::MsgPack, &bytes)?;
// after: use the encoding recorded when the payload was written
let v: MyType = deserialize_payload(encoding_from_header, &bytes)?;

Defensive patterns

Strategy: try-catch

Validate before calling

// confirm bytes are msgpack before typed decode
let probe: Result<rmpv::Value, _> = rmp_serde::from_slice(bytes);
if probe.is_err() { return Err("payload is not valid msgpack".into()); }

Try / catch

match deserialize_payload::<MyType>(encoding, bytes) {
    Ok(v) => use(v),
    Err(e) if e.to_string().contains("Failed to deserialize") => {
        log::error!("payload shape/encoding mismatch: {e}");
        // fall back to default or re-fetch after version alignment
    }
    Err(e) => return Err(e),
}

Prevention

Persist and honor the encoding header used at write time.
Keep reader and writer payload structs (and crate versions) in sync.
Guard against SBE/Capnp encodings, which are unsupported for Redis cache payloads.

When it happens

Trigger: Calling deserialize_payload::<T>(SerializationEncoding::MsgPack, bytes) where bytes were not produced by serialize_payload with MsgPack, were truncated/corrupted, or encode a different type's shape than T.

Common situations: Reading Redis cache entries written by a different app version whose payload struct changed fields; decoding a JSON-encoded payload with the MsgPack encoding parameter; corrupted/truncated bytes fetched from Redis.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/947c227a0f365c07. Report an issue: GitHub.