What causes "Failed to serialize msgpack `payload`: {e}" in nautechsystems/nautilus_trader?

Sanitize the payload: replace or reject NaN/Infinity floats before calling serialize_payload.

Failed to serialize msgpack `payload`: {e}

Error message

Failed to serialize msgpack `payload`: {e}

What it means

serialize_payload in the Redis queries module converts a payload to serde_json::Value, rewrites timestamps, then encodes as MessagePack via rmp_serde::to_vec. This error wraps any rmp_serde serialization failure — typically values not representable in msgpack after the JSON round-trip.

Source

Thrown at crates/infrastructure/src/redis/queries.rs:87

#[derive(Debug)]
pub struct DatabaseQueries;

impl DatabaseQueries {
    /// Serializes the given `payload` using the specified `encoding` to a byte vector.
    ///
    /// # Errors
    ///
    /// Returns an error if serialization to the chosen encoding fails.
    pub fn serialize_payload<T: Serialize>(
        encoding: SerializationEncoding,
        payload: &T,
    ) -> anyhow::Result<Vec<u8>> {
        match encoding {
            SerializationEncoding::MsgPack => {
                let mut value = serde_json::to_value(payload)?;
                convert_timestamps(&mut value);
                rmp_serde::to_vec(&value)
                    .map_err(|e| anyhow::anyhow!("Failed to serialize msgpack `payload`: {e}"))
            }
            SerializationEncoding::Json => {
                let mut value = serde_json::to_value(payload)?;
                convert_timestamps(&mut value);
                serde_json::to_vec(&value)
                    .map_err(|e| anyhow::anyhow!("Failed to serialize json `payload`: {e}"))
            }
            SerializationEncoding::Sbe => {
                anyhow::bail!("SBE encoding is not supported for Redis cache payloads")
            }
            SerializationEncoding::Capnp => {
                anyhow::bail!("Cap'n Proto encoding is not supported for Redis cache payloads")
            }
        }
    }

    /// Deserializes the given byte slice `payload` into type `T` using the specified `encoding`.
    ///

View on GitHub (pinned to 18893faf8b)

Solutions

Sanitize the payload: replace or reject NaN/Infinity floats before calling serialize_payload.
Ensure all payload map keys serialize as strings (HashMap<serde-friendly key, _>).
Fall back to SerializationEncoding::Json to see if the failure is msgpack-specific.
Check whether convert_timestamps corrupted the value shape for your custom types.

Example fix

// before
let payload = Payload { price: f64::NAN };
serialize_payload(SerializationEncoding::MsgPack, &payload)?;
// after
assert!(payload.price.is_finite());
serialize_payload(SerializationEncoding::MsgPack, &payload)?;

Defensive patterns

Strategy: try-catch

Validate before calling

// reject non-finite floats before serialization
fn payload_is_msgpack_safe(v: &serde_json::Value) -> bool {
    match v {
        serde_json::Value::Number(n) => n.as_f64().map(|f| f.is_finite()).unwrap_or(true),
        serde_json::Value::Array(a) => a.iter().all(payload_is_msgpack_safe),
        serde_json::Value::Object(o) => o.values().all(payload_is_msgpack_safe),
        _ => true,
    }
}

Try / catch

match serialize_payload(SerializationEncoding::MsgPack, &payload) {
    Ok(bytes) => write_to_redis(bytes),
    Err(e) if e.to_string().contains("Failed to serialize msgpack") => {
        log::error!("payload not msgpack-encodable: {e}; falling back to json");
        let bytes = serialize_payload(SerializationEncoding::Json, &payload)?;
    }
    Err(e) => return Err(e),
}

Prevention

Keep payload types free of NaN/Infinity floats.
Use string-keyed maps in serializable payload structs.
Unit-test serialize_payload round-trips for every payload type.

When it happens

Trigger: Calling serialize_payload(SerializationEncoding::MsgPack, payload) where the value cannot be encoded by rmp_serde (e.g. non-string map keys introduced by the JSON value conversion, f64 NaN/Infinity payload fields).

Common situations: Payloads containing NaN/Infinity floats (invalid in JSON and unsupported in msgpack), map types with non-string keys, or deeply custom serde impls that produce structures msgpack cannot encode after convert_timestamps mutates the tree.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/917807ff5a29887c. Report an issue: GitHub.