What causes "Failed to deserialize json `payload`: {e}" in nautechsystems/nautilus_trader?

Verify the SerializationEncoding passed matches the encoding used when the data was written (flush/rewrite the cache if the encoding was changed).

nautechsystems/nautilus_trader · error

Failed to deserialize json `payload`: {e}

Error message

Failed to deserialize json `payload`: {e}

What it means

deserialize_payload converts raw Redis bytes into a typed value. When the JSON branch (SerializationEncoding::Json) is selected, serde_json::from_slice fails to parse the stored bytes as JSON (or they do not fit T), so the library wraps the serde error in an anyhow error. This indicates the payload stored in Redis does not match the expected encoding or schema.

Source

Thrown at crates/infrastructure/src/redis/queries.rs:117

                anyhow::bail!("Cap'n Proto encoding is not supported for Redis cache payloads")
            }
        }
    }

    /// Deserializes the given byte slice `payload` into type `T` using the specified `encoding`.
    ///
    /// # Errors
    ///
    /// Returns an error if deserialization from the chosen encoding fails or converting to the target type fails.
    pub fn deserialize_payload<T: DeserializeOwned>(
        encoding: SerializationEncoding,
        payload: &[u8],
    ) -> anyhow::Result<T> {
        let mut value = match encoding {
            SerializationEncoding::MsgPack => rmp_serde::from_slice(payload)
                .map_err(|e| anyhow::anyhow!("Failed to deserialize msgpack `payload`: {e}"))?,
            SerializationEncoding::Json => serde_json::from_slice(payload)
                .map_err(|e| anyhow::anyhow!("Failed to deserialize json `payload`: {e}"))?,
            SerializationEncoding::Sbe => {
                anyhow::bail!("SBE encoding is not supported for Redis cache payloads")
            }
            SerializationEncoding::Capnp => {
                anyhow::bail!("Cap'n Proto encoding is not supported for Redis cache payloads")
            }
        };

        convert_timestamp_strings(&mut value);

        serde_json::from_value(value)
            .map_err(|e| anyhow::anyhow!("Failed to convert value to target type: {e}"))
    }

    /// Scans Redis for keys matching the given `pattern`.
    ///
    /// # Errors
    ///

View on GitHub (pinned to 18893faf8b)

Solutions

Verify the SerializationEncoding passed matches the encoding used when the data was written (flush/rewrite the cache if the encoding was changed).
Inspect the offending Redis value with redis-cli GET/<key> and validate it as JSON.
Check whether the target struct changed between versions and re-snapshot the cache from a fresh backtest/live run.
Ensure the Redis index keys point at the correct value keys (no stale index entries).

Example fix

// before
let closes: Vec<InstrumentClose> = RedisCacheDatabase::load_all(con, "trader", SerializationEncoding::Json).await?;
// after: match the encoding used at write time (e.g. MsgPack)
let closes: Vec<InstrumentClose> = RedisCacheDatabase::load_all(con, "trader", SerializationEncoding::MsgPack).await?;

Defensive patterns

Strategy: try-catch

Validate before calling

// Rust: verify payload is JSON before loading
fn looks_like_json(bytes: &[u8]) -> bool {
    serde_json::from_slice::<serde_json::Value>(bytes).is_ok()
}

Type guard

fn is_json_encoding(encoding: &SerializationEncoding) -> bool {
    matches!(encoding, SerializationEncoding::Json)
}

Try / catch

match RedisCacheDatabase::load_all(&mut con, trader_key, encoding).await {
    Ok(cache) => cache,
    Err(e) => {
        if e.to_string().contains("Failed to deserialize") {
            // fall back: reload with alternate encoding or re-snapshot cache
        }
        return Err(e);
    }
}

Prevention

Pin one SerializationEncoding in config and use it for both write and read paths.
Flush and re-snapshot the Redis cache whenever changing encoding or upgrading versions.
Validate stored payloads parse as JSON before running load_all in production.

When it happens

Trigger: Calling deserialize_payload (directly or via load_all / load_instruments / load_instrument_closes / load_orders / load_positions / etc.) with encoding=SerializationEncoding::Json while the bytes in Redis are MsgPack-encoded, truncated, corrupted, or were written by an older schema version.

Common situations: Encoding was switched between MsgPack and Json after data was written; a manually inserted/corrupted Redis key; a version upgrade changed struct fields so old JSON no longer deserializes; a bulk read returned stale/wrong values due to key-index drift.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/d8491e489e2d55fd. Report an issue: GitHub.