What causes "Failed to parse JSON: {e}" in nautechsystems/nautilus_trader?

Sanitize custom data fields so they contain no NaN/Infinity values before deserialization.

Failed to parse JSON: {e}

Error message

Failed to parse JSON: {e}

What it means

py_json_deserialize_custom_data converts the serialized JSON string into a Python dict via json.loads. If Python's json.loads rejects the string (should be rare since it was produced by serde_json, but possible with non-string keys/edge values or exotic floats), the error is wrapped as 'Failed to parse JSON: {e}'.

Source

Thrown at crates/model/src/python/data/mod.rs:360

/// Deserializes JSON value to `CustomData` via the data class's `from_json`.
#[cfg(feature = "python")]
fn py_json_deserialize_custom_data(
    data_class: &pyo3::Py<pyo3::PyAny>,
    value: &serde_json::Value,
) -> Result<std::sync::Arc<dyn crate::data::CustomDataTrait>, anyhow::Error> {
    use std::sync::Arc;

    use crate::data::PythonCustomDataWrapper;

    pyo3::Python::attach(|py| {
        let json_str = serde_json::to_string(&value)?;
        let json_module = py
            .import("json")
            .map_err(|e| anyhow::anyhow!("Failed to import json: {e}"))?;
        let py_dict = json_module
            .call_method1("loads", (json_str,))
            .map_err(|e| anyhow::anyhow!("Failed to parse JSON: {e}"))?;

        let instance = data_class
            .bind(py)
            .call_method1("from_json", (py_dict,))
            .map_err(|e| anyhow::anyhow!("Failed to call from_json: {e}"))?;

        let wrapper = PythonCustomDataWrapper::new(py, &instance)
            .map_err(|e| anyhow::anyhow!("Failed to create wrapper: {e}"))?;

        Ok(Arc::new(wrapper) as Arc<dyn crate::data::CustomDataTrait>)
    })
}

/// Encodes `CustomData` items to `RecordBatch` via Python `encode_record_batch_py`.
#[allow(unsafe_code)]
#[cfg(all(feature = "python", feature = "arrow"))]
fn py_encode_custom_data_to_record_batch(
    items: &[std::sync::Arc<dyn crate::data::CustomDataTrait>],

View on GitHub (pinned to 18893faf8b)

Solutions

Sanitize custom data fields so they contain no NaN/Infinity values before deserialization.
Implement the custom data class's from_json to accept the dict and validate required fields.
Inspect the serialized string (serde_json::to_string(&value)) to confirm it is valid strict JSON.
Round-trip test the custom data type: serialize then deserialize in a unit test to catch schema drift.

Example fix

// before: NaN breaks Python json.loads
#[derive(Serialize)]
struct MyData { ratio: f64 } // ratio = f64::NAN
// after: sanitize before serialization
#[derive(Serialize)]
struct MyData { ratio: f64 }
impl MyData {
    fn sanitized(mut self) -> Self {
        if !self.ratio.is_finite() { self.ratio = 0.0; }
        self
    }
}

Defensive patterns

Strategy: validation

Validate before calling

// ensure all float fields are finite before deserialize path
fn assert_finite(value: &serde_json::Value) -> bool {
    match value {
        serde_json::Value::Number(n) => n.as_f64().map(|f| f.is_finite()).unwrap_or(true),
        serde_json::Value::Array(a) => a.iter().all(assert_finite),
        serde_json::Value::Object(o) => o.values().all(assert_finite),
        _ => true,
    }
}

Try / catch

match py_json_deserialize_custom_data(data_class, value) {
    Ok(v) => v,
    Err(e) if e.to_string().contains("Failed to parse JSON") => {
        log::error!("payload not strict JSON: {}", serde_json::to_string(&value).unwrap_or_default());
        Err(e)
    }
    Err(e) => return Err(e),
}

Prevention

Keep NaN/Infinity out of custom data fields.
Round-trip test each custom data type (serialize -> deserialize) in CI.
Avoid deeply nested payloads that hit Python recursion limits.

When it happens

Trigger: Deserializing a custom data value whose serde JSON output Python's json.loads cannot parse — e.g. NaN/Infinity floats serialized by serde (invalid strict JSON for Python's parser), or a corrupted/too-deep payload.

Common situations: Custom data structs containing f64::NAN or INFINITY fields; extremely nested custom payloads hitting recursion limits; mismatched serde feature flags producing non-standard JSON.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/fb45677aae758ce2. Report an issue: GitHub.