What causes "CustomData JSON missing 'payload' field" in nautechsystems/nautilus_trader?

Serialize with `CustomData`'s own serializer so the `payload` envelope field is written; never hand-build the JSON.

CustomData JSON missing 'payload' field

Error message

CustomData JSON missing 'payload' field

What it means

CustomData is serialized with an envelope that separates routing/metadata fields from the user payload under a reserved `"payload"` key. `parse_envelope_payload` extracts that key before handing it to the registered type deserializer; if it's missing, the envelope is malformed and this error is raised.

Source

Thrown at crates/model/src/data/registry.rs:127

            let p: Params = serde_json::from_value(m.clone()).ok()?;
            if p.is_empty() { None } else { Some(p) }
        }
    });
    let identifier = obj
        .get("identifier")
        .and_then(|v| v.as_str())
        .map(String::from);
    Some(DataType::new(type_name, metadata, identifier))
}

/// Parses the canonical `CustomData` JSON envelope `{ type, data_type, payload }` and returns
/// the payload value to pass to the registered type deserializer. Does not depend on
/// user payload field names.
fn parse_envelope_payload(value: &serde_json::Value) -> Result<serde_json::Value, anyhow::Error> {
    let payload = value
        .get("payload")
        .cloned()
        .ok_or_else(|| anyhow::anyhow!("CustomData JSON missing 'payload' field"))?;
    Ok(payload)
}

/// Looks up and runs the JSON deserializer for the given type name.
/// Returns `None` if the type is not registered.
///
/// # Errors
/// Returns an error if the deserializer fails.
pub fn deserialize_custom_from_json(
    type_name: &str,
    value: &serde_json::Value,
) -> Result<Option<Data>, anyhow::Error> {
    let reg = registries();
    let deserializer_ref = match reg.json.get(type_name) {
        Some(d) => d,
        None => return Ok(None),
    };
    let data_type = parse_data_type_from_value(value);

View on GitHub (pinned to 18893faf8b)

Solutions

Serialize with `CustomData`'s own serializer so the `payload` envelope field is written; never hand-build the JSON.
Check producer/consumer library versions match; re-serialize old records with the current format.
Inspect the JSON at hand: wrap the user data as `{ ..., "payload": <user data> }` if you must construct it manually.
Before deserializing, guard with `value.get("payload").is_some()` and log the raw JSON when absent.

Example fix

// before
let data = deserialize_custom_from_json(r#"{"price": 1.5}"#)?;

// after
let data = deserialize_custom_from_json(r#"{"type_name": "MyQuote", "payload": {"price": 1.5}}"#)?;

Defensive patterns

Strategy: validation

Validate before calling

fn has_payload(v: &serde_json::Value) -> bool {
    v.get("payload").is_some()
}

Type guard

fn payload_of(v: &serde_json::Value) -> Option<serde_json::Value> {
    v.get("payload").cloned()
}

Try / catch

match deserialize_custom_from_json(s) {
    Ok(d) => d,
    Err(e) if e.to_string().contains("payload") => {
        log::error!("malformed CustomData envelope (missing payload): {e}");
        return Err(e);
    }
    Err(e) => return Err(e),
}

Prevention

Always serialize CustomData through its own serializer so the payload envelope is present.
Keep producer and consumer library versions in sync regarding the envelope format.
When hand-crafting CustomData JSON (tests/configs), always wrap user data under "payload".

When it happens

Trigger: Calling `deserialize_custom_from_json` (which calls `parse_envelope_payload`) with JSON that was serialized without the envelope — e.g. raw user-type JSON, hand-written JSON, or output from a different/older serializer version.

Common situations: Version skew between the writer (old format without envelope) and reader; manually constructing CustomData JSON in tests or configs and forgetting the `payload` wrapper; a serialization bug on the producing side dropping the field.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/b2f5de50dd2c544e. Report an issue: GitHub.