nautechsystems/nautilus_trader · error · anyhow::Error

Failed to serialize data_type for persistence: {e}

Error message

Failed to serialize data_type for persistence: {e}

What it means

Thrown when a custom data type's to_persistence_json() fails while preparing a batch for writing via write_custom_data_batch. The data_type JSON is stored alongside the Arrow batch as schema metadata so the type can be reconstructed on read; if the type cannot serialize itself, persistence cannot proceed and the underlying serialization error is wrapped.

Source

Thrown at crates/persistence/src/backend/custom.rs:145

/// and returns type identity and timestamp range so the catalog can build path and perform I/O.
///
/// # Errors
///
/// Returns an error if encoding or augmentation fails, or if the type is not registered.
pub fn prepare_custom_data_batch(
    data: Vec<CustomData>,
) -> anyhow::Result<(RecordBatch, String, Option<String>, UnixNanos, UnixNanos)> {
    let Some(first_custom) = data.first() else {
        anyhow::bail!("prepare_custom_data_batch called with empty data");
    };

    let type_name = first_custom.data.type_name();
    let identifier = first_custom.data_type.identifier().map(String::from);
    let dt_meta = first_custom.data_type.metadata_string_map();
    let data_type_json = first_custom
        .data_type
        .to_persistence_json()
        .map_err(|e| anyhow::anyhow!("Failed to serialize data_type for persistence: {e}"))?;

    let start_ts = first_custom.data.ts_init();
    let end_ts = data.last().map_or(start_ts, |custom| custom.data.ts_init());
    let items: Vec<Arc<dyn CustomDataTrait>> =
        data.into_iter().map(|c| Arc::clone(&c.data)).collect();

    let batch = encode_custom_to_arrow(type_name, &items)
        .map_err(|e| anyhow::anyhow!("Failed to encode custom data to Arrow: {e}"))?
        .ok_or_else(|| {
            anyhow::anyhow!(
                "Custom data type \"{type_name}\" is not registered for Arrow encoding; \
                 call register_custom_data_class or ensure_custom_data_registered before writing"
            )
        })?;
    let batch =
        augment_batch_with_data_type_column(&batch, &data_type_json, type_name, dt_meta.as_ref())?;

    Ok((batch, type_name.to_string(), identifier, start_ts, end_ts))

View on GitHub (pinned to 18893faf8b)

Solutions

1. Inspect the wrapped {e} message to see which field or check inside to_persistence_json failed.
2. Fix the custom DataType definition so all fields are JSON-serializable and satisfy the persistence schema requirements.
3. If the definition changed, migrate old definitions or implement compatibility in to_persistence_json.
4. Add a unit test calling to_persistence_json before writing to the catalog.

Example fix

// before
fn to_persistence_json(&self) -> anyhow::Result<serde_json::Value> {
    Ok(serde_json::to_value(&self.non_serializable_field)?)
}
// after
fn to_persistence_json(&self) -> anyhow::Result<serde_json::Value> {
    Ok(serde_json::json!({ "schema": self.schema_json()? }))
}

let json = data_type.to_persistence_json().expect("data_type must serialize before write");

match first_custom.data_type.to_persistence_json() {
    Ok(json) => proceed(json),
    Err(e) => eprintln!("fix to_persistence_json for {}: {e}", first_custom.data.type_name()),
}

When it happens

Trigger: Writing custom data whose DataType implementation's to_persistence_json returns an error — e.g. a field that cannot be represented in JSON, an inconsistent/missing persistence definition, or a nested type that fails its own serialization.

Common situations: Custom serde-incompatible fields (non-string map keys, unserializable enums) in a user-defined DataType; version drift where the stored definition schema changed and validation inside to_persistence_json now rejects it.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• Instances must have encode_record_batch_py method
• prepare_custom_data_batch called with empty data
• Unknown data type: {type_name}
• Unknown data type: {type_name}; custom decode only allowed i
• Failed to encode custom data to Arrow: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/34167a56582fc854. Report an issue: GitHub.