nautechsystems/nautilus_trader · error

{e}

Error message

{e}

What it means

In ensure_custom_data_registered, after downcasting arrays to the concrete type, the encoder calls EncodeToRecordBatch::encode_batch; any error returned by the type's own encode implementation is wrapped verbatim into an anyhow error with this message. The underlying cause is whatever the custom type's encode_batch reported (e.g. mismatched metadata, wrong number of columns).

Source

Thrown at crates/serialization/src/arrow/custom.rs:115

    let schema = Arc::new(T::get_schema(None));

    let encoder: ArrowEncoder = Box::new(|items: &[Arc<dyn CustomDataTrait>]| {
        let typed: Result<Vec<T>, _> = items
            .iter()
            .map(|b| {
                b.as_any()
                    .downcast_ref::<T>()
                    .cloned()
                    .ok_or_else(|| anyhow::anyhow!("Expected {}", T::type_name_static()))
            })
            .collect();
        let typed = typed?;
        let metadata = typed
            .first()
            .map(EncodeToRecordBatch::metadata)
            .unwrap_or_default();
        EncodeToRecordBatch::encode_batch(&metadata, &typed).map_err(|e| anyhow::anyhow!("{e}"))
    });

    let decoder: ArrowDecoder = Box::new(|metadata, batch| {
        T::decode_data_batch(metadata, batch).map_err(|e| anyhow::anyhow!("{e}"))
    });

    let _ = ensure_arrow_registered(type_name, schema, encoder, decoder);
}

/// Decoder for custom data types that are identified at runtime by metadata (e.g. `type_name`).
///
/// Only Rust-registered custom types (e.g. `RustTestCustomData`, `MacroYieldCurveData`) can be
/// decoded. Unknown types return an error.
///
/// **Important:** The caller must ensure that any Rust custom data types are registered
/// via [`ensure_custom_data_registered::<T>()`] before use.
#[derive(Debug)]
pub struct CustomDataDecoder;

View on GitHub (pinned to 18893faf8b)

Solutions

1. Read the wrapped inner error ({e}) for the concrete cause — it names the failing encode step
2. Verify the custom type's metadata() output matches the data being encoded (field count, types, ordering)
3. Fix the type's EncodeToRecordBatch implementation if struct fields changed without updating the encoder
4. Add a unit test encoding a representative instance of the custom type to catch drift early

Example fix

// before: metadata defined manually, drifted from fields
fn metadata() -> Metadata { /* stale fields */ }
// after: derive metadata from the struct definition so it cannot drift
fn metadata() -> Metadata {
    Metadata::from_fields(&[("price", ArrowType::Float64), ("qty", ArrowType::UInt64)])
}

let items_typed: Vec<MyData> = ...;
let metadata = <MyData as EncodeToRecordBatch>::metadata();
assert_eq!(items_typed.len(), expected_len, "encoder input must be non-empty and consistent");

match ensure_custom_data_registered::<MyData>() {
    Err(e) => {
        // {e} wraps the encode_batch error verbatim — log with the full chain
        log::error!("custom data encode failed: {e:#}");
    }
    Ok(()) => (),
}

When it happens

Trigger: Encoding custom data where T::encode_batch fails — typically metadata mismatch between the declared EncodeToRecordBatch metadata and the typed items, or a field that cannot be converted to its Arrow representation.

Common situations: Custom data types with hand-written EncodeToRecordBatch impls that drifted from the struct fields; empty or mismatched metadata dictionaries; nulls in fields the encoder assumes non-null.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Instances must have encode_record_batch_py method
• Failed to call encode_record_batch_py: {e}
• Python object has no to_json() method or __dict__ attribute
• Expected {}, was different type
• CustomData envelope type '{envelope_type_name}' does not mat

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/67fe620e4e64b702. Report an issue: GitHub.