nautechsystems/nautilus_trader · error · anyhow::Error

Failed to encode custom data to Arrow: {e}

Error message

Failed to encode custom data to Arrow: {e}

What it means

Thrown when encode_custom_to_arrow itself errors while converting custom data items into a RecordBatch during write_custom_data_batch. Unlike the companion 'not registered' error, this means the type IS registered but its registered encoding function returned an error. The original encoder error is wrapped with context.

Source

Thrown at crates/persistence/src/backend/custom.rs:153

    let Some(first_custom) = data.first() else {
        anyhow::bail!("prepare_custom_data_batch called with empty data");
    };

    let type_name = first_custom.data.type_name();
    let identifier = first_custom.data_type.identifier().map(String::from);
    let dt_meta = first_custom.data_type.metadata_string_map();
    let data_type_json = first_custom
        .data_type
        .to_persistence_json()
        .map_err(|e| anyhow::anyhow!("Failed to serialize data_type for persistence: {e}"))?;

    let start_ts = first_custom.data.ts_init();
    let end_ts = data.last().map_or(start_ts, |custom| custom.data.ts_init());
    let items: Vec<Arc<dyn CustomDataTrait>> =
        data.into_iter().map(|c| Arc::clone(&c.data)).collect();

    let batch = encode_custom_to_arrow(type_name, &items)
        .map_err(|e| anyhow::anyhow!("Failed to encode custom data to Arrow: {e}"))?
        .ok_or_else(|| {
            anyhow::anyhow!(
                "Custom data type \"{type_name}\" is not registered for Arrow encoding; \
                 call register_custom_data_class or ensure_custom_data_registered before writing"
            )
        })?;
    let batch =
        augment_batch_with_data_type_column(&batch, &data_type_json, type_name, dt_meta.as_ref())?;

    Ok((batch, type_name.to_string(), identifier, start_ts, end_ts))
}

/// Decodes a `RecordBatch` to Data objects based on metadata.
///
/// Supports both standard data types and custom data types when `allow_custom_fallback`
/// is true (e.g. when decoding files under `custom/`). When false, unknown type names
/// produce an error instead of attempting custom decode.
///

View on GitHub (pinned to 18893faf8b)

Solutions

1. Read the wrapped {e} message to find the failure inside the registered encoder.
2. Fix the custom type's encode implementation so it builds arrays matching its declared schema for all items.
3. Re-check that all items in the batch come from the same data type version (no mixed schema definitions).
4. Test encode_custom_to_arrow directly with a small sample before writing large batches.

Example fix

// before
let values = StringArray::from_iter(items.iter().map(|i| i.payload.as_deref())); // Option None conflicts with non-nullable schema
// after
let values = StringArray::from_iter(items.iter().map(|i| Some(i.payload.as_str())));

let sample = encode_custom_to_arrow(type_name, &small_sample)?; // fail fast on encoder bugs

match encode_custom_to_arrow(type_name, &items) {
    Ok(Some(batch)) => proceed(batch),
    Ok(None) => register_and_retry(type_name)?,
    Err(e) => log::error!("encoder failed for {type_name}: {e:#}"),
}

When it happens

Trigger: Writing custom data whose registered Arrow encoder returns Err — e.g. mismatched item schema inside the type's encode implementation, failure to build arrays from the items, or an empty/malformed item list the encoder rejects.

Common situations: A custom Data type's registered encoder has a bug (wrong array construction, schema mismatch); items were mutated or serialized through a path that altered the expected field set; version mismatch between the registered encoder and the current data struct definition.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Instances must have encode_record_batch_py method
• Expected {}, was different type
• prepare_custom_data_batch called with empty data
• Unknown data type: {type_name}
• Unknown data type: {type_name}; custom decode only allowed i

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/32be1d82b8cdb0c4. Report an issue: GitHub.