{"record":{"id":"32be1d82b8cdb0c4","repo":"nautechsystems/nautilus_trader","slug":"failed-to-encode-custom-data-to-arrow-e","errorCode":null,"errorMessage":"Failed to encode custom data to Arrow: {e}","messagePattern":"Failed to encode custom data to Arrow: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/custom.rs","lineNumber":153,"sourceCode":"    let Some(first_custom) = data.first() else {\n        anyhow::bail!(\"prepare_custom_data_batch called with empty data\");\n    };\n\n    let type_name = first_custom.data.type_name();\n    let identifier = first_custom.data_type.identifier().map(String::from);\n    let dt_meta = first_custom.data_type.metadata_string_map();\n    let data_type_json = first_custom\n        .data_type\n        .to_persistence_json()\n        .map_err(|e| anyhow::anyhow!(\"Failed to serialize data_type for persistence: {e}\"))?;\n\n    let start_ts = first_custom.data.ts_init();\n    let end_ts = data.last().map_or(start_ts, |custom| custom.data.ts_init());\n    let items: Vec<Arc<dyn CustomDataTrait>> =\n        data.into_iter().map(|c| Arc::clone(&c.data)).collect();\n\n    let batch = encode_custom_to_arrow(type_name, &items)\n        .map_err(|e| anyhow::anyhow!(\"Failed to encode custom data to Arrow: {e}\"))?\n        .ok_or_else(|| {\n            anyhow::anyhow!(\n                \"Custom data type \\\"{type_name}\\\" is not registered for Arrow encoding; \\\n                 call register_custom_data_class or ensure_custom_data_registered before writing\"\n            )\n        })?;\n    let batch =\n        augment_batch_with_data_type_column(&batch, &data_type_json, type_name, dt_meta.as_ref())?;\n\n    Ok((batch, type_name.to_string(), identifier, start_ts, end_ts))\n}\n\n/// Decodes a `RecordBatch` to Data objects based on metadata.\n///\n/// Supports both standard data types and custom data types when `allow_custom_fallback`\n/// is true (e.g. when decoding files under `custom/`). When false, unknown type names\n/// produce an error instead of attempting custom decode.\n///","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/custom.rs#L135-L171","documentation":"Thrown when encode_custom_to_arrow itself errors while converting custom data items into a RecordBatch during write_custom_data_batch. Unlike the companion 'not registered' error, this means the type IS registered but its registered encoding function returned an error. The original encoder error is wrapped with context.","triggerScenarios":"Writing custom data whose registered Arrow encoder returns Err — e.g. mismatched item schema inside the type's encode implementation, failure to build arrays from the items, or an empty/malformed item list the encoder rejects.","commonSituations":"A custom Data type's registered encoder has a bug (wrong array construction, schema mismatch); items were mutated or serialized through a path that altered the expected field set; version mismatch between the registered encoder and the current data struct definition.","solutions":["Read the wrapped {e} message to find the failure inside the registered encoder.","Fix the custom type's encode implementation so it builds arrays matching its declared schema for all items.","Re-check that all items in the batch come from the same data type version (no mixed schema definitions).","Test encode_custom_to_arrow directly with a small sample before writing large batches."],"exampleFix":"// before\nlet values = StringArray::from_iter(items.iter().map(|i| i.payload.as_deref())); // Option None conflicts with non-nullable schema\n// after\nlet values = StringArray::from_iter(items.iter().map(|i| Some(i.payload.as_str())));","handlingStrategy":"try-catch","validationCode":"let sample = encode_custom_to_arrow(type_name, &small_sample)?; // fail fast on encoder bugs","typeGuard":null,"tryCatchPattern":"match encode_custom_to_arrow(type_name, &items) {\n    Ok(Some(batch)) => proceed(batch),\n    Ok(None) => register_and_retry(type_name)?,\n    Err(e) => log::error!(\"encoder failed for {type_name}: {e:#}\"),\n}","preventionTips":["Test the registered encoder with representative items before bulk writes","Ensure all items share one schema/version of the data type","Keep nullable flags consistent between arrays and declared schema fields","Round-trip encode->decode in CI for custom types"],"tags":["rust","arrow","persistence","custom-data"],"backgroundTag":"schema-validation-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}