{"record":{"id":"34167a56582fc854","repo":"nautechsystems/nautilus_trader","slug":"failed-to-serialize-data-type-for-persistence-e","errorCode":null,"errorMessage":"Failed to serialize data_type for persistence: {e}","messagePattern":"Failed to serialize data_type for persistence: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/custom.rs","lineNumber":145,"sourceCode":"/// and returns type identity and timestamp range so the catalog can build path and perform I/O.\n///\n/// # Errors\n///\n/// Returns an error if encoding or augmentation fails, or if the type is not registered.\npub fn prepare_custom_data_batch(\n    data: Vec<CustomData>,\n) -> anyhow::Result<(RecordBatch, String, Option<String>, UnixNanos, UnixNanos)> {\n    let Some(first_custom) = data.first() else {\n        anyhow::bail!(\"prepare_custom_data_batch called with empty data\");\n    };\n\n    let type_name = first_custom.data.type_name();\n    let identifier = first_custom.data_type.identifier().map(String::from);\n    let dt_meta = first_custom.data_type.metadata_string_map();\n    let data_type_json = first_custom\n        .data_type\n        .to_persistence_json()\n        .map_err(|e| anyhow::anyhow!(\"Failed to serialize data_type for persistence: {e}\"))?;\n\n    let start_ts = first_custom.data.ts_init();\n    let end_ts = data.last().map_or(start_ts, |custom| custom.data.ts_init());\n    let items: Vec<Arc<dyn CustomDataTrait>> =\n        data.into_iter().map(|c| Arc::clone(&c.data)).collect();\n\n    let batch = encode_custom_to_arrow(type_name, &items)\n        .map_err(|e| anyhow::anyhow!(\"Failed to encode custom data to Arrow: {e}\"))?\n        .ok_or_else(|| {\n            anyhow::anyhow!(\n                \"Custom data type \\\"{type_name}\\\" is not registered for Arrow encoding; \\\n                 call register_custom_data_class or ensure_custom_data_registered before writing\"\n            )\n        })?;\n    let batch =\n        augment_batch_with_data_type_column(&batch, &data_type_json, type_name, dt_meta.as_ref())?;\n\n    Ok((batch, type_name.to_string(), identifier, start_ts, end_ts))","sourceCodeStart":127,"sourceCodeEnd":163,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/custom.rs#L127-L163","documentation":"Thrown when a custom data type's to_persistence_json() fails while preparing a batch for writing via write_custom_data_batch. The data_type JSON is stored alongside the Arrow batch as schema metadata so the type can be reconstructed on read; if the type cannot serialize itself, persistence cannot proceed and the underlying serialization error is wrapped.","triggerScenarios":"Writing custom data whose DataType implementation's to_persistence_json returns an error — e.g. a field that cannot be represented in JSON, an inconsistent/missing persistence definition, or a nested type that fails its own serialization.","commonSituations":"Custom serde-incompatible fields (non-string map keys, unserializable enums) in a user-defined DataType; version drift where the stored definition schema changed and validation inside to_persistence_json now rejects it.","solutions":["Inspect the wrapped {e} message to see which field or check inside to_persistence_json failed.","Fix the custom DataType definition so all fields are JSON-serializable and satisfy the persistence schema requirements.","If the definition changed, migrate old definitions or implement compatibility in to_persistence_json.","Add a unit test calling to_persistence_json before writing to the catalog."],"exampleFix":"// before\nfn to_persistence_json(&self) -> anyhow::Result<serde_json::Value> {\n    Ok(serde_json::to_value(&self.non_serializable_field)?)\n}\n// after\nfn to_persistence_json(&self) -> anyhow::Result<serde_json::Value> {\n    Ok(serde_json::json!({ \"schema\": self.schema_json()? }))\n}","handlingStrategy":"try-catch","validationCode":"let json = data_type.to_persistence_json().expect(\"data_type must serialize before write\");","typeGuard":null,"tryCatchPattern":"match first_custom.data_type.to_persistence_json() {\n    Ok(json) => proceed(json),\n    Err(e) => eprintln!(\"fix to_persistence_json for {}: {e}\", first_custom.data.type_name()),\n}","preventionTips":["Unit test to_persistence_json for every custom DataType","Keep persistence definitions JSON-serializable (avoid exotic serde types)","Bump/migrate definitions when the persistence schema changes","Call to_persistence_json once at startup as a smoke test"],"tags":["rust","serialization","persistence","custom-data"],"backgroundTag":"json-marshal-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}