{"record":{"id":"da7d12ab9ee9ba10","repo":"nautechsystems/nautilus_trader","slug":"failed-to-merge-custom-data-type-metadata-e","errorCode":null,"errorMessage":"Failed to merge custom data type metadata: {e}","messagePattern":"Failed to merge custom data type metadata: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/persistence/src/backend/custom.rs","lineNumber":89,"sourceCode":"    ));\n    let schema = batch.schema();\n    let mut fields: Vec<_> = schema.fields().iter().cloned().collect();\n    fields.push(Arc::new(Field::new(\n        \"data_type\",\n        ArrowDataType::Utf8,\n        false,\n    )));\n    let mut meta = schema.metadata().clone();\n    meta.insert(\"type_name\".to_string(), type_name.to_string());\n\n    if let Some(m) = dt_meta {\n        meta.extend(m.clone());\n    }\n    let new_schema = Arc::new(Schema::new_with_metadata(fields, meta));\n    let mut columns = batch.columns().to_vec();\n    columns.push(data_type_array);\n    let new_batch = RecordBatch::try_new(new_schema, columns)\n        .map_err(|e| anyhow::anyhow!(\"Failed to merge custom data type metadata: {e}\"))?;\n    Ok(new_batch)\n}\n\n/// Normalizes a custom data identifier for use in directory paths.\n/// Replaces `//` with `/`, and filters out empty segments and `..` to prevent path traversal.\n#[must_use]\nfn safe_directory_identifier(identifier: &str) -> String {\n    let normalized = identifier.replace(\"//\", \"/\");\n    let segments: Vec<&str> = normalized\n        .split('/')\n        .filter(|s| !s.is_empty() && *s != \"..\")\n        .collect();\n    segments.join(\"/\")\n}\n\n/// Returns path components for custom data: `[\"data\", \"custom\", type_name, ...identifier segments]`.\n/// Used by the catalog to build full object-store paths via `make_object_store_path_owned`.\n#[must_use]","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/persistence/src/backend/custom.rs#L71-L107","documentation":"Raised when augment_batch_with_data_type_column fails to construct a new RecordBatch that appends the serialized data_type JSON column with the merged schema metadata. RecordBatch::try_new validates that each column's data type and length match the schema fields, so a mismatch between the declared field and the actual data_type_array triggers this error. It wraps the underlying Arrow validation error.","triggerScenarios":"The data_type_array pushed as the final column has an Arrow type or length that does not match the newly built schema field (e.g. string array vs. dictionary field), or the column count/length is inconsistent with the batch rows.","commonSituations":"A custom Data implementation returns a metadata_string_map or persistence JSON whose schema conflicts with the appended column; an encode_custom_to_arrow implementation returns a batch whose row count differs from the items it was given.","solutions":["Check that the array returned by the custom encoder has exactly the same length as the input batch rows.","Verify the appended data_type_array type matches what Schema::new_with_metadata declared for it (StringArray for a Utf8 field).","Read the wrapped Arrow message (the {e} in the error text) to identify the exact field/column mismatch.","Update the custom Data type's encode implementation to conform to the expected schema."],"exampleFix":"// before\ncolumns.push(data_type_array); // array built with fewer rows than batch\n// after\nassert_eq!(data_type_array.len(), batch.num_rows(), \"data_type column length must match batch\");\ncolumns.push(data_type_array);","handlingStrategy":"validation","validationCode":"anyhow::ensure!(data_type_array.len() == batch.num_rows(), \"data_type column length must equal batch rows\");","typeGuard":"fn can_augment(batch: &RecordBatch, extra: &ArrayRef) -> bool {\n    extra.len() == batch.num_rows()\n}","tryCatchPattern":"match augment_batch_with_data_type_column(&batch, &json, name, meta) {\n    Ok(b) => write(b),\n    Err(e) => log::error!(\"schema merge failed: {e:#}\"),\n}","preventionTips":["Keep the data_type column a plain Utf8/StringArray","Verify array lengths equal batch row count before appending","Unit test each custom type's full write path","Read the wrapped Arrow error to identify the mismatching field"],"tags":["arrow","rust","persistence","schema"],"backgroundTag":"schema-validation-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}