nautechsystems/nautilus_trader · error · anyhow::Error

Failed to merge custom data type metadata: {e}

Error message

Failed to merge custom data type metadata: {e}

What it means

Raised when augment_batch_with_data_type_column fails to construct a new RecordBatch that appends the serialized data_type JSON column with the merged schema metadata. RecordBatch::try_new validates that each column's data type and length match the schema fields, so a mismatch between the declared field and the actual data_type_array triggers this error. It wraps the underlying Arrow validation error.

Source

Thrown at crates/persistence/src/backend/custom.rs:89

    ));
    let schema = batch.schema();
    let mut fields: Vec<_> = schema.fields().iter().cloned().collect();
    fields.push(Arc::new(Field::new(
        "data_type",
        ArrowDataType::Utf8,
        false,
    )));
    let mut meta = schema.metadata().clone();
    meta.insert("type_name".to_string(), type_name.to_string());

    if let Some(m) = dt_meta {
        meta.extend(m.clone());
    }
    let new_schema = Arc::new(Schema::new_with_metadata(fields, meta));
    let mut columns = batch.columns().to_vec();
    columns.push(data_type_array);
    let new_batch = RecordBatch::try_new(new_schema, columns)
        .map_err(|e| anyhow::anyhow!("Failed to merge custom data type metadata: {e}"))?;
    Ok(new_batch)
}

/// Normalizes a custom data identifier for use in directory paths.
/// Replaces `//` with `/`, and filters out empty segments and `..` to prevent path traversal.
#[must_use]
fn safe_directory_identifier(identifier: &str) -> String {
    let normalized = identifier.replace("//", "/");
    let segments: Vec<&str> = normalized
        .split('/')
        .filter(|s| !s.is_empty() && *s != "..")
        .collect();
    segments.join("/")
}

/// Returns path components for custom data: `["data", "custom", type_name, ...identifier segments]`.
/// Used by the catalog to build full object-store paths via `make_object_store_path_owned`.
#[must_use]

View on GitHub (pinned to 18893faf8b)

Solutions

1. Check that the array returned by the custom encoder has exactly the same length as the input batch rows.
2. Verify the appended data_type_array type matches what Schema::new_with_metadata declared for it (StringArray for a Utf8 field).
3. Read the wrapped Arrow message (the {e} in the error text) to identify the exact field/column mismatch.
4. Update the custom Data type's encode implementation to conform to the expected schema.

Example fix

// before
columns.push(data_type_array); // array built with fewer rows than batch
// after
assert_eq!(data_type_array.len(), batch.num_rows(), "data_type column length must match batch");
columns.push(data_type_array);

anyhow::ensure!(data_type_array.len() == batch.num_rows(), "data_type column length must equal batch rows");

fn can_augment(batch: &RecordBatch, extra: &ArrayRef) -> bool {
    extra.len() == batch.num_rows()
}

match augment_batch_with_data_type_column(&batch, &json, name, meta) {
    Ok(b) => write(b),
    Err(e) => log::error!("schema merge failed: {e:#}"),
}

When it happens

Trigger: The data_type_array pushed as the final column has an Arrow type or length that does not match the newly built schema field (e.g. string array vs. dictionary field), or the column count/length is inconsistent with the batch rows.

Common situations: A custom Data implementation returns a metadata_string_map or persistence JSON whose schema conflicts with the appended column; an encode_custom_to_arrow implementation returns a batch whose row count differs from the items it was given.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Failed to concatenate stream batches: {e}
• height must be positive, was {self.height}
• data_type must be a JSON object
• ts_event column not found
• ts_init column not found

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/da7d12ab9ee9ba10. Report an issue: GitHub.