vectordotdev/vector · error · ArrowEncodingError

Invalid Map schema for field '{field_name}': {reason}

Error message

Invalid Map schema for field '{field_name}': {reason}

What it means

The Arrow serializer walks each schema field; for a `DataType::Map`, Arrow requires the inner type to be a Struct named "entries" with exactly two fields (key, value). If the inner type is not a Struct, `InvalidMapSchemaSnafu` fails with this message naming the field and the offending inner type. It is a schema-shape validation error raised while preparing the Arrow encoder, before any events are serialized.

Source

Thrown at lib/codecs/src/encoding/format/arrow.rs:258

        DataType::List(inner_field) => DataType::List(make_field_nullable(inner_field)?.into()),
        DataType::Struct(fields) => DataType::Struct(
            fields
                .iter()
                .map(|f| make_field_nullable(f))
                .collect::<Result<Vec<_>, _>>()?
                .into(),
        ),
        DataType::Map(inner, sorted) => {
            // A Map's inner field is a "entries" Struct<Key, Value>
            let DataType::Struct(fields) = inner.data_type() else {
                return InvalidMapSchemaSnafu {
                    field_name: field.name(),
                    reason: format!("inner type must be Struct, found {:?}", inner.data_type()),
                }
                .fail();
            };

            ensure!(
                fields.len() == 2,
                InvalidMapSchemaSnafu {
                    field_name: field.name(),
                    reason: format!("expected 2 fields (key, value), found {}", fields.len()),
                },
            );
            let key_field = &fields[0];
            let value_field = &fields[1];

            let new_struct_fields: Fields =
                [key_field.clone(), make_field_nullable(value_field)?.into()].into();

            // Reconstruct the inner "entries" field
            // The inner field itself must be non-nullable (only the Map wrapper is nullable)
            let new_inner_field = inner
                .as_ref()
                .clone()
                .with_data_type(DataType::Struct(new_struct_fields))

View on GitHub (pinned to 3708c39b12)

Solutions

1. Rewrite the offending Map field so its inner type is `Struct<key: K, value: V-not-null>` (Arrow's entries convention) — e.g. `Map(Field("entries", Struct[("key", Utf8), ("value", V)]), sorted)`.
2. If the logical data is a nested collection rather than a map, use `DataType::List` instead of `Map`.
3. Validate the schema with arrow-rs `DataType::try_parse`/a small unit test before deploying it to the sink config.

Example fix

// before
DataType::Map(
    Field::new("entries", DataType::Utf8, false).into(),
    false,
)

// after
let kv = DataType::Struct(Fields::from(vec![
    Field::new("key", DataType::Utf8, false),
    Field::new("value", DataType::Utf8, false),
]));
DataType::Map(Field::new("entries", kv, false).into(), false)

fn assert_arrow_maps_valid(schema: &arrow_schema::Schema) -> Result<(), String> {
    for f in schema.fields() {
        if let DataType::Map(inner, _) = f.data_type() {
            let DataType::Struct(fields) = inner.data_type() else {
                return Err(format!("field {} Map inner must be Struct", f.name()));
            };
            if fields.len() != 2 {
                return Err(format!("field {} Map entries must be <key, value>", f.name()));
            }
        }
    }
    Ok(())
}

fn is_valid_arrow_map(field: &Field) -> bool {
    matches!(field.data_type(), DataType::Map(inner, _)
        if matches!(inner.data_type(), DataType::Struct(fs) if fs.len() == 2))
}

When it happens

Trigger: Building/using an Arrow schema (Vector's `encoding.codec = "arrow"` with a supplied schema, or the codecs API directly) where a Map field's inner type is e.g. another Map, a List, or a primitive instead of `Struct<key: K, value: V>`. A Struct inner with ≠2 fields hits the same error variant with a different reason string.

Common situations: Hand-writing Arrow schemas from JSON examples that model maps as `map<string, list<...>>`; converting Parquet/JSON-derived schemas that embed map-of-list types; schema drift after upgrading arrow-rs where Map inner representation changed.

Related errors

• registered log schema required
• source output misconfigured - output for port {:?} missing
• not implemented
• The drain_timeout_ms ({}) must be less than session_timeout_
• component name must be non-empty

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/d544f4316f559e9e. Report an issue: GitHub.