apache/beam · error · ValueError

Unable to convert {avro_type} to a Beam schema.

Error message

Unable to convert {avro_type} to a Beam schema.

What it means

avroio.avro_type_to_beam_type maps Avro schema type descriptors (dicts like {'type':'record',...}, 'string', ['null','int'], etc.) to Beam schema FieldTypes. Avro's type system includes constructs Beam schemas cannot express directly; when the descriptor is none of the supported forms (record, enum, array, map, union, named primitive, or string primitive name), the function raises ValueError with the offending avro_type.

Source

Thrown at sdks/python/apache_beam/io/avroio.py:621

    return schema_pb2.FieldType(
        array_type=schema_pb2.ArrayType(
            element_type=avro_type_to_beam_type(avro_type['items'])))
  elif type_name == 'map':
    return schema_pb2.FieldType(
        map_type=schema_pb2.MapType(
            key_type=schema_pb2.FieldType(atomic_type=schema_pb2.STRING),
            value_type=avro_type_to_beam_type(avro_type['values'])))
  elif type_name == 'record':
    return schema_pb2.FieldType(
        row_type=schema_pb2.RowType(
            schema=schema_pb2.Schema(
                fields=[
                    schemas.schema_field(
                        f['name'], avro_type_to_beam_type(f['type']))
                    for f in avro_type['fields']
                ])))
  else:
    raise ValueError(f'Unable to convert {avro_type} to a Beam schema.')


def avro_schema_to_beam_schema(
    avro_schema: _AvroSchemaType) -> schema_pb2.Schema:
  beam_type = avro_type_to_beam_type(avro_schema)
  if isinstance(avro_schema, dict) and avro_schema['type'] == 'record':
    return beam_type.row_type.schema
  else:
    return schema_pb2.Schema(fields=[schemas.schema_field('record', beam_type)])


def avro_dict_to_beam_row(
    avro_schema: _AvroSchemaType,
    beam_schema: schema_pb2.Schema) -> Callable[[Any], Any]:
  if isinstance(avro_schema, str):
    return avro_dict_to_beam_row({'type': avro_schema})
  if avro_schema['type'] == 'record':
    to_row = avro_value_to_beam_value(

View on GitHub (pinned to 12126d8942)

Solutions

1. Pre-process the Avro schema: replace unsupported types (e.g. 'fixed' with 'bytes', 3+ branch unions with single nullable unions) before conversion
2. Read the file as raw records (ParseAllFromAvro yields dicts) and Map them into beam.Row objects yourself instead of relying on automatic schema conversion
3. Check the Beam version — newer releases support more Avro constructs; upgrade apache-beam
4. Validate the schema dict's 'type' field is one Beam supports (record/enum/array/map/union/string primitives)

Example fix

// before
schema = {'type': 'record', 'name': 'R', 'fields': [{'name': 'h', 'type': {'type': 'fixed', 'name': 'F', 'size': 16}}]}
avroio.avro_schema_to_beam_schema(schema)  # ValueError
// after
schema = {'type': 'record', 'name': 'R', 'fields': [{'name': 'h', 'type': 'bytes'}]}
beam_schema = avroio.avro_schema_to_beam_schema(schema)

SUPPORTED = {'record','enum','array','map','union','string','bytes','int','long','float','double','boolean','null'}
assert all(isinstance(t, str) and t in SUPPORTED or isinstance(t, (dict, list)) for t in flat_types(avro_schema))

try:
    beam_schema = avroio.avro_schema_to_beam_schema(avro_schema)
except ValueError as e:
    logging.error('Avro type not supported: %s', e)
    # fall back to reading raw dicts instead of schema'd rows

When it happens

Trigger: Calling avro_schema_to_beam_schema / avro_type_to_beam_type on an Avro schema containing an unsupported type descriptor — e.g. a dict whose 'type' is 'fixed', a malformed dict without a recognized 'type' key, or a union/list form that is not a simple nullable union.

Common situations: Reading exotic Avro files (fixed-size binary fields, complex unions of 3+ branches) and piping them through Beam schema conversion; hand-written or third-party-generated .avsc schemas with constructs Beam's mapper does not handle.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Unrecognized type_info: {type_info!r}
• An explicit schema is required to write non-schema'd PCollec
• Only strings allowed as map keys when converting from AVRO,
• Unknown BigQuery field mode: {}
• Reserved field name <field.name()> in user schema.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/75a24ac10ed41377. Report an issue: GitHub.