apache/beam · error · ValueError

You are using Avro IO with fastavro (default with Beam on Py

Error message

You are using Avro IO with fastavro (default with Beam on Python 3), but supplying a schema parsed by avro-python3. Please change the schema to a dict.

What it means

apache_beam.io.avroio._create_avro_sink chooses the sink implementation based on the Avro library available. When fastavro is installed (the default for Beam on Python 3), schemas must be dicts (fastavro-style); supplying a schema object parsed by the avro-python3 library (its type string starts with "class 'avro.schema") raises ValueError because fastavro cannot consume avro-python3 schema objects.

Source

Thrown at sdks/python/apache_beam/io/avroio.py:467

          self._sink.shard_name_template)

    return records | beam.io.iobase.Write(self._sink)

  def display_data(self):
    return {'sink_dd': self._sink}


def _create_avro_sink(
    file_path_prefix,
    schema,
    codec,
    file_name_suffix,
    num_shards,
    shard_name_template,
    mime_type,
    triggering_frequency=60):
  if "class 'avro.schema" in str(type(schema)):
    raise ValueError(
        'You are using Avro IO with fastavro (default with Beam on '
        'Python 3), but supplying a schema parsed by avro-python3. '
        'Please change the schema to a dict.')
  return _FastAvroSink(
      file_path_prefix,
      schema,
      codec,
      file_name_suffix,
      num_shards,
      shard_name_template,
      mime_type,
      triggering_frequency)


class _BaseAvroSink(filebasedsink.FileBasedSink):
  """A base for a sink for avro files. """
  def __init__(
      self,

View on GitHub (pinned to 12126d8942)

Solutions

1. Load the schema as a plain dict: json.load(open('schema.avsc')) instead of avro.schema.parse()
2. Uninstall/avoid avro-python3 and rely on fastavro (Beam's Python 3 default)
3. Construct the schema dict inline with {'type': 'record', 'name': ..., 'fields': [...]}
4. If you must use avro-python3 schemas, force the non-fastavro sink by ensuring fastavro is not installed (not recommended)

Example fix

// before
import avro.schema
schema = avro.schema.parse(open('schema.avsc').read())
WriteToAvro('out.avro', schema=schema)  # ValueError
// after
import json
schema = json.load(open('schema.avsc'))
WriteToAvro('out.avro', schema=schema)

is_bad = "class 'avro.schema" in str(type(schema))
assert not is_bad, 'use a dict schema for fastavro'

def is_fastavro_schema(schema):
    return isinstance(schema, dict) and 'type' in schema

def is_avro_python3_schema(schema):
    return "class 'avro.schema" in str(type(schema))

try:
    sink = WriteToAvro(path, schema=schema)
except ValueError as e:
    import json
    if isinstance(schema, str):
        schema = json.loads(schema)
    else:
        schema = schema.to_json()
    sink = WriteToAvro(path, schema=schema)

When it happens

Trigger: Creating WriteToAvro/_create_avro_sink with schema=<avro.schema.Schema object from avro-python3's schema.parse()> while Beam is using the fastavro-based sink.

Common situations: Mixing avro-python3 and fastavro in one project; loading a .avsc file with the wrong library; older Beam code that used avro-python3 being run under a Python 3 Beam install that defaults to fastavro; dependencies pulling in avro-python3 transitively and code paths branching on it.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• repeat(repeats=) value must be an int or a DeferredSeries (e
• Passing a deferred series to round() is not supported, pleas
• str.repeat(repeats=) value must be an int or a DeferredSerie
• Could not find code object with path: {code_object_identifie
• An explicit schema is required to write non-schema'd PCollec

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7447dea8bf28fc93. Report an issue: GitHub.