apache/beam · error

Error writing row to Avro

Error message

Error writing row to Avro: {}
Schema: {}
Row: {}

What it means

apache_beam raises this when a row cannot be serialized to Avro format while writing records for a BigQuery load. The underlying TypeError/ValueError from the Avro writer is re-raised with the failing row and schema embedded in the message so the developer can see exactly which record and which schema field caused the mismatch.

Solutions

1. Validate each row against the Avro schema (field names, types, nullability) before writing
2. Convert values to schema-compatible Python types (int, float, str, bytes, isoformat strings for logical types)
3. Regenerate/refresh the schema so it matches the actual data being written
4. Wrap writes in try/except and log the offending row for offline fixing

Example fix

// before
writer.write({'id': 'abc', 'ts': datetime.now()})
// after
writer.write({'id': int('abc' or 0), 'ts': datetime.now().isoformat()})

def row_matches_schema(row, schema_fields):
    for f in schema_fields:
        if f['name'] not in row:
            raise ValueError(f"missing field {f['name']}")
        v = row[f['name']]
        if v is None and not f.get('nullable', True):
            raise ValueError(f"field {f['name']} is not nullable")
    return True

def is_avro_compatible(row):
    return isinstance(row, dict) and all(isinstance(k, str) for k in row)

try:
    writer.write(row)
except (TypeError, ValueError) as ex:
    logger.error('Avro write failed for row: %r schema: %s', row, writer.writer.schema)
    raise

When it happens

Trigger: Passing write() a row whose fields don't match the Avro schema: wrong types (str where int/long expected), missing required fields, extra fields when strict, None in non-nullable fields, or non-UTF-8/bytes data where string expected.

Common situations: BigQuery sink file loads with dict rows that don't match the inferred or supplied schema; schema drift after a table change; rows produced by a DoFn emitting values of the wrong Python type (e.g. numpy types, datetime objects not Avro logical types).

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• AvroRowWriter is not readable
• flush on closed file
• Object of type ' ' is not JSON serializable
• . . Row: %r
• Unknown BigQuery field mode

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/cc39435e07c5daac. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:1552

  def read(self, size=-1):
    raise io.UnsupportedOperation("AvroRowWriter is not readable")

  def tell(self):
    # Flush the fastavro Writer to the underlying stream, otherwise there isn't
    # a reliable way to determine how many bytes have been written.
    self._avro_writer.flush()
    return self._file_handle.tell()

  def writable(self):
    return self._file_handle.writable()

  def write(self, row):
    try:
      self._avro_writer.write(row)
    except (TypeError, ValueError) as ex:
      _, _, tb = sys.exc_info()
      raise ex.__class__(
          "Error writing row to Avro: {}\nSchema: {}\nRow: {}".format(
              ex, self._avro_writer.schema, row)).with_traceback(tb)


class RetryStrategy(object):
  RETRY_ALWAYS = 'RETRY_ALWAYS'
  RETRY_NEVER = 'RETRY_NEVER'
  RETRY_ON_TRANSIENT_ERROR = 'RETRY_ON_TRANSIENT_ERROR'

  # Values below may be found in reasons provided either in an
  # error returned by a client method or by an http response as
  # defined in google.api_core.exceptions
  _NON_TRANSIENT_ERRORS = {
      'invalid',
      'invalidQuery',
      'notImplemented',
      'Bad Request',
      'Unauthorized',

View on GitHub (pinned to 12126d8942)