{"record":{"id":"cc39435e07c5daac","repo":"apache/beam","slug":"error-writing-row-to-avro-schema-row","errorCode":null,"errorMessage":"Error writing row to Avro: {}\nSchema: {}\nRow: {}","messagePattern":"Error writing row to Avro: (.+?)\nSchema: (.+?)\nRow: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery_tools.py","lineNumber":1552,"sourceCode":"\n  def read(self, size=-1):\n    raise io.UnsupportedOperation(\"AvroRowWriter is not readable\")\n\n  def tell(self):\n    # Flush the fastavro Writer to the underlying stream, otherwise there isn't\n    # a reliable way to determine how many bytes have been written.\n    self._avro_writer.flush()\n    return self._file_handle.tell()\n\n  def writable(self):\n    return self._file_handle.writable()\n\n  def write(self, row):\n    try:\n      self._avro_writer.write(row)\n    except (TypeError, ValueError) as ex:\n      _, _, tb = sys.exc_info()\n      raise ex.__class__(\n          \"Error writing row to Avro: {}\\nSchema: {}\\nRow: {}\".format(\n              ex, self._avro_writer.schema, row)).with_traceback(tb)\n\n\nclass RetryStrategy(object):\n  RETRY_ALWAYS = 'RETRY_ALWAYS'\n  RETRY_NEVER = 'RETRY_NEVER'\n  RETRY_ON_TRANSIENT_ERROR = 'RETRY_ON_TRANSIENT_ERROR'\n\n  # Values below may be found in reasons provided either in an\n  # error returned by a client method or by an http response as\n  # defined in google.api_core.exceptions\n  _NON_TRANSIENT_ERRORS = {\n      'invalid',\n      'invalidQuery',\n      'notImplemented',\n      'Bad Request',\n      'Unauthorized',","sourceCodeStart":1534,"sourceCodeEnd":1570,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L1534-L1570","documentation":"apache_beam raises this when a row cannot be serialized to Avro format while writing records for a BigQuery load. The underlying TypeError/ValueError from the Avro writer is re-raised with the failing row and schema embedded in the message so the developer can see exactly which record and which schema field caused the mismatch.","triggerScenarios":"Passing write() a row whose fields don't match the Avro schema: wrong types (str where int/long expected), missing required fields, extra fields when strict, None in non-nullable fields, or non-UTF-8/bytes data where string expected.","commonSituations":"BigQuery sink file loads with dict rows that don't match the inferred or supplied schema; schema drift after a table change; rows produced by a DoFn emitting values of the wrong Python type (e.g. numpy types, datetime objects not Avro logical types).","solutions":["Validate each row against the Avro schema (field names, types, nullability) before writing","Convert values to schema-compatible Python types (int, float, str, bytes, isoformat strings for logical types)","Regenerate/refresh the schema so it matches the actual data being written","Wrap writes in try/except and log the offending row for offline fixing"],"exampleFix":"// before\nwriter.write({'id': 'abc', 'ts': datetime.now()})\n// after\nwriter.write({'id': int('abc' or 0), 'ts': datetime.now().isoformat()})","handlingStrategy":"validation","validationCode":"def row_matches_schema(row, schema_fields):\n    for f in schema_fields:\n        if f['name'] not in row:\n            raise ValueError(f\"missing field {f['name']}\")\n        v = row[f['name']]\n        if v is None and not f.get('nullable', True):\n            raise ValueError(f\"field {f['name']} is not nullable\")\n    return True","typeGuard":"def is_avro_compatible(row):\n    return isinstance(row, dict) and all(isinstance(k, str) for k in row)","tryCatchPattern":"try:\n    writer.write(row)\nexcept (TypeError, ValueError) as ex:\n    logger.error('Avro write failed for row: %r schema: %s', row, writer.writer.schema)\n    raise","preventionTips":["Validate rows against the schema in a pre-write step","Coerce types explicitly (int(), str(), isoformat) before writing","Add unit tests covering schema vs data drift"],"tags":["python","avro","bigquery","serialization","schema-mismatch"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}