{"record":{"id":"5e35223fcb464666","repo":"apache/beam","slug":"error-converting-field","errorCode":null,"errorMessage":"Error converting field : ","messagePattern":"Error converting field : ","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java","lineNumber":706,"sourceCode":"\n    @Override\n    public TableRow apply(T input) {\n      return toTableRow(toRow.apply(input));\n    }\n  }\n\n  public static Row toBeamRow(GenericRecord record, Schema schema, ConversionOptions options) {\n    List<@Nullable Object> valuesInOrder =\n        schema.getFields().stream()\n            .<@Nullable Object>map(\n                field -> {\n                  try {\n                    org.apache.avro.Schema.Field avroField =\n                        record.getSchema().getField(field.getName());\n                    @Nullable Object value = avroField != null ? record.get(avroField.pos()) : null;\n                    return convertAvroFormat(field.getType(), value, options);\n                  } catch (Exception cause) {\n                    throw new IllegalArgumentException(\n                        \"Error converting field \" + field + \": \" + cause.getMessage(), cause);\n                  }\n                })\n            .collect(toList());\n\n    return Row.withSchema(schema).addValues(valuesInOrder).build();\n  }\n\n  /**\n   * Convert generic record to Bq TableRow.\n   *\n   * @deprecated use {@link #convertGenericRecordToTableRow(GenericRecord)}\n   */\n  @Deprecated\n  public static TableRow convertGenericRecordToTableRow(\n      GenericRecord record, TableSchema tableSchema) {\n    return convertGenericRecordToTableRow(record);\n  }","sourceCodeStart":688,"sourceCodeEnd":724,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L688-L724","documentation":"toBeamRow wraps any exception thrown while converting one Avro record field to its Beam value in an IllegalArgumentException with the message \"Error converting field <field>: <cause>\". The original cause (NumberFormatException, ClassCastException, Avro type mismatch, etc.) is preserved as the cause. It indicates the Avro record read from BigQuery does not match the expected Beam Schema field during the Avro-format to Beam Row conversion.","triggerScenarios":"Reading BigQuery with storage API / Avro format (convertAvroFormat) when a record's field value cannot be converted to the Beam Schema field type: e.g. Avro union branch mismatch, null in a non-nullable field, numeric overflow, string where bytes expected, or schema drift between the BigQuery table and the expected schema.","commonSituations":"BigQuery table schema changed after the pipeline was built; reading with a user-provided Beam Schema that doesn't match the exported Avro schema; corrupt or legacy Avro rows; DATETIME/TIMESTAMP/NUMERIC values outside what the converter's parse logic accepts.","solutions":["Inspect the chained cause (IllegalArgumentException#getCause) to find the actual failing conversion and fix the offending data or field mapping.","Re-derive the Beam Schema from the actual BigQuery table (BigQueryUtils.fromTableSchema / BeamIO's schema inference) instead of a hand-written schema so it matches the Avro export.","Null-out or widen the problematic field in the schema (e.g. make it NULLABLE or STRING) and cast/convert afterwards with map/ParDo logic you control.","Check for schema drift: compare the current table schema against the pipeline's expectations and update/redeploy accordingly."],"exampleFix":"// before (hand-written, drifted schema)\nSchema schema = Schema.builder().addField(\"amount\", FieldType.INT64).build();\n// after (infer from table)\nSchema schema = BigQueryUtils.fromTableSchema(bqClient.getTable(tableId).getDefinition().getSchema());","handlingStrategy":"try-catch","validationCode":"// compare expected schema with the actual BigQuery table before reading\nTable bqTable = bigquery.getTable(tableId);\nSchema actual = BigQueryUtils.fromTableSchema(bqTable.getDefinition().getSchema());\nif (!actual.equivalent(expectedSchema)) {\n  throw new IllegalStateException(\"Schema drift detected between pipeline and BigQuery table\");\n}","typeGuard":"static boolean rowMatchesSchema(org.apache.avro.generic.GenericRecord rec, Schema beamSchema) {\n  for (Field f : beamSchema.getFields()) {\n    org.apache.avro.Schema.Field af = rec.getSchema().getField(f.getName());\n    if (af == null && !f.getType().getNullable()) return false;\n  }\n  return true;\n}","tryCatchPattern":"try {\n  pipeline.apply(BigQueryIO.readTableRows().withMethod(DIRECT_READ)...);\n} catch (IllegalArgumentException e) {\n  if (String.valueOf(e.getMessage()).startsWith(\"Error converting field\")) {\n    log.error(\"Avro->Beam row conversion failed; check schema drift. Cause:\", e.getCause());\n    // route to dead-letter / re-read with inferred schema\n  } else { throw e; }\n}","preventionTips":["Always derive the Beam Schema from the live BigQuery table schema rather than hardcoding it.","Alert on BigQuery table schema changes (INFORMATION_SCHEMA / audit logs) that a running pipeline consumes.","Inspect e.getCause() first when debugging — the wrapper message hides the real conversion failure.","Add dead-letter handling in the pipeline for records that fail row conversion instead of failing the whole job."],"tags":["bigquery","beam","avro","row-conversion","type-mismatch"],"backgroundTag":"type-mismatch","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}