{"record":{"id":"6f194b199b68bb3c","repo":"apache/beam","slug":"field-s-not-nullable-bigqueryutils","errorCode":null,"errorMessage":"Field %s not nullable","messagePattern":"Field (.+?) not nullable","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java","lineNumber":1019,"sourceCode":"\n  /**\n   * Tries to convert an Avro decoded value to a Beam field value based on the target type of the\n   * Beam field.\n   *\n   * <p>For the Avro formats of BigQuery types, see\n   * https://cloud.google.com/bigquery/docs/exporting-data#avro_export_details and\n   * https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#avro_conversions\n   */\n  public static @Nullable Object convertAvroFormat(\n      FieldType beamFieldType,\n      @Nullable Object avroValue,\n      BigQueryUtils.ConversionOptions options) {\n    TypeName beamFieldTypeName = beamFieldType.getTypeName();\n    if (avroValue == null) {\n      if (beamFieldType.getNullable()) {\n        return null;\n      } else {\n        throw new IllegalArgumentException(String.format(\"Field %s not nullable\", beamFieldType));\n      }\n    }\n    switch (beamFieldTypeName) {\n      case BYTE:\n      case INT16:\n      case INT32:\n      case INT64:\n      case FLOAT:\n      case DOUBLE:\n      case STRING:\n      case BYTES:\n      case BOOLEAN:\n        return convertAvroPrimitiveTypes(beamFieldTypeName, avroValue);\n      case DATETIME:\n        // Expecting value in microseconds.\n        switch (options.getTruncateTimestamps()) {\n          case TRUNCATE:\n            return truncateToMillis(avroValue);","sourceCodeStart":1001,"sourceCodeEnd":1037,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L1001-L1037","documentation":"Thrown by BigQueryUtils conversion from Avro to Beam Row (avroGenericRecordToBeamRow path) when the Avro GenericRecord value is null but the corresponding Beam FieldType is non-nullable. The message includes the Beam field type. This mirrors the JSON-side null check: required Beam fields cannot be built from Avro nulls, so the library throws IllegalArgumentException.","triggerScenarios":"Calling BigQueryUtils.avroGenericRecordToBeamRow(schema, options, record) where record.get(fieldName) is null for a Beam field with nullable=false; typically when reading BigQuery export Avro files whose column is nullable at the Avro level but the Beam schema marks it REQUIRED.","commonSituations":"Reading BigQuery Storage Read / export-to-GCS Avro files where all columns are Avro-nullable, while the Beam schema (e.g. generated from a REQUIRED table schema or hand-written) expects non-null; legacy rows written before a column became REQUIRED.","solutions":["Mark the field nullable in the Beam Schema (FieldType.withNullable(true)) since BigQuery Avro representations are typically nullable.","Fill defaults for null Avro values before conversion (record.put(field, default) or a Map/DoFn step).","Filter records containing nulls in required fields before avroGenericRecordToBeamRow.","Generate the Beam schema with fromAvroSchema / fromTableSchema so nullability derives from the source."],"exampleFix":"// before\nField f = Field.of(\"user_id\", FieldType.INT64); // Avro value often null\n\n// after\nField f = Field.of(\"user_id\", FieldType.INT64.withNullable(true));\n// or pre-fill: if (record.get(\"user_id\") == null) record.put(\"user_id\", 0L);","handlingStrategy":"validation","validationCode":"// Java: validate Avro record nulls vs Beam schema before conversion\nfor (Schema.Field f : beamSchema.getFields()) {\n  if (!f.getType().getNullable() && record.get(f.getName()) == null) {\n    throw new IllegalArgumentException(\"Avro null in required field: \" + f.getName());\n  }\n}","typeGuard":"// Java\nstatic boolean avroFieldNonNull(GenericRecord r, String name) {\n  return r.get(name) != null;\n}","tryCatchPattern":"try {\n  Row row = BigQueryUtils.avroGenericRecordToBeamRow(beamSchema, options, record);\n} catch (IllegalArgumentException e) {\n  // dead-letter record; message names the offending field type\n}","preventionTips":["Remember BigQuery Avro exports mark all columns nullable — set withNullable(true) unless you guarantee values.","Pre-scan exported Avro files for nulls in required columns during pipeline dry runs.","Use Avro schema evolution checks when BigQuery table schemas change over time."],"tags":["java","apache-beam","bigquery","avro","nullability"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}