{"record":{"id":"ab15e34d95c9dc65","repo":"apache/beam","slug":"received-null-value-for-non-nullable-field-fielddescriptor","errorCode":null,"errorMessage":"Received null value for non-nullable field \" + fieldDescriptor.getName()","messagePattern":"Received null value for non-nullable field \" \\+ fieldDescriptor\\.getName\\(\\)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroGenericRecordToStorageApiProto.java","lineNumber":461,"sourceCode":"      }\n    }\n    if (field.doc() != null) {\n      builder = builder.setDescription(field.doc());\n    }\n    return builder.build();\n  }\n\n  @Nullable\n  private static Object messageValueFromGenericRecordValue(\n      FieldDescriptor fieldDescriptor, Schema.Field avroField, String name, GenericRecord record) {\n    @Nullable Object value = record.get(name);\n    if (value == null) {\n      if (fieldDescriptor.isOptional()\n          || avroField.schema().getTypes().stream()\n              .anyMatch(t -> t.getType() == Schema.Type.NULL)) {\n        return null;\n      } else {\n        throw new IllegalArgumentException(\n            \"Received null value for non-nullable field \" + fieldDescriptor.getName());\n      }\n    }\n    return toProtoValue(fieldDescriptor, avroField.schema(), value);\n  }\n\n  private static Object toProtoValue(\n      FieldDescriptor fieldDescriptor, Schema avroSchema, Object value) {\n    switch (avroSchema.getType()) {\n      case RECORD:\n        return messageFromGenericRecord(\n            fieldDescriptor.getMessageType(), (GenericRecord) value, null, -1);\n      case ARRAY:\n        Iterable<Object> iterable = (Iterable<Object>) value;\n        @Nullable Schema arrayElementType = avroSchema.getElementType();\n        if (arrayElementType == null) {\n          throw new RuntimeException(\"Unexpected null element type!\");\n        }","sourceCodeStart":443,"sourceCodeEnd":479,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroGenericRecordToStorageApiProto.java#L443-L479","documentation":"messageValueFromGenericRecordValue throws IllegalArgumentException when a null value arrives for a BigQuery Storage API field that is neither optional in the proto descriptor nor nullable in the Avro union schema - i.e. a required field received null.","triggerScenarios":"Converting a GenericRecord to proto where a field is null, the TableFieldSchema is not OPTIONAL (REQUIRED mode), and the Avro field's union does not include NULL.","commonSituations":"Upstream data producing nulls for columns declared REQUIRED in BigQuery; Avro schema says non-nullable but actual records contain nulls (schema/data drift); backfills of legacy data.","solutions":["Declare the BigQuery column NULLABLE (or mark the Avro field as [\"null\",\"type\"])","Cleanse/replace null values with defaults before writing","Coerce upstream producers to always populate the field"],"exampleFix":"// before\n{\"name\":\"id\",\"type\":\"long\"} // non-nullable, but records may contain null\n// after\n{\"name\":\"id\",\"type\":[\"null\",\"long\"],\"default\":null} // or filter null records before the sink","handlingStrategy":"validation","validationCode":"if (record.get(fieldName) == null && fieldMode == Field.Mode.REQUIRED) throw new IllegalStateException(\"null for REQUIRED field \" + fieldName);","typeGuard":"boolean nullSafeForBq(org.apache.avro.generic.GenericRecord r, org.apache.avro.Schema.Field f) { return r.get(f.name()) != null || f.schema().getTypes().stream().anyMatch(t -> t.getType() == org.apache.avro.Schema.Type.NULL); }","tryCatchPattern":"try { proto = AvroGenericRecordToStorageApiProto.messageValueFromGenericRecordValue(fieldDescriptor, avroField, record); } catch (IllegalArgumentException e) { if (e.getMessage().contains(\"non-nullable field\")) { applyDefaultOrDrop(); } else throw e; }","preventionTips":["Make BigQuery columns NULLABLE when upstream nulls are possible","Declare Avro unions with null for optional data","Scrub/cleanse nulls before the Storage Write sink"],"tags":["java","apache-beam","bigquery","avro","null"],"backgroundTag":"null-argument","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}