{"record":{"id":"a3ca9fefb616bb05","repo":"apache/beam","slug":"received-null-value-for-non-nullable-field-bigqueryutils","errorCode":null,"errorMessage":"Received null value for non-nullable field \"\"","messagePattern":"Received null value for non-nullable field \"\"","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java","lineNumber":904,"sourceCode":"\n    return IntStream.range(0, rowSchema.getFieldCount())\n        .boxed()\n        .<@Nullable Object>map(\n            index -> toBeamValue(rowSchema.getField(index), rawJsonValues.get(index)))\n        .collect(toRow(rowSchema));\n  }\n\n  private static @Nullable Object toBeamValue(Field field, @Nullable Object jsonBQValue) {\n    FieldType fieldType = field.getType();\n\n    if (jsonBQValue == null) {\n      if (fieldType.getNullable()) {\n        return null;\n      } else {\n        if (fieldType.getTypeName().isCollectionType()) {\n          return Collections.emptyList();\n        }\n        throw new IllegalArgumentException(\n            \"Received null value for non-nullable field \\\"\" + field.getName() + \"\\\"\");\n      }\n    }\n\n    if (jsonBQValue instanceof String\n        || jsonBQValue instanceof Number\n        || jsonBQValue instanceof Boolean) {\n      String jsonBQString = jsonBQValue.toString();\n      if (JSON_VALUE_PARSERS.containsKey(fieldType.getTypeName())) {\n        return JSON_VALUE_PARSERS.get(fieldType.getTypeName()).apply(jsonBQString);\n      } else if (fieldType.isLogicalType(SqlTypes.DATETIME.getIdentifier())) {\n        try {\n          // Handle if datetime value is in micros ie. 123456789\n          Long value = Long.parseLong(jsonBQString);\n          return CivilTimeEncoder.decodePacked64DatetimeMicrosAsJavaTime(value);\n        } catch (NumberFormatException e) {\n          // Handle as a String, ie. \"2023-02-16 12:00:00\"\n          try {","sourceCodeStart":886,"sourceCodeEnd":922,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L886-L922","documentation":"Thrown during BigQuery JSON -> Beam Row conversion (fromJsonToBeamField path in BigQueryUtils) when the JSON value for a field is null but the Beam FieldType is non-nullable. Collection-typed fields are converted to an empty list instead, but scalar required fields cause an immediate IllegalArgumentException naming the offending field. The library refuses to invent a value for a required scalar.","triggerScenarios":"Calling BigQueryUtils.toBeamRow(schema, tableRow) (or the JSON variant) where TableRow.get(field) returns null for a field whose Beam FieldType has nullable=false and is not a collection type.","commonSituations":"Reading from a BigQuery table whose column is NULLABLE/MODE NULLABLE while the Beam schema was generated as REQUIRED (or vice versa after a table schema change); querying with SELECT that omits a column expected by the schema; stale table schemas.","solutions":["Align the Beam schema with the actual BigQuery table schema: mark the field nullable (FieldType.withNullable(true)).","Use toBeamRow with a schema generated from the table via BigQueryUtils.fromTableSchema / fromBeamType so nullability matches.","Coalesce nulls upstream in the query (IFNULL(col, default)) or fill defaults in the TableRow before conversion.","Drop or filter rows containing nulls in required columns before calling toBeamRow."],"exampleFix":"// before\nField f = Field.of(\"amount\", FieldType.DOUBLE); // required, but table column NULLABLE\n\n// after\nField f = Field.of(\"amount\", FieldType.DOUBLE.withNullable(true));\n// or: row.set(\"amount\", row.get(\"amount\") == null ? 0.0 : row.get(\"amount\"));","handlingStrategy":"validation","validationCode":"// Java: check TableRow against Beam schema before toBeamRow\nschema.getFields().forEach(f -> {\n  if (!f.getType().getNullable()\n      && !f.getType().getTypeName().isCollectionType()\n      && tableRow.get(f.getName()) == null) {\n    throw new IllegalArgumentException(\"Missing required field: \" + f.getName());\n  }\n});","typeGuard":"// Java\nstatic boolean requiredFieldsPresent(TableRow tr, Schema schema) {\n  return schema.getFields().stream()\n      .filter(f -> !f.getType().getNullable())\n      .allMatch(f -> tr.containsKey(f.getName()) && tr.get(f.getName()) != null);\n}","tryCatchPattern":"try {\n  Row row = BigQueryUtils.toBeamRow(schema, tableRow);\n} catch (IllegalArgumentException e) {\n  // dead-letter tableRow with the exception message\n}","preventionTips":["Generate the Beam schema from the live table schema (fromTableSchema) instead of hand-writing it.","Use IFNULL/COALESCE in the BigQuery SQL to eliminate NULLs for required columns.","Re-derive schemas after any BigQuery table schema evolution."],"tags":["java","apache-beam","bigquery","nullability","table-row"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}