{"record":{"id":"ca88dbf0690b604c","repo":"apache/beam","slug":"received-none-as-the-value-for-the-field-s-but-the-field-is","errorCode":null,"errorMessage":"Received 'None' as the value for the field %s but the field is not NULLABLE.","messagePattern":"Received 'None' as the value for the field (.+?) but the field is not NULLABLE\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery_tools.py","lineNumber":1407,"sourceCode":"      value = None\n      if isinstance(schema, bigquery.TableSchema):\n        cell = row.f[index]\n        value = from_json_value(cell.v) if cell.v is not None else None\n      elif isinstance(schema, bigquery.TableFieldSchema):\n        cell = row['f'][index]\n        value = cell['v'] if 'v' in cell else None\n      if field.mode == 'REPEATED':\n        if value is None:\n          # Ideally this should never happen as repeated fields default to\n          # returning an empty list\n          result[field.name] = []\n        else:\n          result[field.name] = [\n              self._convert_cell_value_to_dict(x['v'], field) for x in value\n          ]\n      elif value is None:\n        if not field.mode == 'NULLABLE':\n          raise ValueError(\n              'Received \\'None\\' as the value for the field %s '\n              'but the field is not NULLABLE.' % field.name)\n        result[field.name] = None\n      else:\n        result[field.name] = self._convert_cell_value_to_dict(value, field)\n    return result\n\n  @staticmethod\n  def from_pipeline_options(pipeline_options: PipelineOptions):\n    return BigQueryWrapper(\n        client=BigQueryWrapper._bigquery_client(pipeline_options))\n\n  @staticmethod\n  def _bigquery_client(pipeline_options: PipelineOptions):\n    return bigquery.BigqueryV2(\n        http=get_new_http(),\n        credentials=auth.get_service_credentials(pipeline_options),\n        response_encoding='utf8',","sourceCodeStart":1389,"sourceCodeEnd":1425,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L1389-L1425","documentation":"Raised by `convert_row_to_dict` when a cell value is None but the schema field's mode is REQUIRED (not NULLABLE/REPEATED). BigQuery should not return NULL for a REQUIRED field, so Beam treats the response as inconsistent with the declared schema and raises ValueError. This detects corrupted or mismatched schema/row data.","triggerScenarios":"Parsing query/read results where a REQUIRED field contains null — typically after the field's mode was changed from NULLABLE/REQUIRED in the table (BigQuery allows relaxing REQUIRED to NULLABLE), leaving Beam with a stale REQUIRED schema while data now contains NULLs.","commonSituations":"Schema evolution: columns relaxed from REQUIRED to NULLABLE in BigQuery while a cached/derived Beam schema still says REQUIRED; manually constructed schema objects declaring REQUIRED wrongly; legacy tables with NULLs backfilled after schema relaxation.","solutions":["Refresh the schema Beam uses — fetch it from the live table instead of a hardcoded/serialized REQUIRED schema.","Declare the field as NULLABLE in the schema passed to Beam (BigQuery already relaxed it).","Filter/handle nulls in the query: IFNULL(col, default) or WHERE col IS NOT NULL.","Coalesce defaults in downstream code after converting rows if nulls are expected and legitimate."],"exampleFix":"// before\nbigquery.TableFieldSchema(name='user_id', type='STRING', mode='REQUIRED')\n// after\nbigquery.TableFieldSchema(name='user_id', type='STRING', mode='NULLABLE')","handlingStrategy":"validation","validationCode":"def modes_match(live_schema, beam_schema):\n    live = {f.name: f.mode for f in live_schema.fields}\n    return all(f.mode == 'NULLABLE' or live.get(f.name) == f.mode for f in beam_schema.fields)","typeGuard":null,"tryCatchPattern":"try:\n    d = convert_row_to_dict(row, schema)\nexcept ValueError as e:\n    if 'not NULLABLE' in str(e):\n        log.error('schema drift: REQUIRED field now receives NULL; refresh schema')\n    raise","preventionTips":["Prefer NULLABLE mode in Beam schemas; treat REQUIRED as legacy.","Fetch the live table schema instead of hardcoding field modes.","Watch for BigQuery schema relaxations (REQUIRED -> NULLABLE) and update pipelines accordingly."],"tags":["bigquery","python","schema","null"],"backgroundTag":"unexpected-response-shape","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}