apache/beam · error · ValueError

Received 'None' as the value for the field

Error message

Received 'None' as the value for the field %s but the field is not NULLABLE.

What it means

Raised by `convert_row_to_dict` when a cell value is None but the schema field's mode is REQUIRED (not NULLABLE/REPEATED). BigQuery should not return NULL for a REQUIRED field, so Beam treats the response as inconsistent with the declared schema and raises ValueError. This detects corrupted or mismatched schema/row data.

Solutions

Refresh the schema Beam uses — fetch it from the live table instead of a hardcoded/serialized REQUIRED schema.
Declare the field as NULLABLE in the schema passed to Beam (BigQuery already relaxed it).
Filter/handle nulls in the query: IFNULL(col, default) or WHERE col IS NOT NULL.
Coalesce defaults in downstream code after converting rows if nulls are expected and legitimate.

Example fix

// before
bigquery.TableFieldSchema(name='user_id', type='STRING', mode='REQUIRED')
// after
bigquery.TableFieldSchema(name='user_id', type='STRING', mode='NULLABLE')

Defensive patterns

Strategy: validation

Validate before calling

def modes_match(live_schema, beam_schema):
    live = {f.name: f.mode for f in live_schema.fields}
    return all(f.mode == 'NULLABLE' or live.get(f.name) == f.mode for f in beam_schema.fields)

Try / catch

try:
    d = convert_row_to_dict(row, schema)
except ValueError as e:
    if 'not NULLABLE' in str(e):
        log.error('schema drift: REQUIRED field now receives NULL; refresh schema')
    raise

Prevention

Prefer NULLABLE mode in Beam schemas; treat REQUIRED as legacy.
Fetch the live table schema instead of hardcoding field modes.
Watch for BigQuery schema relaxations (REQUIRED -> NULLABLE) and update pipelines accordingly.

When it happens

Trigger: Parsing query/read results where a REQUIRED field contains null — typically after the field's mode was changed from NULLABLE/REQUIRED in the table (BigQuery allows relaxing REQUIRED to NULLABLE), leaving Beam with a stale REQUIRED schema while data now contains NULLs.

Common situations: Schema evolution: columns relaxed from REQUIRED to NULLABLE in BigQuery while a cached/derived Beam schema still says REQUIRED; manually constructed schema objects declaring REQUIRED wrongly; legacy tables with NULLs backfilled after schema relaxation.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ca88dbf0690b604c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:1407

      value = None
      if isinstance(schema, bigquery.TableSchema):
        cell = row.f[index]
        value = from_json_value(cell.v) if cell.v is not None else None
      elif isinstance(schema, bigquery.TableFieldSchema):
        cell = row['f'][index]
        value = cell['v'] if 'v' in cell else None
      if field.mode == 'REPEATED':
        if value is None:
          # Ideally this should never happen as repeated fields default to
          # returning an empty list
          result[field.name] = []
        else:
          result[field.name] = [
              self._convert_cell_value_to_dict(x['v'], field) for x in value
          ]
      elif value is None:
        if not field.mode == 'NULLABLE':
          raise ValueError(
              'Received \'None\' as the value for the field %s '
              'but the field is not NULLABLE.' % field.name)
        result[field.name] = None
      else:
        result[field.name] = self._convert_cell_value_to_dict(value, field)
    return result

  @staticmethod
  def from_pipeline_options(pipeline_options: PipelineOptions):
    return BigQueryWrapper(
        client=BigQueryWrapper._bigquery_client(pipeline_options))

  @staticmethod
  def _bigquery_client(pipeline_options: PipelineOptions):
    return bigquery.BigqueryV2(
        http=get_new_http(),
        credentials=auth.get_service_credentials(pipeline_options),
        response_encoding='utf8',

View on GitHub (pinned to 12126d8942)