What causes "Unknown BigQuery field mode: {}" in apache/beam?

Inspect the offending table_field['mode'] value and set it to one of 'NULLABLE', 'REQUIRED', or 'REPEATED' (exact case).

apache/beam · error · ValueError

Unknown BigQuery field mode: {}

Error message

Unknown BigQuery field mode: {}

What it means

table_field_to_avro_field converts a BigQuery table schema field into an Avro field schema. BigQuery field modes must be NULLABLE, REQUIRED, or REPEATED; any other mode string cannot be mapped to an Avro type, so the function raises ValueError. This indicates malformed or unexpected schema metadata rather than a data problem.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_avro_tools.py:134

  if avro_type == "record":
    element_type = get_record_schema_from_dict_table_schema(
        table_field["name"],
        table_field,
        namespace=".".join((namespace, table_field["name"])))
  else:
    element_type = avro_type

  field_mode = table_field.get("mode", "NULLABLE")

  if field_mode in (None, "NULLABLE"):
    field_type = ["null", element_type]
  elif field_mode == "REQUIRED":
    field_type = element_type
  elif field_mode == "REPEATED":
    field_type = {"type": "array", "items": element_type}
  else:
    raise ValueError("Unknown BigQuery field mode: {}".format(field_mode))

  avro_field = {"type": field_type, "name": table_field["name"]}

  doc = table_field.get("description")
  if doc:
    avro_field["doc"] = doc

  return avro_field

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the offending table_field['mode'] value and set it to one of 'NULLABLE', 'REQUIRED', or 'REPEATED' (exact case).
If mode is absent, add an explicit mode to each field in the dict table schema (BigQuery defaults to NULLABLE).
Fetch the schema via a supported BigQuery client/API (e.g. client.get_table(table).to_api_repr()) instead of hand-writing it.
If constructing programmatically, validate modes against {'NULLABLE','REQUIRED','REPEATED'} before passing the schema in.

Example fix

# before
field = {"name": "ids", "type": "INTEGER", "mode": "repeated"}
# after
field = {"name": "ids", "type": "INTEGER", "mode": "REPEATED"}

Defensive patterns

Strategy: validation

Validate before calling

VALID_MODES = {"NULLABLE", "REQUIRED", "REPEATED"}
assert all(f.get("mode") in VALID_MODES for f in table_schema["fields"]), \
    "table schema field mode must be NULLABLE, REQUIRED, or REPEATED"

Type guard

def has_valid_modes(schema: dict) -> bool:
    return all(f.get("mode") in ("NULLABLE", "REQUIRED", "REPEATED") for f in schema.get("fields", []))

Try / catch

try:
    avro_schema = get_record_schema_from_dict_table_table(table_schema)
except ValueError as e:
    log.error("invalid BigQuery field mode: %s", e)
    raise

Prevention

Use client.get_table(...).to_api_repr() to obtain schemas instead of hand-writing them
Validate mode strings against the exact uppercase enum before conversion
Never omit 'mode' when constructing dict table schemas manually

When it happens

Trigger: Calling get_record_schema_from_dict_table_schema with a dict-based BigQuery table schema whose field has a 'mode' value that is not exactly 'NULLABLE', 'REQUIRED', or 'REPEATED' (e.g. 'Optional', lowercase 'nullable', empty string, or None for a field where the nested element_type path was taken).

Common situations: Hand-constructed or third-party-generated table schemas with missing or misspelled 'mode'; schemas fetched with a client that returns 'REPEATED' differently; typos when writing dict table schemas by hand.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8330c32f049183f1. Report an issue: GitHub.