apache/beam · error · KeyError

Missing required field: {name}

Error message

Missing required field: {name}

What it means

In json_to_row's convert_row, when converting a JSON object to a Beam Row, any schema field absent from the JSON value must be nullable; if it is neither present in the input nor nullable, KeyError('Missing required field: ...') is raised. This enforces the schema's required-field contract at conversion time.

Source

Thrown at sdks/python/apache_beam/yaml/json_utils.py:215

    field_nullable_status = {
        field.name: field.type.nullable
        for field in beam_type.row_type.schema.fields
    }

    converters = {
        field.name: json_to_row(field.type)
        for field in beam_type.row_type.schema.fields
    }

    def convert_row(value):
      kwargs = {}
      for name, convert in converters.items():
        if name in value:
          kwargs[name] = convert(value[name])
        elif field_nullable_status[name]:
          kwargs[name] = convert(None)
        else:
          raise KeyError(f"Missing required field: {name}")
      return beam.Row(**kwargs)

    return convert_row
  elif type_info == "logical_type":
    return lambda value: value
  else:
    raise ValueError(f"Unrecognized type_info: {type_info!r}")


def json_parser(
    beam_schema: schema_pb2.Schema,
    json_schema: Optional[dict[str,
                               Any]] = None) -> Callable[[bytes], beam.Row]:
  """Returns a callable converting Json strings to Beam rows of the given type.

  The input to the returned callable is expected to conform to the Json schema
  corresponding to this Beam type.
  """

View on GitHub (pinned to 12126d8942)

Solutions

  1. Add the missing field to the JSON records at the source
  2. Mark the field optional: remove it from the JSON schema 'required' list (making it nullable in the Beam schema)
  3. Pre-validate records and reject/skip invalid ones before conversion
  4. Provide a default by enriching records in a prior step

Example fix

// before
{"type": "object", "required": ["id"], "properties": {"id": {"type": "integer"}, "name": {"type": "string"}}}
// after (make 'name' optional)
{"type": "object", "required": ["id"], "properties": {"id": {"type": "integer"}, "name": {"type": "string"}}, "required": ["id"]} // keep only truly required fields in required
Defensive patterns

Strategy: validation

Validate before calling

def assert_required_fields(records, beam_schema):
    required = [f.name for f in beam_schema.fields if not f.nullable]
    for i, rec in enumerate(records):
        missing = [n for n in required if n not in rec]
        if missing:
            raise KeyError(f'Record {i} missing required fields: {missing}')

Type guard

def record_satisfies(rec, beam_schema) -> bool:
    return all(f.name in rec or f.nullable for f in beam_schema.fields)

Try / catch

try:
    row = convert(value)
except KeyError as e:
    logging.warning('Dropping malformed record missing %s', e)
    return None

Prevention

When it happens

Trigger: JSON records missing a field that the Beam schema marks non-nullable (field not in the schema's 'required' complement), passed through json_to_row / json_parser, e.g. an event record missing 'id'.

Common situations: Upstream producers emitting partial records; schema declared in pipeline YAML stricter than the actual JSON data; version skew where new required fields were added to the schema before producers updated.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/69bf318d6d30297e. Report an issue: GitHub.