{"record":{"id":"69bf318d6d30297e","repo":"apache/beam","slug":"missing-required-field-name","errorCode":null,"errorMessage":"Missing required field: {name}","messagePattern":"Missing required field: (.+?)","errorType":"exception","errorClass":"KeyError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/yaml/json_utils.py","lineNumber":215,"sourceCode":"    field_nullable_status = {\n        field.name: field.type.nullable\n        for field in beam_type.row_type.schema.fields\n    }\n\n    converters = {\n        field.name: json_to_row(field.type)\n        for field in beam_type.row_type.schema.fields\n    }\n\n    def convert_row(value):\n      kwargs = {}\n      for name, convert in converters.items():\n        if name in value:\n          kwargs[name] = convert(value[name])\n        elif field_nullable_status[name]:\n          kwargs[name] = convert(None)\n        else:\n          raise KeyError(f\"Missing required field: {name}\")\n      return beam.Row(**kwargs)\n\n    return convert_row\n  elif type_info == \"logical_type\":\n    return lambda value: value\n  else:\n    raise ValueError(f\"Unrecognized type_info: {type_info!r}\")\n\n\ndef json_parser(\n    beam_schema: schema_pb2.Schema,\n    json_schema: Optional[dict[str,\n                               Any]] = None) -> Callable[[bytes], beam.Row]:\n  \"\"\"Returns a callable converting Json strings to Beam rows of the given type.\n\n  The input to the returned callable is expected to conform to the Json schema\n  corresponding to this Beam type.\n  \"\"\"","sourceCodeStart":197,"sourceCodeEnd":233,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/yaml/json_utils.py#L197-L233","documentation":"In json_to_row's convert_row, when converting a JSON object to a Beam Row, any schema field absent from the JSON value must be nullable; if it is neither present in the input nor nullable, KeyError('Missing required field: ...') is raised. This enforces the schema's required-field contract at conversion time.","triggerScenarios":"JSON records missing a field that the Beam schema marks non-nullable (field not in the schema's 'required' complement), passed through json_to_row / json_parser, e.g. an event record missing 'id'.","commonSituations":"Upstream producers emitting partial records; schema declared in pipeline YAML stricter than the actual JSON data; version skew where new required fields were added to the schema before producers updated.","solutions":["Add the missing field to the JSON records at the source","Mark the field optional: remove it from the JSON schema 'required' list (making it nullable in the Beam schema)","Pre-validate records and reject/skip invalid ones before conversion","Provide a default by enriching records in a prior step"],"exampleFix":"// before\n{\"type\": \"object\", \"required\": [\"id\"], \"properties\": {\"id\": {\"type\": \"integer\"}, \"name\": {\"type\": \"string\"}}}\n// after (make 'name' optional)\n{\"type\": \"object\", \"required\": [\"id\"], \"properties\": {\"id\": {\"type\": \"integer\"}, \"name\": {\"type\": \"string\"}}, \"required\": [\"id\"]} // keep only truly required fields in required","handlingStrategy":"validation","validationCode":"def assert_required_fields(records, beam_schema):\n    required = [f.name for f in beam_schema.fields if not f.nullable]\n    for i, rec in enumerate(records):\n        missing = [n for n in required if n not in rec]\n        if missing:\n            raise KeyError(f'Record {i} missing required fields: {missing}')","typeGuard":"def record_satisfies(rec, beam_schema) -> bool:\n    return all(f.name in rec or f.nullable for f in beam_schema.fields)","tryCatchPattern":"try:\n    row = convert(value)\nexcept KeyError as e:\n    logging.warning('Dropping malformed record missing %s', e)\n    return None","preventionTips":["Keep the JSON schema 'required' list in sync with what producers actually emit","Mark fields nullable whenever upstream data can be partial","Pre-validate incoming records at the source transform","Add new required fields to producers before tightening the schema"],"tags":["python","apache-beam","json","schema","missing-field"],"backgroundTag":"missing-required-argument","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}