{"record":{"id":"fafd04229b6ad39e","repo":"apache/beam","slug":"missing-or-unknown-property-required","errorCode":null,"errorMessage":"Missing or unknown property '{required}'","messagePattern":"Missing or unknown property '(.+?)'","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/yaml/json_utils.py","lineNumber":331,"sourceCode":"  if not weak_schema:\n    return\n  if weak_schema['type'] != strong_schema['type']:\n    raise ValueError(\n        f\"Incompatible types: {weak_schema['type']} vs {strong_schema['type']}\")\n  if weak_schema['type'] == 'array':\n    _validate_compatible(weak_schema['items'], strong_schema['items'])\n  elif weak_schema['type'] == 'object':\n    # If the weak schema allows for arbitrary keys (is a map),\n    # the strong schema must also allow for arbitrary keys.\n    if weak_schema.get('additionalProperties'):\n      if not strong_schema.get('additionalProperties', True):\n        raise ValueError('Incompatible types: map vs object')\n      _validate_compatible(\n          weak_schema['additionalProperties'],\n          strong_schema['additionalProperties'])\n    for required in strong_schema.get('required', []):\n      if required not in weak_schema['properties']:\n        raise ValueError(f\"Missing or unknown property '{required}'\")\n    for name, spec in weak_schema.get('properties', {}).items():\n\n      if name in strong_schema['properties']:\n        try:\n          _validate_compatible(spec, strong_schema['properties'][name])\n        except Exception as exn:\n          raise ValueError(f\"Incompatible schema for '{name}'\") from exn\n      elif not strong_schema.get('additionalProperties', True):\n        # The property is not explicitly in the strong schema, and the strong\n        # schema does not allow for extra properties.\n        raise ValueError(\n            f\"Prohibited property: '{name}'; \"\n            \"perhaps additionalProperties: False is missing?\")\n\n\ndef row_validator(beam_schema: schema_pb2.Schema,\n                  json_schema: dict[str, Any]) -> Callable[[Any], Any]:\n  \"\"\"Returns a callable that will fail on elements not respecting json_schema.","sourceCodeStart":313,"sourceCodeEnd":349,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/yaml/json_utils.py#L313-L349","documentation":"During schema compatibility checking, every property listed as 'required' in the strong schema must exist in the weak schema's 'properties'. If a required property is absent from the weak schema, this ValueError names the missing property.","triggerScenarios":"row_validator where strong_schema['required'] contains a key (e.g. 'id') that is not present in weak_schema['properties'].","commonSituations":"Declaring required fields in a JSON schema that the producing transform never outputs; typos in property names; schema drift after refactoring a YAML pipeline's transform outputs.","solutions":["Add the required property to the weak schema's properties (and make sure the producing transform actually emits it).","Remove the property from the strong schema's 'required' list if it is not actually produced.","Fix typos so the required name matches an existing property key exactly.","Run the validator locally (row_validator) on both schemas to enumerate all missing properties before launching the pipeline."],"exampleFix":"# before\nweak = {'type': 'object', 'properties': {'name': {'type': 'string'}}}\nstrong = {'type': 'object', 'properties': {'id': {'type': 'string'}}, 'required': ['id']}\n# after\nweak = {'type': 'object', 'properties': {'id': {'type': 'string'}, 'name': {'type': 'string'}}}","handlingStrategy":"validation","validationCode":"def required_present(weak, strong):\n    props = weak.get('properties', {})\n    missing = [r for r in strong.get('required', []) if r not in props]\n    return missing\nmissing = required_present(weak_schema, strong_schema)\nassert not missing, f'missing required: {missing}'","typeGuard":"def is_object_schema(s):\n    return isinstance(s, dict) and s.get('type') == 'object' and isinstance(s.get('properties', {}), dict)","tryCatchPattern":"try:\n    row_validator(beam_schema, json_schema)\nexcept ValueError as e:\n    if str(e).startswith(\"Missing or unknown property\"):\n        log.error('Producer does not declare required property: %s', e)\n    raise","preventionTips":["Only mark fields 'required' after confirming the producing transform emits them.","Grep the producing transform's output fields and cross-check against required lists.","Keep one canonical schema file and generate the weak/strong variants from it."],"tags":["python","apache-beam","yaml","json-schema","required-property"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}