apache/beam · error · ValueError

Missing or unknown property '{required}'

Error message

Missing or unknown property '{required}'

What it means

During schema compatibility checking, every property listed as 'required' in the strong schema must exist in the weak schema's 'properties'. If a required property is absent from the weak schema, this ValueError names the missing property.

Source

Thrown at sdks/python/apache_beam/yaml/json_utils.py:331

  if not weak_schema:
    return
  if weak_schema['type'] != strong_schema['type']:
    raise ValueError(
        f"Incompatible types: {weak_schema['type']} vs {strong_schema['type']}")
  if weak_schema['type'] == 'array':
    _validate_compatible(weak_schema['items'], strong_schema['items'])
  elif weak_schema['type'] == 'object':
    # If the weak schema allows for arbitrary keys (is a map),
    # the strong schema must also allow for arbitrary keys.
    if weak_schema.get('additionalProperties'):
      if not strong_schema.get('additionalProperties', True):
        raise ValueError('Incompatible types: map vs object')
      _validate_compatible(
          weak_schema['additionalProperties'],
          strong_schema['additionalProperties'])
    for required in strong_schema.get('required', []):
      if required not in weak_schema['properties']:
        raise ValueError(f"Missing or unknown property '{required}'")
    for name, spec in weak_schema.get('properties', {}).items():

      if name in strong_schema['properties']:
        try:
          _validate_compatible(spec, strong_schema['properties'][name])
        except Exception as exn:
          raise ValueError(f"Incompatible schema for '{name}'") from exn
      elif not strong_schema.get('additionalProperties', True):
        # The property is not explicitly in the strong schema, and the strong
        # schema does not allow for extra properties.
        raise ValueError(
            f"Prohibited property: '{name}'; "
            "perhaps additionalProperties: False is missing?")


def row_validator(beam_schema: schema_pb2.Schema,
                  json_schema: dict[str, Any]) -> Callable[[Any], Any]:
  """Returns a callable that will fail on elements not respecting json_schema.

View on GitHub (pinned to 12126d8942)

Solutions

1. Add the required property to the weak schema's properties (and make sure the producing transform actually emits it).
2. Remove the property from the strong schema's 'required' list if it is not actually produced.
3. Fix typos so the required name matches an existing property key exactly.
4. Run the validator locally (row_validator) on both schemas to enumerate all missing properties before launching the pipeline.

Example fix

# before
weak = {'type': 'object', 'properties': {'name': {'type': 'string'}}}
strong = {'type': 'object', 'properties': {'id': {'type': 'string'}}, 'required': ['id']}
# after
weak = {'type': 'object', 'properties': {'id': {'type': 'string'}, 'name': {'type': 'string'}}}

def required_present(weak, strong):
    props = weak.get('properties', {})
    missing = [r for r in strong.get('required', []) if r not in props]
    return missing
missing = required_present(weak_schema, strong_schema)
assert not missing, f'missing required: {missing}'

def is_object_schema(s):
    return isinstance(s, dict) and s.get('type') == 'object' and isinstance(s.get('properties', {}), dict)

try:
    row_validator(beam_schema, json_schema)
except ValueError as e:
    if str(e).startswith("Missing or unknown property"):
        log.error('Producer does not declare required property: %s', e)
    raise

When it happens

Trigger: row_validator where strong_schema['required'] contains a key (e.g. 'id') that is not present in weak_schema['properties'].

Common situations: Declaring required fields in a JSON schema that the producing transform never outputs; typos in property names; schema drift after refactoring a YAML pipeline's transform outputs.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Incompatible types: {weak_schema['type']} vs {strong_schema[
• Incompatible types: map vs object
• Incompatible schema for '{name}'
• Prohibited property: '{name}'; perhaps additionalProperties:
• Node ID cannot be empty

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/fafd04229b6ad39e. Report an issue: GitHub.