apache/beam · error · ValueError

Incompatible schema for '{name}'

Error message

Incompatible schema for '{name}'

What it means

When comparing a property present in both weak and strong schemas, any incompatibility raised by the recursive _validate_compatible call is re-raised as "Incompatible schema for '<name>'" with the original error chained as __cause__, so the developer knows which named field failed and why.

Source

Thrown at sdks/python/apache_beam/yaml/json_utils.py:338

  elif weak_schema['type'] == 'object':
    # If the weak schema allows for arbitrary keys (is a map),
    # the strong schema must also allow for arbitrary keys.
    if weak_schema.get('additionalProperties'):
      if not strong_schema.get('additionalProperties', True):
        raise ValueError('Incompatible types: map vs object')
      _validate_compatible(
          weak_schema['additionalProperties'],
          strong_schema['additionalProperties'])
    for required in strong_schema.get('required', []):
      if required not in weak_schema['properties']:
        raise ValueError(f"Missing or unknown property '{required}'")
    for name, spec in weak_schema.get('properties', {}).items():

      if name in strong_schema['properties']:
        try:
          _validate_compatible(spec, strong_schema['properties'][name])
        except Exception as exn:
          raise ValueError(f"Incompatible schema for '{name}'") from exn
      elif not strong_schema.get('additionalProperties', True):
        # The property is not explicitly in the strong schema, and the strong
        # schema does not allow for extra properties.
        raise ValueError(
            f"Prohibited property: '{name}'; "
            "perhaps additionalProperties: False is missing?")


def row_validator(beam_schema: schema_pb2.Schema,
                  json_schema: dict[str, Any]) -> Callable[[Any], Any]:
  """Returns a callable that will fail on elements not respecting json_schema.
  """
  if not json_schema:
    return lambda x: None

  # Validate that this compiles, but avoid pickling the validator itself.
  _ = jsonschema.validators.validator_for(json_schema)(json_schema)
  _validate_compatible(beam_schema_to_json_schema(beam_schema), json_schema)

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the chained __cause__ (printed traceback shows 'The above exception was the direct cause...') to find the underlying nested mismatch.
2. Fix the named property's schema in either the weak or strong schema so types/requireds align.
3. Keep weak and strong schema definitions in sync, ideally deriving one from the other instead of duplicating them.
4. Validate schemas in a unit test (test_validate_compatible-style) before deploying the pipeline.

Example fix

# before
weak = {'type': 'object', 'properties': {'ts': {'type': 'string'}}}
strong = {'type': 'object', 'properties': {'ts': {'type': 'integer'}}}
# after
weak = {'type': 'object', 'properties': {'ts': {'type': 'integer'}}}

for name, spec in weak_schema.get('properties', {}).items():
    if name in strong_schema['properties']:
        try:
            _validate_compatible(spec, strong_schema['properties'][name])
        except Exception as exn:
            print(f'property {name}: {exn!r}')

def nested_schemas_share_shape(weak, strong, name):
    return name in weak.get('properties', {}) and name in strong.get('properties', {})

try:
    row_validator(beam_schema, json_schema)
except ValueError as e:
    if str(e).startswith('Incompatible schema for'):
        log.error('%s | cause: %r', e, e.__cause__)
    raise

When it happens

Trigger: row_validator where for some property 'name' present in both schemas, _validate_compatible(spec_weak, spec_strong) raises (type mismatch, nested required missing, prohibited property, etc.) — the message wraps it.

Common situations: A nested field changed type (e.g. string to int) in one schema but not the other; nested required/properties inconsistencies; deeply nested schema drift in Beam YAML pipelines.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Incompatible types: {weak_schema['type']} vs {strong_schema[
• Incompatible types: map vs object
• Missing or unknown property '{required}'
• Prohibited property: '{name}'; perhaps additionalProperties:
• Node ID cannot be empty

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e0042889ddfd7caa. Report an issue: GitHub.