apache/beam · error · TypeError

violates schema

Error message

{result} violates schema {explicit_type}

What it means

When a MapToFields field declares `output_type`, the resulting checking_func validates every produced value against the declared Beam type at runtime and raises this TypeError if the UDF's result does not satisfy the schema validator. It is a per-element data error, not a config error.

Solutions

1. Make the UDF coerce its result to the declared type, e.g. `str(x)` or `int(x)`.
2. Handle None explicitly: use `expression: "x if x is not None else ''"` or declare nullable output_type if supported.
3. Relax or correct the output_type declaration to match what the function actually returns.
4. Attach error_handling config so bad rows go to an error output instead of failing the pipeline.

Example fix

# before (may return None)
fields:
  name:
    expression: "row.get('name')"
    output_type: string
# after
fields:
  name:
    expression: "row.get('name', '')"
    output_type: string

# sanity-test the UDF against the declared type on sample data
result = udf(sample_row)
assert isinstance(result, str), f'{result!r} does not match output_type'

def matches_output_type(result, declared: str) -> bool:
    checks = {'string': str, 'integer': int, 'number': (int, float), 'boolean': bool, 'bytes': bytes}
    return isinstance(result, checks.get(declared, object)) and result is not None

config:
  error_handling:
    output: errors
# then consume MyTransform.errors downstream so TypeError rows are captured, not fatal

When it happens

Trigger: A mapping function returns a value that fails the declared output_type validator at runtime, e.g. declaring output_type: string while the expression returns None or an int; also triggered inside try/except error handling wrappers for failing rows.

Common situations: Expressions returning None for missing data (None violates every declared type), int-vs-float mismatches, forgetting that 'number' accepts int/float but 'integer' rejects floats, arrays containing elements of the wrong element_type.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Can only use expressions on a schema'd input.
• Config for transform at
• Converting YAML type
• The input to this transform does not appear to be an error…
• Unknown grouping columns

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ef2bb9ff8ecd0d8d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_mapping.py:427

    func = _expand_javascript_mapping_func(original_fields, **expr)
  elif language in ("python", "generic", None):
    func = _expand_python_mapping_func(original_fields, **expr)
  else:
    raise ValueError(
        f'Unknown language for mapping transform: {language}. '
        'Supported languages are "javascript" and "python."')

  if explicit_type:
    if isinstance(explicit_type, str):
      explicit_type = {'type': explicit_type}
    beam_type = json_utils.json_type_to_beam_type(explicit_type)
    validator = _validator(beam_type)

    @beam.typehints.with_output_types(schemas.typing_from_runner_api(beam_type))
    def checking_func(row):
      result = func(row)
      if not validator(result):
        raise TypeError(f'{result} violates schema {explicit_type}')
      return result

    return checking_func

  elif original_type:
    return beam.typehints.with_output_types(
        convert_to_beam_type(original_type))(
            func)

  else:
    return func


class _StripErrorMetadata(beam.PTransform):
  """Strips error metadata from outputs returned via error handling.

  Generally the error outputs for transformations return information about
  the error encountered (e.g. error messages and tracebacks) in addition to the

View on GitHub (pinned to 12126d8942)