apache/beam · error · TypeError

element types has different number of fields

Error message

element types has different number of fields

What it means

apache_beam's union_schema_type() builds a schema for a Union of named-tuple-like element types by requiring every element type to have the same number of fields. Before zipping fields across element types it compares each type's field count with the previous one's and raises TypeError when they differ. This guards against silently producing a malformed union schema.

Solutions

  1. Make every element type in the Union have the same number of fields with identical names and ordering.
  2. Remove the mismatched type from the Union, or widen the narrower type to match the wider one.
  3. Convert the types to a common schema explicitly (e.g. use beam.Row or a single NamedTuple) instead of relying on union schema inference.

Example fix

// before
class EventV1(NamedTuple):
  id: int

class EventV2(NamedTuple):
  id: int
  ts: str

# Union[EventV1, EventV2] -> TypeError

// after
class EventV1(NamedTuple):
  id: int
  ts: Optional[str] = None

class EventV2(NamedTuple):
  id: int
  ts: str

# Union[EventV1, EventV2]
Defensive patterns

Strategy: validation

Validate before calling

from typing import get_type_hints, NamedTuple

def check_same_field_count(types):
    counts = {len(t._fields) if issubclass(t, tuple) else None for t in types}
    if None in counts or len(counts) != 1:
        raise TypeError("Union element types must have equal field counts")

Type guard

def same_arity(types):
    return len(types) > 0 and all(hasattr(t, '_fields') and len(t._fields) == len(types[0]._fields) for t in types)

Try / catch

try:
    schema = union_schema_type(element_types)
except TypeError as e:
    if "different number of fields" in str(e):
        schema = None  # fall back to a common row schema
    else:
        raise

Prevention

When it happens

Trigger: Calling union_schema_type() (directly or via schema coercion of typing.Union[...]) with element types such as typing.NamedTuple classes or beam Rows that declare different numbers of fields, e.g. Union[Row(a=int), Row(a=int, b=str)] or Union[NamedTupleA(x,y), NamedTupleB(x)].

Common situations: Evolving a NamedTuple class in one place but not another version of the pipeline; mixing a dataclass/Row and a NamedTuple with different arity in a Union annotation on a PTransform; generating schemas from type hints after a schema migration changed field count.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/63fbd2e347d09119. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/typehints/schemas.py:731

        "https://s.apache.org/beam-python-schemas")


def named_fields_from_element_type(
    element_type: type) -> List[Tuple[str, type]]:
  return named_fields_from_schema(schema_from_element_type(element_type))


def union_schema_type(element_types):
  """Returns a schema whose fields are the union of each corresponding field.

  element_types must be a set of schema-aware types whose fields have the
  same naming and ordering.
  """
  named_fields_and_types = []
  for t in element_types:
    n = named_fields_from_element_type(t)
    if named_fields_and_types and len(named_fields_and_types[-1]) != len(n):
      raise TypeError("element types has different number of fields")
    named_fields_and_types.append(n)

  union_fields_and_types = []
  for field in zip(*named_fields_and_types):
    names, types = zip(*field)
    name_set = set(names)
    if len(name_set) != 1:
      raise TypeError(
          f"Could not determine schema for type hints {element_types!r}: "
          f"Inconsistent names: {name_set}")
    union_fields_and_types.append(
        (next(iter(name_set)), typehints.Union[types]))
  return named_tuple_from_schema(named_fields_to_schema(union_fields_and_types))


class _Ephemeral:
  """Helper class for wrapping unpicklable objects."""
  def __init__(self, obj):

View on GitHub (pinned to 12126d8942)