apache/beam · error · TypeError

Could not determine schema for type hint

Error message

Could not determine schema for type hint {element_type!r}. Did you mean to create a schema-aware PCollection? See https://s.apache.org/beam-python-schemas

What it means

`schema_from_element_type` derives a Schema proto from a Python type hint. It handles NamedTuples, BeamSchema rows, RowTypeConstraints, and simple primitives; a hint that maps to none of these (e.g. plain dict, arbitrary class, dict[str, Any]) cannot be turned into a schema, so it raises this TypeError pointing at schema-aware PCollection docs.

Solutions

1. Emit NamedTuple instances (or BeamSchema rows) from your transform instead of dicts
2. Apply `Map(lambda d: MyNamedTuple(**d))` to convert dicts to a NamedTuple before the typed transform
3. Annotate your DoFn's process return type with the schema type so the schema can be inferred

Example fix

// before
p | beam.Map(lambda x: {'name': x[0], 'age': x[1]}) | SqlTransform(...)
// after
class Row(typing.NamedTuple):
  name: str
  age: int
p | beam.Map(lambda x: Row(name=x[0], age=x[1])) | SqlTransform(...)

import typing
if not (isinstance(element_type, type) and issubclass(element_type, tuple) and hasattr(element_type, '_fields')):
  raise TypeError('element_type must be a NamedTuple/row type')

def is_schema_type(t) -> bool:
  return isinstance(t, type) and issubclass(t, tuple) and hasattr(t, '_fields')

try:
  schema = schema_from_element_type(hint)
except TypeError:
  raise TypeError('convert your elements to a NamedTuple first') from None

When it happens

Trigger: Calling `schema_from_element_type` with a non-schema type hint — e.g. `dict`, a plain class, `Any`, or a Union — via `from_type_hint`, `expand` on a PTransform expecting typed input, or `named_fields_from_element_type`.

Common situations: Applying a typed transform (like `MapToFields`, `to_row`, or SQL) to a PCollection of dicts instead of NamedTuples/rows; forgetting to annotate DoFn output types.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• A schema is required to write non-schema'd data.
• All dicts in batch must have the same keys. extra keys
• An explicit schema is required to write non-schema'd…
• An Option type-hint only accepts a single type parameter.
• Arrow map key field cannot be nullable

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b10d837a1bdf3475. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/typehints/schemas.py:710

def schema_from_element_type(element_type: type) -> schema_pb2.Schema:
  """Get a schema for the given PCollection element_type.

  Returns schema as a list of (name, python_type) tuples"""
  if isinstance(element_type, row_type.RowTypeConstraint):
    return named_fields_to_schema(element_type._fields)
  elif match_is_named_tuple(element_type) or match_dataclass_for_row(
      element_type):
    # schema id does not inherit from base classes
    if row_type._BEAM_SCHEMA_ID in element_type.__dict__:
      # if the named tuple's schema is in registry, we just use it instead of
      # regenerating one.
      schema_id = getattr(element_type, row_type._BEAM_SCHEMA_ID)
      schema = SCHEMA_REGISTRY.get_schema_by_id(schema_id)
      if schema is not None:
        return schema
    return named_tuple_to_schema(element_type)
  else:
    raise TypeError(
        f"Could not determine schema for type hint {element_type!r}. Did you "
        "mean to create a schema-aware PCollection? See "
        "https://s.apache.org/beam-python-schemas")


def named_fields_from_element_type(
    element_type: type) -> List[Tuple[str, type]]:
  return named_fields_from_schema(schema_from_element_type(element_type))


def union_schema_type(element_types):
  """Returns a schema whose fields are the union of each corresponding field.

  element_types must be a set of schema-aware types whose fields have the
  same naming and ordering.
  """
  named_fields_and_types = []
  for t in element_types:

View on GitHub (pinned to 12126d8942)