apache/beam · error · RuntimeError

Union type is not supported for column

Error message

Union type is not supported for column: {col_name}. Please pass a PCollection with valid schema for column {col_name} by passing a single type in container. For example, list[int].

What it means

When building the feature spec for each column, MLTransform inspects the PCollection row type. If a column's type is a container whose element type is a Union (e.g. list[Union[int, str]] or list[int|str]), or has multiple type args, it cannot map to a single tensor dtype, so RuntimeError is raised.

Solutions

1. Constrain the column to a single concrete type, e.g. list[int] instead of list[Union[int, str]].
2. Split mixed-type data into separate columns, one per type.
3. Coerce/normalize values before the transform so the annotation is a single type.

Example fix

// before
class Row(TypedDict):
    values: List[Union[int, str]]
// after
class Row(TypedDict):
    values: List[int]

import typing
def col_type_is_simple(typ) -> bool:
    origin = typing.get_origin(typ)
    if origin in (list, tuple, set):
        args = typing.get_args(typ)
        return len(args) == 1 and typing.get_origin(args[0]) != Union
    return True

def has_union_element(typ) -> bool:
    if typing.get_origin(typ) in (list, tuple, set):
        args = typing.get_args(typ)
        return len(args) > 1 or typing.get_origin(args[0]) == Union
    return False

try:
    spec = mltransform.get_raw_data_feature_spec(schema)
except RuntimeError as e:
    if 'Union type is not supported' in str(e):
        raise SchemaError('normalize column types before MLTransform') from e

When it happens

Trigger: Declaring a PCollection schema column typed as Optional/list with union element types, then running MLTransform.get_raw_data_feature_spec over it.

Common situations: Using TypedDict/NamedTuple rows where a field is Optional[str] inside a list; inference-produced types like list[int | None]; loose annotations like list[Any] resolving to Union.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• A schema is required to write non-schema'd data.
• All dicts in batch must have the same keys. extra keys
• An explicit schema is required to write non-schema'd…
• Arrow map key field cannot be nullable
• artifact_mode must be either `produce` or `consume`.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8a5c984e8e52b75d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/transforms/handlers.py:263

    Args:
      typ: A type of the column.
      col_name: A name of the column.
    Returns:
      A FeatureSpec object.
    """
    # lets conver the builtin types to typing types for consistency.
    typ = native_type_compatibility.convert_builtin_to_typing(typ)
    primitive_containers_type = (
        list,
        collections.abc.Sequence,
    )
    is_primitive_container = (
        typing.get_origin(typ) in primitive_containers_type)

    if is_primitive_container:
      dtype = typing.get_args(typ)[0]
      if len(typing.get_args(typ)) > 1 or typing.get_origin(dtype) == Union:
        raise RuntimeError(
            f"Union type is not supported for column: {col_name}. "
            f"Please pass a PCollection with valid schema for column "
            f"{col_name} by passing a single type "
            "in container. For example, list[int].")
    elif issubclass(typ, np.generic) or typ in _default_type_to_tensor_type_map:
      dtype = typ
    else:
      raise TypeError(
          f"Unable to identify type: {typ} specified on column: {col_name}. "
          f"Please provide a valid type from the following: "
          f"{_default_type_to_tensor_type_map.keys()}")
    return tf.io.VarLenFeature(_default_type_to_tensor_type_map[dtype])

  def get_raw_data_metadata(
      self, input_types: dict[str, type]) -> dataset_metadata.DatasetMetadata:
    raw_data_feature_spec = self.get_raw_data_feature_spec(input_types)
    raw_data_feature_spec[_TEMP_KEY] = tf.io.VarLenFeature(dtype=tf.string)
    return self.convert_raw_data_feature_spec_to_dataset_metadata(

View on GitHub (pinned to 12126d8942)