{"record":{"id":"add32e029b52ec5b","repo":"apache/beam","slug":"unable-to-identify-type-typ-specified-on-column-col-name","errorCode":null,"errorMessage":"Unable to identify type: {typ} specified on column: {col_name}. Please provide a valid type from the following: {_default_type_to_tensor_type_map.keys()}","messagePattern":"Unable to identify type: (.+?) specified on column: (.+?)\\. Please provide a valid type from the following: (.+?)","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/ml/transforms/handlers.py","lineNumber":271,"sourceCode":"    primitive_containers_type = (\n        list,\n        collections.abc.Sequence,\n    )\n    is_primitive_container = (\n        typing.get_origin(typ) in primitive_containers_type)\n\n    if is_primitive_container:\n      dtype = typing.get_args(typ)[0]\n      if len(typing.get_args(typ)) > 1 or typing.get_origin(dtype) == Union:\n        raise RuntimeError(\n            f\"Union type is not supported for column: {col_name}. \"\n            f\"Please pass a PCollection with valid schema for column \"\n            f\"{col_name} by passing a single type \"\n            \"in container. For example, list[int].\")\n    elif issubclass(typ, np.generic) or typ in _default_type_to_tensor_type_map:\n      dtype = typ\n    else:\n      raise TypeError(\n          f\"Unable to identify type: {typ} specified on column: {col_name}. \"\n          f\"Please provide a valid type from the following: \"\n          f\"{_default_type_to_tensor_type_map.keys()}\")\n    return tf.io.VarLenFeature(_default_type_to_tensor_type_map[dtype])\n\n  def get_raw_data_metadata(\n      self, input_types: dict[str, type]) -> dataset_metadata.DatasetMetadata:\n    raw_data_feature_spec = self.get_raw_data_feature_spec(input_types)\n    raw_data_feature_spec[_TEMP_KEY] = tf.io.VarLenFeature(dtype=tf.string)\n    return self.convert_raw_data_feature_spec_to_dataset_metadata(\n        raw_data_feature_spec)\n\n  def write_transform_artifacts(self, transform_fn, location):\n    \"\"\"\n    Write transform artifacts to the given location.\n    Args:\n      transform_fn: A transform_fn object.\n      location: A location to write the artifacts.","sourceCodeStart":253,"sourceCodeEnd":289,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/ml/transforms/handlers.py#L253-L289","documentation":"TFTProcessHandler._get_raw_data_feature_spec_per_column builds a tf.io feature spec for each column and only recognizes a fixed set of types (numpy scalar subtypes and keys of _default_type_to_tensor_type_map). When a column's declared type is outside that set, it cannot be mapped to a TensorInfo and a TypeError is raised. This often happens when a type is wrapped in a container like list[int] without unpacking, which the code explicitly rejects earlier.","triggerScenarios":"Passing an MLTransform (TFT-based) a PCollection schema column whose type annotation is not a np.generic subclass nor a key of _default_type_to_tensor_type_map — e.g. a custom Python class, list[int] passed as the whole annotation where a bare type is expected, or str vs np.bytes_ mismatches.","commonSituations":"Declaring beam.Row schemas with Python typing wrappers (Optional[int], list[int]), using custom types in typed DoBags, or typos in type names when constructing TransformConfigs.","solutions":["Change the column's type annotation to a supported type: int, float, bool, bytes, str, or the corresponding np.generic types.","Unwrap container annotations: use the inner scalar type (int instead of list[int]) or pass the list types explicitly as supported.","Check _default_type_to_tensor_type_map in handlers.py for the exact supported set before annotating the schema.","If a custom type is required, pre-convert the column to a supported dtype in a prior beam.Map before MLTransform."],"exampleFix":"// before\nbeam.Row(x=[1, 2, 3]).with_output_types(...)  # column typed as list[int] container\n\n// after\nbeam.Row(x=1).with_output_types(...)  # bare supported scalar type, e.g. int / np.int64","handlingStrategy":"type-guard","validationCode":"import numpy as np\nfrom apache_beam.ml.transforms.handlers import _default_type_to_tensor_type_map\n\ndef is_supported_column_type(typ) -> bool:\n    return (isinstance(typ, type) and issubclass(typ, np.generic)) or typ in _default_type_to_tensor_type_map\n\nassert all(is_supported_column_type(t) for t in my_column_types)","typeGuard":"def is_supported_column_type(typ) -> bool:\n    import numpy as np\n    from apache_beam.ml.transforms.handlers import _default_type_to_tensor_type_map\n    return (isinstance(typ, type) and issubclass(typ, np.generic)) or typ in _default_type_to_tensor_type_map","tryCatchPattern":"try:\n    result = pcoll | MLTransform(...).with_write_artifact_location(loc)\nexcept TypeError as e:\n    if 'Unable to identify type' in str(e):\n        # fix schema annotation or convert column to a supported dtype\n        raise ValueError(f'Unsupported column type in schema: {e}') from e\n    raise","preventionTips":["Annotate PCollection schemas only with types verified against _default_type_to_tensor_type_map.","Avoid typing wrappers (Optional[T], list[T]) in MLTransform-bound schemas.","Pre-convert custom columns to numpy scalars before MLTransform.","Write a unit test asserting every schema column type is supported."],"tags":["python","apache-beam","tft","type-mapping"],"backgroundTag":"unsupported-dtype","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}