{"record":{"id":"14077974a1f592a3","repo":"apache/beam","slug":"element-type-must-be-compatible-with-beam-schemas-https-beam","errorCode":null,"errorMessage":"Element type must be compatible with Beam Schemas (https://beam.apache.org/documentation/programming-guide/#schemas) for batch type pd.DataFrame","messagePattern":"Element type must be compatible with Beam Schemas \\(https://beam\\.apache\\.org/documentation/programming-guide/#schemas\\) for batch type pd\\.DataFrame","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/typehints/pandas_type_compatibility.py","lineNumber":168,"sourceCode":"\n\nclass DataFrameBatchConverter(BatchConverter):\n  def __init__(\n      self,\n      element_type: RowTypeConstraint,\n  ):\n    super().__init__(pd.DataFrame, element_type)\n    self._columns = [name for name, _ in element_type._fields]\n\n  @staticmethod\n  def from_typehints(element_type,\n                     batch_type) -> Optional['DataFrameBatchConverter']:\n    assert batch_type == pd.DataFrame\n\n    if not isinstance(element_type, RowTypeConstraint):\n      element_type = RowTypeConstraint.from_user_type(element_type)\n      if element_type is None:\n        raise TypeError(\n            \"Element type must be compatible with Beam Schemas (\"\n            \"https://beam.apache.org/documentation/programming-guide/#schemas) \"\n            \"for batch type pd.DataFrame\")\n\n    index_columns = [\n        field_name\n        for (field_name, field_options) in element_type._field_options.items()\n        if any(key == INDEX_OPTION_NAME for key, value in field_options)\n    ]\n\n    if index_columns:\n      return DataFrameBatchConverterKeepIndex(element_type, index_columns)\n    else:\n      return DataFrameBatchConverterDropIndex(element_type)\n\n  def _get_series(self, batch: pd.DataFrame):\n    raise NotImplementedError\n","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/typehints/pandas_type_compatibility.py#L150-L186","documentation":"DataFrameBatchConverter.from_typehints requires the element type to be a Beam schema-compatible row type because batches are pd.DataFrame, whose columns must correspond to schema fields. If the element_type is not already a RowTypeConstraint and RowTypeConstraint.from_user_type fails to derive one (e.g. the user type is not a NamedTuple or a class annotated with @beam_typehints row schema), this TypeError is raised.","triggerScenarios":"Calling DataFrameBatchConverter.from_typehints (via create_pandas_batch_converter or BatchConverter.from_typehints) with batch_type=pd.DataFrame and an element_type that cannot be converted to a Beam schema: a primitive like int, a plain dict-typed hint without schema, or an unannotated class.","commonSituations":"Batching scalar/primitive elements into DataFrames; forgetting to decorate element classes with typing.NamedTuple or beam schema annotations; using Optional[Any] element hints in pipelines that also use beam.BatchElements.","solutions":["Change the element type to a schema-compatible row type, e.g. a typing.NamedTuple or a class whose fields are Beam schema-compatible.","Alternatively use batch_type=pd.Series, which supports scalar (non-row) element types.","If the user type is a plain class, add type annotations to its fields so RowTypeConstraint.from_user_type can build a schema.","Verify element_type is not None/dynamic before constructing the converter and fail with a clearer message."],"exampleFix":"// before\nclass MyElement:  # no schema\n    pass\nconverter = BatchConverter.from_typehints(element_type=MyElement, batch_type=pd.DataFrame)\n// after\nimport typing\nimport pandas as pd\nclass MyElement(typing.NamedTuple):\n    x: int\n    y: str\nconverter = BatchConverter.from_typehints(element_type=MyElement, batch_type=pd.DataFrame)","handlingStrategy":"type-guard","validationCode":"from apache_beam.typehints.row_type import RowTypeConstraint\ndef is_df_batch_compatible(element_type, batch_type) -> bool:\n    import pandas as pd\n    if batch_type == pd.Series:\n        return True\n    return isinstance(element_type, RowTypeConstraint) or RowTypeConstraint.from_user_type(element_type) is not None\n","typeGuard":"def is_row_schema_type(element_type) -> bool:\n    from apache_beam.typehints.row_type import RowTypeConstraint\n    return isinstance(element_type, RowTypeConstraint) or RowTypeConstraint.from_user_type(element_type) is not None\n","tryCatchPattern":"if not is_row_schema_type(element_type):\n    # switch to Series batching or fix the element type before constructing\n    batch_type = pd.Series\nconverter = create_pandas_batch_converter(element_type=element_type, batch_type=batch_type)\n","preventionTips":["Define batching element types as typing.NamedTuple or Beam-schema-annotated classes.","Use pd.Series batching for scalar elements, pd.DataFrame only for rows.","Run RowTypeConstraint.from_user_type early to validate element classes.","Keep element field annotations Beam-schema-compatible (primitives, lists, nested rows)."],"tags":["python","apache-beam","pandas","schemas","batching"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}