{"record":{"id":"3fe149930b85bf20","repo":"apache/beam","slug":"get-dummies-of-non-categorical-type-is-not-supported-because","errorCode":null,"errorMessage":"get_dummies() of non-categorical type is not supported because the type of the output column depends on the data. Please use pd.CategoricalDtype with explicit categories.","messagePattern":"get_dummies\\(\\) of non-categorical type is not supported because the type of the output column depends on the data\\. Please use pd\\.CategoricalDtype with explicit categories\\.","errorType":"exception","errorClass":"WontImplementError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/frames.py","lineNumber":5101,"sourceCode":"    elif isinstance(repeats, list):\n      raise frame_base.WontImplementError(\n          \"str.repeat(repeats=) repeats must be an int or a DeferredSeries. \"\n          \"Lists are not supported because they make this operation sensitive \"\n          \"to the order of the data.\", reason=\"order-sensitive\")\n    else:\n      raise TypeError(\"str.repeat(repeats=) value must be an int or a \"\n                      f\"DeferredSeries (encountered {type(repeats)}).\")\n\n  @frame_base.with_docs_from(pd.Series.str)\n  @frame_base.args_to_kwargs(pd.Series.str)\n  def get_dummies(self, **kwargs):\n    \"\"\"\n    Series must be categorical dtype. Please cast to ``CategoricalDtype``\n    to ensure correct categories.\n    \"\"\"\n    dtype = self._expr.proxy().dtype\n    if not isinstance(dtype, pd.CategoricalDtype):\n      raise frame_base.WontImplementError(\n          \"get_dummies() of non-categorical type is not supported because \"\n          \"the type of the output column depends on the data. Please use \"\n          \"pd.CategoricalDtype with explicit categories.\",\n          reason=\"non-deferred-columns\")\n\n    split_cats = [\n      cat.split(sep=kwargs.get('sep', '|')) for cat in dtype.categories\n    ]\n\n    # Adding the nan category because there could be the case that\n    # the data includes NaNs, which is not valid to be casted as a Category,\n    # but nevertheless would be broadcasted as a column in get_dummies()\n    columns = sorted(set().union(*split_cats))\n    if _DUMMY_NAN_COLUMN not in columns:\n      columns = columns + [_DUMMY_NAN_COLUMN]\n\n    proxy = pd.DataFrame(columns=columns).astype(int)\n","sourceCodeStart":5083,"sourceCodeEnd":5119,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/frames.py#L5083-L5119","documentation":"str.get_dummies on a deferred Beam Series raises WontImplementError when the series dtype is not pandas CategoricalDtype. With a non-categorical dtype, the set of dummy columns would be discovered from the data at runtime, making the output columns data-dependent, which Beam's deferred model cannot support.","triggerScenarios":"Calling s.str.get_dummies(sep=...) where s.dtype is object/string rather than pd.CategoricalDtype on a deferred Beam Series.","commonSituations":"One-hot encoding free-text tag columns split by a delimiter; porting pandas get_dummies feature engineering to Beam; forgetting to declare the category set up front.","solutions":["Cast the series to an explicit categorical dtype first: s.astype(pd.CategoricalDtype(categories=[...])).","List all possible categories explicitly so the output columns are known statically.","Build dummy columns manually with boolean expressions per known category (s.str.contains('cat')).","Perform the get_dummies step outside Beam in plain pandas."],"exampleFix":"// before\ndummies = s.str.get_dummies(sep=',')\n\n// after\ns = s.astype(pd.CategoricalDtype(categories=['a', 'b', 'c']))\ndummies = s.str.get_dummies(sep=',')","handlingStrategy":"validation","validationCode":"if not isinstance(s._expr.proxy().dtype, pd.CategoricalDtype):\n    s = s.astype(pd.CategoricalDtype(categories=known_categories))","typeGuard":"def is_categorical(s):\n    return isinstance(s._expr.proxy().dtype, pd.CategoricalDtype)","tryCatchPattern":"try:\n    dummies = s.str.get_dummies(sep=',')\nexcept apachebeam.WontImplementError:\n    dummies = s.astype(pd.CategoricalDtype(categories=known_categories)).str.get_dummies(sep=',')","preventionTips":["Cast to CategoricalDtype with an explicit category list before any column-creating string op.","Keep a canonical list of allowed category values in config/code.","Audit string feature-engineering steps for data-dependent output columns."],"tags":["apache-beam","dataframe","deferred-columns","categorical","pandas"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}