{"record":{"id":"709ea57a41622bc3","repo":"apache/beam","slug":"pivot-of-non-categorical-type-is-not-supported-because-the","errorCode":null,"errorMessage":"pivot() of non-categorical type is not supported because the type of the output column depends on the data. Please use pd.CategoricalDtype with explicit categories.","messagePattern":"pivot\\(\\) of non-categorical type is not supported because the type of the output column depends on the data\\. Please use pd\\.CategoricalDtype with explicit categories\\.","errorType":"exception","errorClass":"WontImplementError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/frames.py","lineNumber":3822,"sourceCode":"  count = _agg_method(pd.DataFrame, 'count')\n  describe = _agg_method(pd.DataFrame, 'describe')\n  max = _agg_method(pd.DataFrame, 'max')\n  min = _agg_method(pd.DataFrame, 'min')\n\n  @frame_base.with_docs_from(pd.DataFrame)\n  @frame_base.args_to_kwargs(pd.DataFrame)\n  @frame_base.populate_defaults(pd.DataFrame)\n  def pivot(self, index=None, columns=None, values=None, **kwargs):\n    \"\"\"Because pivot is a non-deferred method, any columns specified in\n    ``columns`` must be CategoricalDType so we can determine the output column\n    names.\"\"\"\n\n    def verify_all_categorical(all_cols_are_categorical):\n      if not all_cols_are_categorical:\n        message = \"pivot() of non-categorical type is not supported because \" \\\n            \"the type of the output column depends on the data. Please use \" \\\n            \"pd.CategoricalDtype with explicit categories.\"\n        raise frame_base.WontImplementError(\n          message, reason=\"non-deferred-columns\")\n\n    # If values not provided, take all remaining columns of dataframe\n    if not values:\n      tmp = self._expr.proxy()\n      if index:\n        tmp = tmp.drop(index, axis=1)\n      if columns:\n        tmp = tmp.drop(columns, axis=1)\n      values = tmp.columns.values\n\n    # Construct column index\n    if is_list_like(columns) and len(columns) <= 1:\n      columns = columns[0]\n    selected_cols = self._expr.proxy()[columns]\n    if isinstance(selected_cols, pd.Series):\n      all_cols_are_categorical = isinstance(\n        selected_cols.dtype, pd.CategoricalDtype","sourceCodeStart":3804,"sourceCodeEnd":3840,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/frames.py#L3804-L3840","documentation":"Beam's DataFrame.pivot throws WontImplementError when the values (or index) columns are not of CategoricalDtype, because the set of output columns would depend on the data encountered at runtime, violating Beam's requirement of a statically-known schema. Explicit pd.CategoricalDtype categories make the output columns known up front.","triggerScenarios":"Calling df.pivot(...) on a DeferredDataFrame where index/values columns have plain (non-categorical) dtypes.","commonSituations":"Pivoting string-keyed data straight from pandas code; users unaware Beam needs pre-declared pivot categories.","solutions":["Cast the relevant columns to pd.CategoricalDtype with an explicit categories list before pivoting.","Restrict values to columns already typed as categorical.","Use groupby-agg plus an explicit reshape (e.g. unstack on known keys) instead.","Fall back to local pandas if categories are not known in advance."],"exampleFix":"// before\ndf['k'] = df['k'].astype('object')\nresult = df.pivot(index='i', columns='k')\n// after\ndf['k'] = df['k'].astype(pd.CategoricalDtype(categories=['a', 'b', 'c']))\nresult = df.pivot(index='i', columns='k')","handlingStrategy":"validation","validationCode":"def check_pivot_dtypes(df, index, values):\n    cols = list(values or [c for c in df.columns if c not in (index,)])\n    bad = [c for c in cols if not isinstance(df[c].dtype, pd.CategoricalDtype)]\n    if bad:\n        raise ValueError(f'Non-categorical pivot columns: {bad}')","typeGuard":"def is_categorical(col_dtype) -> bool:\n    return isinstance(col_dtype, pd.CategoricalDtype)","tryCatchPattern":"from apache_beam.dataframe import frame_base\ntry:\n    result = df.pivot(index='i', columns='k')\nexcept frame_base.WontImplementError:\n    df = df.assign(k=df['k'].astype(pd.CategoricalDtype(categories=['a', 'b'])))\n    result = df.pivot(index='i', columns='k')","preventionTips":["Declare pivot keys as pd.CategoricalDtype with explicit categories.","Enumerate expected pivot columns at pipeline-construction time.","Avoid pivoting free-form string columns in distributed pipelines."],"tags":["pandas","apache-beam","dataframe","dtype"],"backgroundTag":"unsupported-dtype","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}