{"record":{"id":"45b6f39e39a0921a","repo":"dbt-labs/dbt-core","slug":"type-df-is-not-a-supported-type-for-dbt-python-45b6f3","errorCode":null,"errorMessage":"{type(df)} is not a supported type for dbt Python materialization","messagePattern":"(.+?) is not a supported type for dbt Python materialization","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"crates/dbt-loader/src/dbt_macro_assets/dbt-databricks/macros/adapters/python.sql","lineNumber":53,"sourceCode":"            else:\n                raise e\n    elif koalas_available:\n        df = databricks.koalas.frame.DataFrame(df)\n\n# convert to pyspark.sql.dataframe.DataFrame\nif isinstance(df, pyspark.sql.dataframe.DataFrame):\n    pass  # since it is already a Spark DataFrame\nelif newer_pyspark_available and isinstance(df, pyspark.sql.connect.dataframe.DataFrame):\n    pass  # since it is already a Spark DataFrame\nelif pyspark_pandas_api_available and isinstance(df, pyspark.pandas.frame.DataFrame):\n    df = df.to_spark()\nelif koalas_available and isinstance(df, databricks.koalas.frame.DataFrame):\n    df = df.to_spark()\nelif pandas_available and isinstance(df, pandas.core.frame.DataFrame):\n    df = spark.createDataFrame(df)\nelse:\n    msg = f\"{type(df)} is not a supported type for dbt Python materialization\"\n    raise Exception(msg)\n\nwriter = (\n    df.write\n        .mode(\"overwrite\")\n        .option(\"overwriteSchema\", \"true\")\n{{ py_get_writer_options()|indent(8, True) }}\n)\n\nwriter.saveAsTable(\"{{ target_relation }}\")\n{% endmacro %}\n\n# Note: this is not the code used for performing incremental merges.\n# The current process uses this code to create a staging table that is\n# merged in using a SQL statement.  To see your incremental config in action,\n# look in the dbt.log\n\n{%- macro py_get_writer_options() -%}\n{%- set location_root = config.get('location_root', validator=validation.any[basestring]) -%}","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/dbt-labs/dbt-core/blob/0267ce9170576975b76b64ce856b2e5848e96617/crates/dbt-loader/src/dbt_macro_assets/dbt-databricks/macros/adapters/python.sql#L35-L71","documentation":"Raised by the dbt-databricks Python model adapter macro when the DataFrame returned by a Python model is not a Spark DataFrame, a Koalas DataFrame, or a pandas DataFrame convertible via spark.createDataFrame. Only these three types can be written back to the table with the subsequent df.write call.","triggerScenarios":"A Databricks Python model returns a type other than pyspark.sql.DataFrame, databricks.koalas DataFrame, or pandas.DataFrame — e.g. a polars DataFrame, a list, a NumPy array, or a pyspark.pandas DataFrame (which this macro does not check). Also occurs when the pandas/koalas availability checks are False in the cluster environment despite the model returning pandas.","commonSituations":"Returning polars or other modern DataFrame libraries from a Python model; returning a plain Python collection instead of a DataFrame; missing pandas in the cluster's dbt environment so pandas_available is False; copy-pasting model code that returns Koalas when Koalas is unavailable on the runtime.","solutions":["Return a pyspark.sql.DataFrame from the model (e.g. df = spark.createDataFrame(...)) so no conversion is needed.","Convert other DataFrame types to pandas before returning (polars: df.to_pandas()), letting spark.createDataFrame handle it.","Ensure pandas is available on the Databricks cluster runtime used for Python models so the pandas branch works.","Verify the model function returns a DataFrame and not a list/dict/NumPy array.","On newer runtimes, return a pyspark DataFrame rather than relying on the deprecated databricks.koalas branch."],"exampleFix":"# before\ndef model(dbt, session):\n    return pl.from_pandas(pandas_df)  # polars -> exception\n# after\ndef model(dbt, session):\n    return pandas_df  # pandas is converted via spark.createDataFrame","handlingStrategy":"type-guard","validationCode":"def ensure_supported_df(df):\n    from pyspark.sql import DataFrame as SparkDF\n    if isinstance(df, SparkDF):\n        return df\n    import pandas as pd\n    if isinstance(df, pd.DataFrame):\n        return df  # converted by macro via spark.createDataFrame\n    if hasattr(df, 'to_pandas'):\n        return df.to_pandas()\n    raise TypeError(f'{type(df)} unsupported; return Spark/Koalas/pandas DataFrame')","typeGuard":"def is_supported_df(df) -> bool:\n    try:\n        from pyspark.sql import DataFrame as SparkDF\n        if isinstance(df, SparkDF):\n            return True\n    except ImportError:\n        pass\n    try:\n        import pandas as pd\n        if isinstance(df, pd.DataFrame):\n            return True\n    except ImportError:\n        pass\n    try:\n        import databricks.koalas\n        if isinstance(df, databricks.koalas.frame.DataFrame):\n            return True\n    except ImportError:\n        pass\n    return False","tryCatchPattern":"try:\n    result = runner.invoke(['run', '--select', 'my_python_model'])\nexcept Exception as e:\n    if 'is not a supported type for dbt Python materialization' in str(e):\n        fix_model_return_type('my_python_model')\n    else:\n        raise","preventionTips":["Prefer returning a pyspark.sql.DataFrame directly from Databricks Python models.","Convert polars/other DataFrames with .to_pandas() before returning.","Verify pandas is present on the cluster runtime used for Python models.","Add a unit test asserting the model returns a supported DataFrame type.","Avoid relying on databricks.koalas on newer runtimes where it is deprecated."],"tags":["dbt","databricks","python-model","dataframe","spark"],"backgroundTag":"type-mismatch","analyzedSha":"0267ce9170576975b76b64ce856b2e5848e96617","analyzedAt":"2026-09-07T21:53:39.732Z","contentChangedAt":"2026-09-07T21:53:39.732Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}