{"record":{"id":"c6409901eeba0cce","repo":"dbt-labs/dbt-core","slug":"type-df-is-not-a-supported-type-for-dbt-python-c64099","errorCode":null,"errorMessage":"{type(df)} is not a supported type for dbt Python materialization","messagePattern":"(.+?) is not a supported type for dbt Python materialization","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"crates/dbt-loader/src/dbt_macro_assets/dbt-spark/macros/materializations/table.sql","lineNumber":99,"sourceCode":"# since they know how to convert pandas DataFrames better than `spark.createDataFrame(df)`\n# and converting from pandas-on-Spark to Spark DataFrame has no overhead\nif pyspark_pandas_api_available and pandas_available and isinstance(df, pandas.core.frame.DataFrame):\n  df = pyspark.pandas.frame.DataFrame(df)\nelif koalas_available and pandas_available and isinstance(df, pandas.core.frame.DataFrame):\n  df = databricks.koalas.frame.DataFrame(df)\n\n# convert to pyspark.sql.dataframe.DataFrame\nif isinstance(df, pyspark.sql.dataframe.DataFrame):\n  pass  # since it is already a Spark DataFrame\nelif pyspark_pandas_api_available and isinstance(df, pyspark.pandas.frame.DataFrame):\n  df = df.to_spark()\nelif koalas_available and isinstance(df, databricks.koalas.frame.DataFrame):\n  df = df.to_spark()\nelif pandas_available and isinstance(df, pandas.core.frame.DataFrame):\n  df = spark.createDataFrame(df)\nelse:\n  msg = f\"{type(df)} is not a supported type for dbt Python materialization\"\n  raise Exception(msg)\n\ndf.write.mode(\"overwrite\").format(\"{{ config.get('file_format', 'delta') }}\").option(\"overwriteSchema\", \"true\").saveAsTable(\"{{ target_relation }}\")\n{%- endmacro -%}\n\n{%macro py_script_comment()%}\n# how to execute python model in notebook\n# dbt = dbtObj(spark.table)\n# df = model(dbt, spark)\n{%endmacro%}\n","sourceCodeStart":81,"sourceCodeEnd":109,"githubUrl":"https://github.com/dbt-labs/dbt-core/blob/0267ce9170576975b76b64ce856b2e5848e96617/crates/dbt-loader/src/dbt_macro_assets/dbt-spark/macros/materializations/table.sql#L81-L109","documentation":"The Apache Spark (dbt-spark) dbt Python table materialization macro checks the `df` object against known DataFrame types — Spark, Databricks Koalas (via to_spark), and pandas (via spark.createDataFrame) — before writing with df.write.saveAsTable. Any other object type raises this Exception because the macro cannot persist it to the target relation.","triggerScenarios":"A Python model on Spark whose function returns None (no return statement), or returns a type not covered by the isinstance chain: a list/tuple from collect(), a numpy array, a polars DataFrame, a pyspark.pandas DataFrame when neither koalas nor the pandas check matches, or an RDD.","commonSituations":"Returning df.collect() output instead of the DataFrame; forgetting `return df`; using pyspark.pandas on a runtime where the availability flags don't detect it; accidentally shadowing `df` with a converted value (e.g. df = df.toPandas().values).","solutions":["Ensure the model function returns a pyspark.sql.DataFrame, pandas DataFrame, or databricks.koalas DataFrame.","Explicitly convert unsupported types before returning (e.g. `spark.createDataFrame(pandas_df)` or `pdf.to_spark()`).","Remove any code that reassigns df to a non-DataFrame (collect()/take()/head() results).","Install pyspark/pandas (or databricks.koalas for older runtimes) so the type detection succeeds."],"exampleFix":"# before\ndef model(dbt, session):\n    df = dbt.ref('upstream')\n    df = df.collect()  # list\n    return df\n\n# after\ndef model(dbt, session):\n    df = dbt.ref('upstream')\n    return df  # keep df as a DataFrame","handlingStrategy":"type-guard","validationCode":"supported = False\ntry:\n    import pyspark\n    supported = supported or isinstance(df, pyspark.sql.DataFrame)\nexcept ImportError:\n    pass\ntry:\n    import pandas\n    supported = supported or isinstance(df, pandas.core.frame.DataFrame)\nexcept ImportError:\n    pass\nassert supported, f'dbt-spark Python model returned unsupported type {type(df)}'","typeGuard":"def is_spark_supported_df(df) -> bool:\n    checks = []\n    try:\n        import pyspark\n        checks.append(isinstance(df, pyspark.sql.DataFrame))\n    except ImportError:\n        pass\n    try:\n        import pandas\n        checks.append(isinstance(df, pandas.core.frame.DataFrame))\n    except ImportError:\n        pass\n    try:\n        import databricks.koalas\n        checks.append(isinstance(df, databricks.koalas.frame.DataFrame))\n    except ImportError:\n        pass\n    return any(checks)","tryCatchPattern":"try:\n    dbt_runner.run(model_sql)\nexcept Exception as e:\n    if 'is not a supported type for dbt Python materialization' in str(e):\n        logger.error('Return a Spark, pandas, or koalas DataFrame from the Python model')\n        raise RuntimeError('Unsupported return type in dbt-spark Python model') from e","preventionTips":["Keep df a DataFrame through the whole model body; never overwrite it with collect()/head() results.","Convert pyspark.pandas frames to spark via to_spark() if koalas detection is unavailable.","Ensure pyspark and pandas are installed in the cluster environment.","Add a smoke test that runs the model and checks the returned type before full dbt runs."],"tags":["python-model","dataframe","spark","materialization"],"backgroundTag":"type-mismatch","analyzedSha":"0267ce9170576975b76b64ce856b2e5848e96617","analyzedAt":"2026-09-07T21:53:39.732Z","contentChangedAt":"2026-09-07T21:53:39.732Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}