{"record":{"id":"cc52f2da12b19fea","repo":"apache/iceberg","slug":"cannot-convert-spark-filter-filter-to-iceberg-ex-cc52f2","errorCode":null,"errorMessage":"Cannot convert Spark filter: $filter to Iceberg expression","messagePattern":"Cannot convert Spark filter: \\$filter to Iceberg expression","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"spark/v4.2/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala","lineNumber":43,"sourceCode":"import org.apache.spark.sql.catalyst.expressions.Literal\nimport org.apache.spark.sql.catalyst.plans.logical.Filter\nimport org.apache.spark.sql.catalyst.plans.logical.LeafNode\nimport org.apache.spark.sql.catalyst.plans.logical.LocalRelation\nimport org.apache.spark.sql.classic.SparkSession\nimport org.apache.spark.sql.execution.datasources.v2.DataSourceV2Strategy\n\nobject SparkExpressionConverter {\n\n  def convertToIcebergExpression(\n      sparkExpression: Expression): org.apache.iceberg.expressions.Expression = {\n    // Currently, it is a double conversion as we are converting Spark expression to Spark predicate\n    // and then converting Spark predicate to Iceberg expression.\n    // But these two conversions already exist and well tested. So, we are going with this approach.\n    DataSourceV2Strategy.translateFilterV2(sparkExpression) match {\n      case Some(filter) =>\n        val converted = SparkV2Filters.convert(filter)\n        if (converted == null) {\n          throw new IllegalArgumentException(\n            s\"Cannot convert Spark filter: $filter to Iceberg expression\")\n        }\n\n        converted\n      case _ =>\n        throw new IllegalArgumentException(\n          s\"Cannot translate Spark expression: $sparkExpression to data source filter\")\n    }\n  }\n\n  @throws[IcebergAnalysisException]\n  def collectResolvedSparkExpression(\n      session: SparkSession,\n      tableName: String,\n      where: String): Expression = {\n    val tableAttrs = session.table(tableName).queryExecution.analyzed.output\n    val unresolvedExpression = session.sessionState.sqlParser.parseExpression(where)\n    val filter = Filter(unresolvedExpression, DummyRelation(tableAttrs))","sourceCodeStart":25,"sourceCodeEnd":61,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala#L25-L61","documentation":"Iceberg converts Spark filter expressions to its own expression tree by first translating the Spark expression into a DataSource V2 filter (`translateFilterV2`) and then converting that filter with `SparkV2Filters.convert`. If Spark produced a filter that Iceberg's converter does not recognize (returns null), the converter throws an IllegalArgumentException because pushdown cannot proceed safely.","triggerScenarios":"`SparkExpressionConverter.convertFilters`/`convertToIcebergExpression` called with a Spark expression that `translateFilterV2` maps to a `Filter` that `SparkV2Filters.convert` cannot map (null result) — e.g. newly added or exotic Spark V2 filters not yet handled.","commonSituations":"Pushdown of unusual predicates (new Spark functions/operators) during batch/micro-batch scans; Spark version upgrades introducing new filter types ahead of Iceberg support; UDF-based predicates.","solutions":["Simplify or rewrite the filter using standard comparison/logical operators","Disable pushdown for the unsupported predicate (e.g. cast the column or wrap in a no-op so it stays post-scan)","Upgrade Iceberg to a version that supports the filter","Check Spark/Iceberg version compatibility"],"exampleFix":"// before\nspark.read...where(\"myUdf(col) > 10\")\n// after\nspark.read...filter(row => myUdf(row.getAs[Int](\"col\")) > 10) // keep UDF post-pushdown","handlingStrategy":"try-catch","validationCode":"// keep pushdown-safe filters only\nval safe = col(\"a\") > 10 && col(\"b\").isin(1, 2, 3) // built-in comparison/logical ops\n","typeGuard":null,"tryCatchPattern":"try { df.filter(expr) } catch { case e: IllegalArgumentException if e.getMessage.startsWith(\"Cannot convert Spark filter\") => df.filter(row => /* evaluate post-scan */) }","preventionTips":["Use built-in comparison and logical operators in pushed filters","Avoid custom/exotic filters that Spark V2 may emit but Iceberg cannot convert","Keep Spark and Iceberg versions aligned","Test pushdown with explain() to confirm which filters are pushed"],"tags":["spark","predicate-pushdown","expression-conversion"],"backgroundTag":"unsupported-operation","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}