{"record":{"id":"c4eb7799196d1088","repo":"apache/iceberg","slug":"cannot-convert-spark-filter-filter-to-iceberg-ex","errorCode":null,"errorMessage":"Cannot convert Spark filter: $filter to Iceberg expression","messagePattern":"Cannot convert Spark filter: \\$filter to Iceberg expression","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"spark/v3.5/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala","lineNumber":43,"sourceCode":"import org.apache.spark.sql.catalyst.expressions.Expression\nimport org.apache.spark.sql.catalyst.expressions.Literal\nimport org.apache.spark.sql.catalyst.plans.logical.Filter\nimport org.apache.spark.sql.catalyst.plans.logical.LeafNode\nimport org.apache.spark.sql.catalyst.plans.logical.LocalRelation\nimport org.apache.spark.sql.execution.datasources.v2.DataSourceV2Strategy\n\nobject SparkExpressionConverter {\n\n  def convertToIcebergExpression(\n      sparkExpression: Expression): org.apache.iceberg.expressions.Expression = {\n    // Currently, it is a double conversion as we are converting Spark expression to Spark predicate\n    // and then converting Spark predicate to Iceberg expression.\n    // But these two conversions already exist and well tested. So, we are going with this approach.\n    DataSourceV2Strategy.translateFilterV2(sparkExpression) match {\n      case Some(filter) =>\n        val converted = SparkV2Filters.convert(filter)\n        if (converted == null) {\n          throw new IllegalArgumentException(\n            s\"Cannot convert Spark filter: $filter to Iceberg expression\")\n        }\n\n        converted\n      case _ =>\n        throw new IllegalArgumentException(\n          s\"Cannot translate Spark expression: $sparkExpression to data source filter\")\n    }\n  }\n\n  @throws[AnalysisException]\n  def collectResolvedSparkExpression(\n      session: SparkSession,\n      tableName: String,\n      where: String): Expression = {\n    val tableAttrs = session.table(tableName).queryExecution.analyzed.output\n    val unresolvedExpression = session.sessionState.sqlParser.parseExpression(where)\n    val filter = Filter(unresolvedExpression, DummyRelation(tableAttrs))","sourceCodeStart":25,"sourceCodeEnd":61,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala#L25-L61","documentation":"SparkExpressionConverter.convertFilter translates a Spark Catalyst expression to an Iceberg expression by first lowering it to a DataSource V2 filter via Spark's translateFilterV2, then converting the filter with SparkV2Filters.convert. When the V2 filter exists but the conversion returns null (the filter kind is not representable as an Iceberg expression), this IllegalArgumentException is thrown.","triggerScenarios":"Calling SparkExpressionConverter.convertFilter (directly or via Iceberg scans/pushdown paths) with a Spark expression that translates to a V2 filter Iceberg's SparkV2Filters does not handle, e.g. unsupported predicate types.","commonSituations":"Using exotic SQL predicates (e.g. certain CASE/struct/arrays-containing predicates) in queries that Iceberg attempts to push down; Spark version drift introducing new V2 filter types that Iceberg's converter has not mapped yet.","solutions":["Rewrite the filter into predicate forms Iceberg supports (comparisons, IN, IS NULL, AND/OR/NOT)","Wrap the predicate so it is evaluated by Spark instead of pushed down (e.g. disable pushdown for that filter)","Upgrade Iceberg so SparkV2Filters supports the new V2 filter type introduced by the Spark version"],"exampleFix":"// before\nval expr = expr(\"case when id > 5 then true else false end\")\nSparkExpressionConverter.convertFilter(schema, expr) // throws for unsupported predicate\n// after\nval expr = expr(\"id > 5\") // supported comparison predicate\nSparkExpressionConverter.convertFilter(schema, expr)","handlingStrategy":"try-catch","validationCode":"val v2Filter = DataSourceV2Strategy.translateFilterV2(sparkExpr)\nval convertible = v2Filter.exists(f => SparkV2Filters.convert(f) != null)","typeGuard":"def isConvertibleToIceberg(expr: Expression): Boolean =\n  DataSourceV2Strategy.translateFilterV2(expr).exists(SparkV2Filters.convert(_) != null)","tryCatchPattern":"try {\n  val icebergExpr = SparkExpressionConverter.convertFilter(schema, sparkExpr)\n  pushdown(icebergExpr)\n} catch {\n  case e: IllegalArgumentException if e.getMessage.startsWith(\"Cannot convert Spark filter\") =>\n    logWarning(s\"Filter not pushed down: ${sparkExpr.sql}; evaluating in Spark\", e)\n}","preventionTips":["Restrict pushed-down predicates to comparisons, IN, IS NULL/NOT NULL, and boolean connectives","Keep Spark and Iceberg versions matched to a tested combination","Test custom predicates against pushdown before shipping queries"],"tags":["spark","iceberg","filter-pushdown","expression-conversion"],"backgroundTag":"unsupported-operation","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}