{"record":{"id":"89a5c33bb31be91a","repo":"apache/iceberg","slug":"cannot-convert-spark-filter-filter-to-iceberg-ex-89a5c3","errorCode":null,"errorMessage":"Cannot convert Spark filter: $filter to Iceberg expression","messagePattern":"Cannot convert Spark filter: \\$filter to Iceberg expression","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"spark/v4.1/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala","lineNumber":43,"sourceCode":"import org.apache.spark.sql.catalyst.expressions.Literal\nimport org.apache.spark.sql.catalyst.plans.logical.Filter\nimport org.apache.spark.sql.catalyst.plans.logical.LeafNode\nimport org.apache.spark.sql.catalyst.plans.logical.LocalRelation\nimport org.apache.spark.sql.classic.SparkSession\nimport org.apache.spark.sql.execution.datasources.v2.DataSourceV2Strategy\n\nobject SparkExpressionConverter {\n\n  def convertToIcebergExpression(\n      sparkExpression: Expression): org.apache.iceberg.expressions.Expression = {\n    // Currently, it is a double conversion as we are converting Spark expression to Spark predicate\n    // and then converting Spark predicate to Iceberg expression.\n    // But these two conversions already exist and well tested. So, we are going with this approach.\n    DataSourceV2Strategy.translateFilterV2(sparkExpression) match {\n      case Some(filter) =>\n        val converted = SparkV2Filters.convert(filter)\n        if (converted == null) {\n          throw new IllegalArgumentException(\n            s\"Cannot convert Spark filter: $filter to Iceberg expression\")\n        }\n\n        converted\n      case _ =>\n        throw new IllegalArgumentException(\n          s\"Cannot translate Spark expression: $sparkExpression to data source filter\")\n    }\n  }\n\n  @throws[IcebergAnalysisException]\n  def collectResolvedSparkExpression(\n      session: SparkSession,\n      tableName: String,\n      where: String): Expression = {\n    val tableAttrs = session.table(tableName).queryExecution.analyzed.output\n    val unresolvedExpression = session.sessionState.sqlParser.parseExpression(where)\n    val filter = Filter(unresolvedExpression, DummyRelation(tableAttrs))","sourceCodeStart":25,"sourceCodeEnd":61,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala#L25-L61","documentation":"SparkExpressionConverter throws IllegalArgumentException when it cannot translate a Spark DataSourceV2 filter into an Iceberg expression. It first asks Spark's translateFilterV2 to produce a V2 filter, then SparkV2Filters.convert to map it to an Iceberg expression; a null result means the filter kind (or nested structure, e.g. unsupported literals or compound types) has no Iceberg equivalent, so conversion fails loudly instead of producing wrong pushdown results.","triggerScenarios":"Pushing down predicates containing types or functions SparkV2Filters.convert cannot handle — e.g. filters on complex/nested types, unsupported literal values, or a V2 filter that converts to null; typically hit inside Iceberg scans with pushed filters.","commonSituations":"Queries with exotic predicates (arrays/maps/structs comparisons, exotic casts) against Iceberg tables with filter pushdown enabled; Spark version upgrades changing translateFilterV2 output; custom predicates from third-party libraries.","solutions":["Simplify or rewrite the predicate using supported types/operators so it converts cleanly","Cast or restructure the filter (e.g. compare scalar columns rather than nested fields) to a supported form","Check the Spark/Iceberg version pair for known conversion gaps and upgrade the runtime","As a workaround, disable pushdown for the problematic predicate so it is evaluated post-scan (note this affects performance)"],"exampleFix":"// before (filter that fails conversion)\ndf.filter($\"struct_col\" === someStructValue)\n// after\nimport org.apache.spark.sql.functions.col\ndf.filter(col(\"struct_col.field\") === lit(someValue))","handlingStrategy":"validation","validationCode":"import org.apache.spark.sql.connector.expressions.filter.Predicate\n// Ensure predicates only use scalar columns with supported types before pushdown\nval unsupported = df.schema.fields.filter(f => Seq(\"array\", \"map\", \"struct\").contains(f.dataType.typeName))\nif (unsupported.nonEmpty) {\n  log.warn(\"Filters on complex types may fail Iceberg pushdown conversion; evaluate post-scan instead\")\n}","typeGuard":"def isPushdownSafe(dataType: org.apache.spark.sql.types.DataType): Boolean =\n  !dataType.isInstanceOf[org.apache.spark.sql.types.ArrayType] &&\n  !dataType.isInstanceOf[org.apache.spark.sql.types.MapType] &&\n  !dataType.isInstanceOf[org.apache.spark.sql.types.StructType]","tryCatchPattern":"try { icebergScan.filter(expr) } catch { case e: IllegalArgumentException if e.getMessage.startsWith(\"Cannot convert Spark filter\") => log.warn(\"Pushdown unsupported for this predicate; falling back to post-scan filtering\", e); postScanFilter(df, expr) }","preventionTips":["Keep pushed-down predicates to scalar columns and comparison operators supported by Iceberg","Pin compatible Spark/Iceberg version pairs when relying on pushdown","Wrap pushdown-heavy scans in a fallback that re-applies predicates after the scan","Avoid filtering on nested/complex types in pushdown paths"],"tags":["spark","filter-pushdown","expression-conversion","iceberg"],"backgroundTag":"unsupported-operation","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}