{"record":{"id":"25a2c0d60d52a0f8","repo":"apache/iceberg","slug":"cannot-translate-spark-expression-sparkexpressio","errorCode":null,"errorMessage":"Cannot translate Spark expression: $sparkExpression to data source filter","messagePattern":"Cannot translate Spark expression: \\$sparkExpression to data source filter","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"spark/v3.5/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala","lineNumber":49,"sourceCode":"\nobject SparkExpressionConverter {\n\n  def convertToIcebergExpression(\n      sparkExpression: Expression): org.apache.iceberg.expressions.Expression = {\n    // Currently, it is a double conversion as we are converting Spark expression to Spark predicate\n    // and then converting Spark predicate to Iceberg expression.\n    // But these two conversions already exist and well tested. So, we are going with this approach.\n    DataSourceV2Strategy.translateFilterV2(sparkExpression) match {\n      case Some(filter) =>\n        val converted = SparkV2Filters.convert(filter)\n        if (converted == null) {\n          throw new IllegalArgumentException(\n            s\"Cannot convert Spark filter: $filter to Iceberg expression\")\n        }\n\n        converted\n      case _ =>\n        throw new IllegalArgumentException(\n          s\"Cannot translate Spark expression: $sparkExpression to data source filter\")\n    }\n  }\n\n  @throws[AnalysisException]\n  def collectResolvedSparkExpression(\n      session: SparkSession,\n      tableName: String,\n      where: String): Expression = {\n    val tableAttrs = session.table(tableName).queryExecution.analyzed.output\n    val unresolvedExpression = session.sessionState.sqlParser.parseExpression(where)\n    val filter = Filter(unresolvedExpression, DummyRelation(tableAttrs))\n    val optimizedLogicalPlan = session.sessionState.executePlan(filter).optimizedPlan\n    optimizedLogicalPlan\n      .collectFirst {\n        case filter: Filter => filter.condition\n        case _: DummyRelation => Literal.TrueLiteral\n        case _: LocalRelation => Literal.FalseLiteral","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala#L31-L67","documentation":"convertFilter first asks Spark's DataSourceV2Strategy.translateFilterV2 to lower the Spark expression into a DataSource V2 filter. If Spark cannot produce any V2 filter for the expression, this IllegalArgumentException is thrown, since without a V2 filter there is nothing to convert to an Iceberg expression.","triggerScenarios":"Calling convertFilter with a Spark expression that Spark's translateFilterV2 returns None for — expressions Spark itself cannot lower to a data source filter (complex expressions, non-deterministic functions, unsupported operators).","commonSituations":"Pushing down user-defined functions or non-deterministic predicates; passing unresolved or non-boolean expressions; Spark/Iceberg version mismatches where an expression is not yet covered by translateFilterV2.","solutions":["Simplify or rewrite the predicate to forms Spark can lower to V2 filters (=, <, >, IN, IS NULL, AND/OR/NOT)","Avoid non-deterministic or UDF-based predicates in pushdown contexts","Check Spark and Iceberg versions are a matched pair"],"exampleFix":"// before\nSparkExpressionConverter.convertFilter(schema, expr(\"rand() > 0.5\")) // not translatable\n// after\nSparkExpressionConverter.convertFilter(schema, expr(\"id > 5\"))","handlingStrategy":"try-catch","validationCode":"val lowered = DataSourceV2Strategy.translateFilterV2(sparkExpr)\nif (lowered.isEmpty) logWarning(s\"Expression ${sparkExpr.sql} cannot be lowered to a V2 filter\")","typeGuard":null,"tryCatchPattern":"try {\n  val icebergExpr = SparkExpressionConverter.convertFilter(schema, sparkExpr)\n} catch {\n  case e: IllegalArgumentException if e.getMessage.startsWith(\"Cannot translate Spark expression\") =>\n    // fall back to in-Spark evaluation of the predicate\n}","preventionTips":["Avoid non-deterministic functions and UDFs in predicates intended for pushdown","Use standard SQL predicate operators only","Verify translateFilterV2 lowering in unit tests for custom predicates"],"tags":["spark","iceberg","filter-pushdown","expression-conversion"],"backgroundTag":"unsupported-operation","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}