apache/iceberg · error · IllegalArgumentException

Cannot convert Spark filter: $filter to Iceberg expression

Error message

Cannot convert Spark filter: $filter to Iceberg expression

What it means

Iceberg converts Spark filter expressions to its own expression tree by first translating the Spark expression into a DataSource V2 filter (`translateFilterV2`) and then converting that filter with `SparkV2Filters.convert`. If Spark produced a filter that Iceberg's converter does not recognize (returns null), the converter throws an IllegalArgumentException because pushdown cannot proceed safely.

Solutions

1. Simplify or rewrite the filter using standard comparison/logical operators
2. Disable pushdown for the unsupported predicate (e.g. cast the column or wrap in a no-op so it stays post-scan)
3. Upgrade Iceberg to a version that supports the filter
4. Check Spark/Iceberg version compatibility

Example fix

// before
spark.read...where("myUdf(col) > 10")
// after
spark.read...filter(row => myUdf(row.getAs[Int]("col")) > 10) // keep UDF post-pushdown

// keep pushdown-safe filters only
val safe = col("a") > 10 && col("b").isin(1, 2, 3) // built-in comparison/logical ops

try { df.filter(expr) } catch { case e: IllegalArgumentException if e.getMessage.startsWith("Cannot convert Spark filter") => df.filter(row => /* evaluate post-scan */) }

When it happens

Trigger: `SparkExpressionConverter.convertFilters`/`convertToIcebergExpression` called with a Spark expression that `translateFilterV2` maps to a `Filter` that `SparkV2Filters.convert` cannot map (null result) — e.g. newly added or exotic Spark V2 filters not yet handled.

Common situations: Pushdown of unusual predicates (new Spark functions/operators) during batch/micro-batch scans; Spark version upgrades introducing new filter types ahead of Iceberg support; UDF-based predicates.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Cannot translate Spark expression: $sparkExpression to data…
• Cannot convert Spark filter: $filter to Iceberg expression
• Cannot convert Spark filter: $filter to Iceberg expression
• Cannot convert unknown expression:
• Cannot translate Spark expression: $sparkExpression to data…

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/cc52f2da12b19fea. Report an issue: GitHub.

Appendix: source

Thrown at spark/v4.2/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala:43

import org.apache.spark.sql.catalyst.expressions.Literal
import org.apache.spark.sql.catalyst.plans.logical.Filter
import org.apache.spark.sql.catalyst.plans.logical.LeafNode
import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
import org.apache.spark.sql.classic.SparkSession
import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Strategy

object SparkExpressionConverter {

  def convertToIcebergExpression(
      sparkExpression: Expression): org.apache.iceberg.expressions.Expression = {
    // Currently, it is a double conversion as we are converting Spark expression to Spark predicate
    // and then converting Spark predicate to Iceberg expression.
    // But these two conversions already exist and well tested. So, we are going with this approach.
    DataSourceV2Strategy.translateFilterV2(sparkExpression) match {
      case Some(filter) =>
        val converted = SparkV2Filters.convert(filter)
        if (converted == null) {
          throw new IllegalArgumentException(
            s"Cannot convert Spark filter: $filter to Iceberg expression")
        }

        converted
      case _ =>
        throw new IllegalArgumentException(
          s"Cannot translate Spark expression: $sparkExpression to data source filter")
    }
  }

  @throws[IcebergAnalysisException]
  def collectResolvedSparkExpression(
      session: SparkSession,
      tableName: String,
      where: String): Expression = {
    val tableAttrs = session.table(tableName).queryExecution.analyzed.output
    val unresolvedExpression = session.sessionState.sqlParser.parseExpression(where)
    val filter = Filter(unresolvedExpression, DummyRelation(tableAttrs))

View on GitHub (pinned to 86d9c8fc54)