apache/iceberg · error

Cannot convert Spark filter: $filter to Iceberg expression

Error message

Cannot convert Spark filter: $filter to Iceberg expression

What it means

SparkExpressionConverter.convertFilter translates a Spark Catalyst expression to an Iceberg expression by first lowering it to a DataSource V2 filter via Spark's translateFilterV2, then converting the filter with SparkV2Filters.convert. When the V2 filter exists but the conversion returns null (the filter kind is not representable as an Iceberg expression), this IllegalArgumentException is thrown.

Source

Thrown at spark/v3.5/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala:43

import org.apache.spark.sql.catalyst.expressions.Expression
import org.apache.spark.sql.catalyst.expressions.Literal
import org.apache.spark.sql.catalyst.plans.logical.Filter
import org.apache.spark.sql.catalyst.plans.logical.LeafNode
import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Strategy

object SparkExpressionConverter {

  def convertToIcebergExpression(
      sparkExpression: Expression): org.apache.iceberg.expressions.Expression = {
    // Currently, it is a double conversion as we are converting Spark expression to Spark predicate
    // and then converting Spark predicate to Iceberg expression.
    // But these two conversions already exist and well tested. So, we are going with this approach.
    DataSourceV2Strategy.translateFilterV2(sparkExpression) match {
      case Some(filter) =>
        val converted = SparkV2Filters.convert(filter)
        if (converted == null) {
          throw new IllegalArgumentException(
            s"Cannot convert Spark filter: $filter to Iceberg expression")
        }

        converted
      case _ =>
        throw new IllegalArgumentException(
          s"Cannot translate Spark expression: $sparkExpression to data source filter")
    }
  }

  @throws[AnalysisException]
  def collectResolvedSparkExpression(
      session: SparkSession,
      tableName: String,
      where: String): Expression = {
    val tableAttrs = session.table(tableName).queryExecution.analyzed.output
    val unresolvedExpression = session.sessionState.sqlParser.parseExpression(where)
    val filter = Filter(unresolvedExpression, DummyRelation(tableAttrs))

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Rewrite the filter into predicate forms Iceberg supports (comparisons, IN, IS NULL, AND/OR/NOT)
2. Wrap the predicate so it is evaluated by Spark instead of pushed down (e.g. disable pushdown for that filter)
3. Upgrade Iceberg so SparkV2Filters supports the new V2 filter type introduced by the Spark version

Example fix

// before
val expr = expr("case when id > 5 then true else false end")
SparkExpressionConverter.convertFilter(schema, expr) // throws for unsupported predicate
// after
val expr = expr("id > 5") // supported comparison predicate
SparkExpressionConverter.convertFilter(schema, expr)

val v2Filter = DataSourceV2Strategy.translateFilterV2(sparkExpr)
val convertible = v2Filter.exists(f => SparkV2Filters.convert(f) != null)

def isConvertibleToIceberg(expr: Expression): Boolean =
  DataSourceV2Strategy.translateFilterV2(expr).exists(SparkV2Filters.convert(_) != null)

try {
  val icebergExpr = SparkExpressionConverter.convertFilter(schema, sparkExpr)
  pushdown(icebergExpr)
} catch {
  case e: IllegalArgumentException if e.getMessage.startsWith("Cannot convert Spark filter") =>
    logWarning(s"Filter not pushed down: ${sparkExpr.sql}; evaluating in Spark", e)
}

When it happens

Trigger: Calling SparkExpressionConverter.convertFilter (directly or via Iceberg scans/pushdown paths) with a Spark expression that translates to a V2 filter Iceberg's SparkV2Filters does not handle, e.g. unsupported predicate types.

Common situations: Using exotic SQL predicates (e.g. certain CASE/struct/arrays-containing predicates) in queries that Iceberg attempts to push down; Spark version drift introducing new V2 filter types that Iceberg's converter has not mapped yet.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Cannot translate Spark expression: $sparkExpression to data
• Cannot convert Spark filter: $filter to Iceberg expression
• Cannot convert unknown expression:
• AS OF is not supported for changelogs
• Unknown Spark table type:

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/c4eb7799196d1088. Report an issue: GitHub.