apache/iceberg · error · IllegalArgumentException

Cannot convert Spark filter: $filter to Iceberg expression

Error message

Cannot convert Spark filter: $filter to Iceberg expression

What it means

SparkExpressionConverter throws IllegalArgumentException when it cannot translate a Spark DataSourceV2 filter into an Iceberg expression. It first asks Spark's translateFilterV2 to produce a V2 filter, then SparkV2Filters.convert to map it to an Iceberg expression; a null result means the filter kind (or nested structure, e.g. unsupported literals or compound types) has no Iceberg equivalent, so conversion fails loudly instead of producing wrong pushdown results.

Source

Thrown at spark/v4.1/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala:43

import org.apache.spark.sql.catalyst.expressions.Literal
import org.apache.spark.sql.catalyst.plans.logical.Filter
import org.apache.spark.sql.catalyst.plans.logical.LeafNode
import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
import org.apache.spark.sql.classic.SparkSession
import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Strategy

object SparkExpressionConverter {

  def convertToIcebergExpression(
      sparkExpression: Expression): org.apache.iceberg.expressions.Expression = {
    // Currently, it is a double conversion as we are converting Spark expression to Spark predicate
    // and then converting Spark predicate to Iceberg expression.
    // But these two conversions already exist and well tested. So, we are going with this approach.
    DataSourceV2Strategy.translateFilterV2(sparkExpression) match {
      case Some(filter) =>
        val converted = SparkV2Filters.convert(filter)
        if (converted == null) {
          throw new IllegalArgumentException(
            s"Cannot convert Spark filter: $filter to Iceberg expression")
        }

        converted
      case _ =>
        throw new IllegalArgumentException(
          s"Cannot translate Spark expression: $sparkExpression to data source filter")
    }
  }

  @throws[IcebergAnalysisException]
  def collectResolvedSparkExpression(
      session: SparkSession,
      tableName: String,
      where: String): Expression = {
    val tableAttrs = session.table(tableName).queryExecution.analyzed.output
    val unresolvedExpression = session.sessionState.sqlParser.parseExpression(where)
    val filter = Filter(unresolvedExpression, DummyRelation(tableAttrs))

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Simplify or rewrite the predicate using supported types/operators so it converts cleanly
2. Cast or restructure the filter (e.g. compare scalar columns rather than nested fields) to a supported form
3. Check the Spark/Iceberg version pair for known conversion gaps and upgrade the runtime
4. As a workaround, disable pushdown for the problematic predicate so it is evaluated post-scan (note this affects performance)

Example fix

// before (filter that fails conversion)
df.filter($"struct_col" === someStructValue)
// after
import org.apache.spark.sql.functions.col
df.filter(col("struct_col.field") === lit(someValue))

import org.apache.spark.sql.connector.expressions.filter.Predicate
// Ensure predicates only use scalar columns with supported types before pushdown
val unsupported = df.schema.fields.filter(f => Seq("array", "map", "struct").contains(f.dataType.typeName))
if (unsupported.nonEmpty) {
  log.warn("Filters on complex types may fail Iceberg pushdown conversion; evaluate post-scan instead")
}

def isPushdownSafe(dataType: org.apache.spark.sql.types.DataType): Boolean =
  !dataType.isInstanceOf[org.apache.spark.sql.types.ArrayType] &&
  !dataType.isInstanceOf[org.apache.spark.sql.types.MapType] &&
  !dataType.isInstanceOf[org.apache.spark.sql.types.StructType]

try { icebergScan.filter(expr) } catch { case e: IllegalArgumentException if e.getMessage.startsWith("Cannot convert Spark filter") => log.warn("Pushdown unsupported for this predicate; falling back to post-scan filtering", e); postScanFilter(df, expr) }

When it happens

Trigger: Pushing down predicates containing types or functions SparkV2Filters.convert cannot handle — e.g. filters on complex/nested types, unsupported literal values, or a V2 filter that converts to null; typically hit inside Iceberg scans with pushed filters.

Common situations: Queries with exotic predicates (arrays/maps/structs comparisons, exotic casts) against Iceberg tables with filter pushdown enabled; Spark version upgrades changing translateFilterV2 output; custom predicates from third-party libraries.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Cannot convert Spark filter: $filter to Iceberg expression
• Cannot translate Spark expression: $sparkExpression to data
• Cannot convert unknown expression:
• AS OF is not supported for changelogs
• Unknown Spark table type:

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/89a5c33bb31be91a. Report an issue: GitHub.