apache/iceberg · error

Cannot translate Spark expression: $sparkExpression to data

Error message

Cannot translate Spark expression: $sparkExpression to data source filter

What it means

convertFilter first asks Spark's DataSourceV2Strategy.translateFilterV2 to lower the Spark expression into a DataSource V2 filter. If Spark cannot produce any V2 filter for the expression, this IllegalArgumentException is thrown, since without a V2 filter there is nothing to convert to an Iceberg expression.

Source

Thrown at spark/v3.5/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala:49

object SparkExpressionConverter {

  def convertToIcebergExpression(
      sparkExpression: Expression): org.apache.iceberg.expressions.Expression = {
    // Currently, it is a double conversion as we are converting Spark expression to Spark predicate
    // and then converting Spark predicate to Iceberg expression.
    // But these two conversions already exist and well tested. So, we are going with this approach.
    DataSourceV2Strategy.translateFilterV2(sparkExpression) match {
      case Some(filter) =>
        val converted = SparkV2Filters.convert(filter)
        if (converted == null) {
          throw new IllegalArgumentException(
            s"Cannot convert Spark filter: $filter to Iceberg expression")
        }

        converted
      case _ =>
        throw new IllegalArgumentException(
          s"Cannot translate Spark expression: $sparkExpression to data source filter")
    }
  }

  @throws[AnalysisException]
  def collectResolvedSparkExpression(
      session: SparkSession,
      tableName: String,
      where: String): Expression = {
    val tableAttrs = session.table(tableName).queryExecution.analyzed.output
    val unresolvedExpression = session.sessionState.sqlParser.parseExpression(where)
    val filter = Filter(unresolvedExpression, DummyRelation(tableAttrs))
    val optimizedLogicalPlan = session.sessionState.executePlan(filter).optimizedPlan
    optimizedLogicalPlan
      .collectFirst {
        case filter: Filter => filter.condition
        case _: DummyRelation => Literal.TrueLiteral
        case _: LocalRelation => Literal.FalseLiteral

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Simplify or rewrite the predicate to forms Spark can lower to V2 filters (=, <, >, IN, IS NULL, AND/OR/NOT)
2. Avoid non-deterministic or UDF-based predicates in pushdown contexts
3. Check Spark and Iceberg versions are a matched pair

Example fix

// before
SparkExpressionConverter.convertFilter(schema, expr("rand() > 0.5")) // not translatable
// after
SparkExpressionConverter.convertFilter(schema, expr("id > 5"))

val lowered = DataSourceV2Strategy.translateFilterV2(sparkExpr)
if (lowered.isEmpty) logWarning(s"Expression ${sparkExpr.sql} cannot be lowered to a V2 filter")

try {
  val icebergExpr = SparkExpressionConverter.convertFilter(schema, sparkExpr)
} catch {
  case e: IllegalArgumentException if e.getMessage.startsWith("Cannot translate Spark expression") =>
    // fall back to in-Spark evaluation of the predicate
}

When it happens

Trigger: Calling convertFilter with a Spark expression that Spark's translateFilterV2 returns None for — expressions Spark itself cannot lower to a data source filter (complex expressions, non-deterministic functions, unsupported operators).

Common situations: Pushing down user-defined functions or non-deterministic predicates; passing unresolved or non-boolean expressions; Spark/Iceberg version mismatches where an expression is not yet covered by translateFilterV2.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Cannot convert Spark filter: $filter to Iceberg expression
• Cannot convert Spark filter: $filter to Iceberg expression
• Cannot convert unknown expression:
• AS OF is not supported for changelogs
• Unknown Spark table type:

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/25a2c0d60d52a0f8. Report an issue: GitHub.