apache/iceberg · error · IcebergAnalysisException

Failed to find filter expression

Error message

Failed to find filter expression

What it means

collectResolvedSparkExpression scans the optimized logical plan for the Filter condition that Spark wants Iceberg to explain/convert, and throws IcebergAnalysisException when no Filter node is found. The helper is only valid for plans that contain exactly one filter; a plan without a Filter (or DummyRelation/LocalRelation stand-in) means there is no resolvable filter expression to collect.

Solutions

1. Ensure the DataFrame passed in actually has a WHERE/Filter condition that survives to the optimized plan.
2. Verify the plan via df.queryExecution.optimizedPlan before calling; if the filter was optimized away (e.g. Literal.TrueLiteral), skip the call instead of invoking the resolver.
3. If you intentionally want a trivially-true/false result, model your input with the corresponding case (DummyRelation / LocalRelation) rather than a plan with no filter.
4. Update to a newer Iceberg release if your Spark optimizer legitimately produces a Filter shape this collector doesn't recognize.

Example fix

// before
val filtered = spark.table("t") // no filter -> no Filter node in plan
procedure.collectResolvedSparkExpression(filtered.queryExecution)
// after
val filtered = spark.table("t").where($"ts" < lit("2024-01-01"))

val hasFilter = df.queryExecution.optimizedPlan.exists {
  case _: org.apache.spark.sql.catalyst.plans.logical.Filter => true
  case _ => false
}
require(hasFilter, "Plan must contain a Filter node before collecting the resolved expression")

def planHasFilter(plan: LogicalPlan): Boolean = plan.exists {
  case _: Filter => true
  case _: DummyRelation | _: LocalRelation => true
  case _ => false
}

When it happens

Trigger: Invoking collectResolvedSparkExpression (used by procedures like the extended filter-resolution path, e.g. rewrite/expire procedures that take a filter) with an optimized plan that has no Filter operator — e.g. the input DataFrame has no WHERE clause, or the optimizer rewrote/eliminated the filter before this point.

Common situations: Passing a DataFrame with no predicate to a procedure expecting a filtered plan; the filter being pushed below/away by Catalyst optimization (constant-folding a FALSE predicate); wiring the helper against a plan produced from a different query.

Understand the failure class

Background: EmptyResultError / "no results found": when an API or scraper succeeds but returns zero rows — this error's family across 9 libraries.

Related errors

• Failed to find filter expression
• Failed to find filter expression
• Already closed files for partition:
• ALTER TABLE contains multiple distribution clauses
• ALTER TABLE contains multiple distribution clauses

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/8e3ce9701c2c5284. Report an issue: GitHub.

Appendix: source

Thrown at spark/v4.1/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkExpressionConverter.scala:69

    }
  }

  @throws[IcebergAnalysisException]
  def collectResolvedSparkExpression(
      session: SparkSession,
      tableName: String,
      where: String): Expression = {
    val tableAttrs = session.table(tableName).queryExecution.analyzed.output
    val unresolvedExpression = session.sessionState.sqlParser.parseExpression(where)
    val filter = Filter(unresolvedExpression, DummyRelation(tableAttrs))
    val optimizedLogicalPlan = session.sessionState.executePlan(filter).optimizedPlan
    optimizedLogicalPlan
      .collectFirst {
        case filter: Filter => filter.condition
        case _: DummyRelation => Literal.TrueLiteral
        case _: LocalRelation => Literal.FalseLiteral
      }
      .getOrElse(throw new IcebergAnalysisException("Failed to find filter expression"))
  }

  case class DummyRelation(output: Seq[Attribute]) extends LeafNode
}

View on GitHub (pinned to 86d9c8fc54)