apache/iceberg · warning

Failed to check if can be pushed down

Error message

Failed to check if {} can be pushed down: {}

What it means

A LOG.warn in BaseSparkScanBuilder.pushPredicates: while checking whether a Spark predicate can be converted and pushed down into Iceberg's expression model, an unexpected exception occurred. The predicate is conservatively kept as a post-scan (residual) filter, so results stay correct but the filter is evaluated after reading data instead of being pushed down.

Solutions

1. Inspect the logged predicate and exception message; rewrite the failing predicate into a pushable form (simple comparisons on primitive columns).
2. Move unsupported logic out of the WHERE clause (post-filter in a CTE/outer query) if pushdown is not essential.
3. If a Spark version introduced new filter types, upgrade Iceberg to a build matching your Spark version.
4. Correctness is unaffected — the row-level filter still applies; only performance (data skipping) is reduced.

Example fix

// before
df.filter(myUdf(col("x")) > 10) // UDF predicate fails pushdown check
// after
val pushed = df.filter(col("x") > 10) // pushable, enables data skipping
pushed.filter(myUdf(col("x")) > 10)

// Prefer simple comparison predicates on primitive columns for pushdown
// e.g. col > literal, col IN (...), isNotNull — avoid UDFs in filtered scans

try {
  scan = scanBuilder.filter(predicates).build();
} catch (Exception e) {
  LOG.warn("Predicate could not be pushed down; applying post-scan", e);
  // fall back to unfiltered scan + df.filter(...)
}

When it happens

Trigger: pushPredicates encountering a Spark Catalyst predicate whose translation via SparkV2Filters/SparkFilters throws — typically unsupported expression shapes (non-deterministic UDFs, exotic types, nested structs) or Spark version-specific filter classes Iceberg does not recognize.

Common situations: Queries with UDF-based or complex-type predicates on Iceberg tables; Spark upgrades introducing new filter types; third-party catalog plugins altering filter representation.

Related errors

• Failed to check if can be pushed down
• Failed to check if can be pushed down
• Cannot convert Spark filter: $filter to Iceberg expression
• Cannot translate Spark expression: $sparkExpression to data…
• Failed to close changelog scan:

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/80a00765d1ef0a2f. Report an issue: GitHub.

Appendix: source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/BaseSparkScanBuilder.java:176

    for (Predicate predicate : predicates) {
      try {
        Expression expr = SparkV2Filters.convert(predicate);

        if (expr != null) {
          // try binding the expression to ensure it can be pushed down
          Binder.bind(projection.asStruct(), expr, caseSensitive);
          expressions.add(expr);
          pushablePredicates.add(predicate);
        }

        if (expr == null || !ExpressionUtil.selectsPartitions(expr, table, caseSensitive)) {
          postScanPredicates.add(predicate);
        } else {
          LOG.info("Evaluating completely on Iceberg side: {}", predicate);
        }

      } catch (Exception e) {
        LOG.warn("Failed to check if {} can be pushed down: {}", predicate, e.getMessage());
        postScanPredicates.add(predicate);
      }
    }

    this.filters = expressions;
    this.pushedPredicates = pushablePredicates.toArray(new Predicate[0]);

    return postScanPredicates.toArray(new Predicate[0]);
  }

  // logic necessary for SupportsPushDownV2Filters
  public Predicate[] pushedPredicates() {
    return pushedPredicates;
  }

  // logic necessary for SupportsPushDownLimit
  public boolean pushLimit(int newLimit) {
    this.limit = newLimit;

View on GitHub (pinned to 86d9c8fc54)