{"record":{"id":"80a00765d1ef0a2f","repo":"apache/iceberg","slug":"failed-to-check-if-can-be-pushed-down-80a007","errorCode":null,"errorMessage":"Failed to check if {} can be pushed down: {}","messagePattern":"Failed to check if (.+?) can be pushed down: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/BaseSparkScanBuilder.java","lineNumber":176,"sourceCode":"    for (Predicate predicate : predicates) {\n      try {\n        Expression expr = SparkV2Filters.convert(predicate);\n\n        if (expr != null) {\n          // try binding the expression to ensure it can be pushed down\n          Binder.bind(projection.asStruct(), expr, caseSensitive);\n          expressions.add(expr);\n          pushablePredicates.add(predicate);\n        }\n\n        if (expr == null || !ExpressionUtil.selectsPartitions(expr, table, caseSensitive)) {\n          postScanPredicates.add(predicate);\n        } else {\n          LOG.info(\"Evaluating completely on Iceberg side: {}\", predicate);\n        }\n\n      } catch (Exception e) {\n        LOG.warn(\"Failed to check if {} can be pushed down: {}\", predicate, e.getMessage());\n        postScanPredicates.add(predicate);\n      }\n    }\n\n    this.filters = expressions;\n    this.pushedPredicates = pushablePredicates.toArray(new Predicate[0]);\n\n    return postScanPredicates.toArray(new Predicate[0]);\n  }\n\n  // logic necessary for SupportsPushDownV2Filters\n  public Predicate[] pushedPredicates() {\n    return pushedPredicates;\n  }\n\n  // logic necessary for SupportsPushDownLimit\n  public boolean pushLimit(int newLimit) {\n    this.limit = newLimit;","sourceCodeStart":158,"sourceCodeEnd":194,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/BaseSparkScanBuilder.java#L158-L194","documentation":"A LOG.warn in BaseSparkScanBuilder.pushPredicates: while checking whether a Spark predicate can be converted and pushed down into Iceberg's expression model, an unexpected exception occurred. The predicate is conservatively kept as a post-scan (residual) filter, so results stay correct but the filter is evaluated after reading data instead of being pushed down.","triggerScenarios":"pushPredicates encountering a Spark Catalyst predicate whose translation via SparkV2Filters/SparkFilters throws — typically unsupported expression shapes (non-deterministic UDFs, exotic types, nested structs) or Spark version-specific filter classes Iceberg does not recognize.","commonSituations":"Queries with UDF-based or complex-type predicates on Iceberg tables; Spark upgrades introducing new filter types; third-party catalog plugins altering filter representation.","solutions":["Inspect the logged predicate and exception message; rewrite the failing predicate into a pushable form (simple comparisons on primitive columns).","Move unsupported logic out of the WHERE clause (post-filter in a CTE/outer query) if pushdown is not essential.","If a Spark version introduced new filter types, upgrade Iceberg to a build matching your Spark version.","Correctness is unaffected — the row-level filter still applies; only performance (data skipping) is reduced."],"exampleFix":"// before\ndf.filter(myUdf(col(\"x\")) > 10) // UDF predicate fails pushdown check\n// after\nval pushed = df.filter(col(\"x\") > 10) // pushable, enables data skipping\npushed.filter(myUdf(col(\"x\")) > 10)","handlingStrategy":"fallback","validationCode":"// Prefer simple comparison predicates on primitive columns for pushdown\n// e.g. col > literal, col IN (...), isNotNull — avoid UDFs in filtered scans","typeGuard":null,"tryCatchPattern":"try {\n  scan = scanBuilder.filter(predicates).build();\n} catch (Exception e) {\n  LOG.warn(\"Predicate could not be pushed down; applying post-scan\", e);\n  // fall back to unfiltered scan + df.filter(...)\n}","preventionTips":["Keep scan filters to simple comparisons on primitive columns","Avoid non-deterministic UDFs inside WHERE clauses on Iceberg scans","Upgrade Iceberg and Spark in lockstep to keep filter translation current","Treat this warning as a performance signal, not a correctness issue"],"tags":["spark","predicate-pushdown","scan-planning","fallback"],"backgroundTag":"predicate-pushdown-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}