{"record":{"id":"640f9ea29fea617c","repo":"apache/iceberg","slug":"failed-to-check-if-can-be-pushed-down-640f9e","errorCode":null,"errorMessage":"Failed to check if {} can be pushed down: {}","messagePattern":"Failed to check if (.+?) can be pushed down: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/BaseSparkScanBuilder.java","lineNumber":176,"sourceCode":"    for (Predicate predicate : predicates) {\n      try {\n        Expression expr = SparkV2Filters.convert(predicate);\n\n        if (expr != null) {\n          // try binding the expression to ensure it can be pushed down\n          Binder.bind(projection.asStruct(), expr, caseSensitive);\n          expressions.add(expr);\n          pushablePredicates.add(predicate);\n        }\n\n        if (expr == null || !ExpressionUtil.selectsPartitions(expr, table, caseSensitive)) {\n          postScanPredicates.add(predicate);\n        } else {\n          LOG.info(\"Evaluating completely on Iceberg side: {}\", predicate);\n        }\n\n      } catch (Exception e) {\n        LOG.warn(\"Failed to check if {} can be pushed down: {}\", predicate, e.getMessage());\n        postScanPredicates.add(predicate);\n      }\n    }\n\n    this.filters = expressions;\n    this.pushedPredicates = pushablePredicates.toArray(new Predicate[0]);\n\n    return postScanPredicates.toArray(new Predicate[0]);\n  }\n\n  // logic necessary for SupportsPushDownV2Filters\n  public Predicate[] pushedPredicates() {\n    return pushedPredicates;\n  }\n\n  // logic necessary for SupportsPushDownLimit\n  public boolean pushLimit(int newLimit) {\n    this.limit = newLimit;","sourceCodeStart":158,"sourceCodeEnd":194,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/BaseSparkScanBuilder.java#L158-L194","documentation":"A WARN log in BaseSparkScanBuilder.pushPredicates emitted when evaluating whether a Spark filter can be converted/pushed to Iceberg throws an exception. The predicate is conservatively kept as a post-scan (post-download) filter, so correctness is preserved but pushdown is lost and the scan reads more data than necessary.","triggerScenarios":"Calling SparkScanBuilder.pushFilters/pushPredicates with a filter whose Spark-to-Iceberg conversion or evaluation (e.g. SparkFilters.convert, Binder.bind, or supported-type checks) throws at runtime, such as exotic expressions, non-literal bounds, or unsupported types like nested struct fields in the filter.","commonSituations":"Queries with complex filters (UDF-derived literals, struct/array types, NaN or malformed literals) pushed into a Spark Iceberg read; newer Spark filter shapes not yet handled by the converter.","solutions":["Inspect the full exception via DEBUG logging or the code path and identify which predicate fails conversion","Simplify or rewrite the offending WHERE clause into supported Iceberg predicate forms (comparisons on primitive columns)","Upgrade Iceberg to a version with broader Spark filter conversion support","If acceptable, leave it: the filter is still applied post-scan, only performance is affected"],"exampleFix":"// before\nSELECT * FROM t WHERE struct_col.field > 10\n// after\nSELECT * FROM t WHERE struct_col.field > 10 /* accept post-scan eval, or rewrite */\nSELECT * FROM t WHERE plain_col > 10","handlingStrategy":"fallback","validationCode":"// check predicate columns are simple primitives before pushing\nsparkSession.sql(\"SELECT * FROM t WHERE col > 10\").explain(true); // verify PushedFilters in plan","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep WHERE clauses on primitive columns with literal comparisons","Check the physical plan for pushed filters after query changes","Monitor logs during query development to catch lost pushdown early"],"tags":["spark","predicate-pushdown","scan-planning"],"backgroundTag":"unsupported-operation","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}