{"record":{"id":"c3274f4b43287230","repo":"apache/iceberg","slug":"failed-to-check-if-can-be-pushed-down","errorCode":null,"errorMessage":"Failed to check if {} can be pushed down: {}","messagePattern":"Failed to check if (.+?) can be pushed down: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java","lineNumber":186,"sourceCode":"        Expression expr = SparkV2Filters.convert(predicate);\n\n        if (expr != null) {\n          // try binding the expression to ensure it can be pushed down\n          Binder.bind(schema.asStruct(), expr, caseSensitive);\n          expressions.add(expr);\n          pushableFilters.add(predicate);\n        }\n\n        if (expr == null\n            || unpartitioned()\n            || !ExpressionUtil.selectsPartitions(expr, table, caseSensitive)) {\n          postScanFilters.add(predicate);\n        } else {\n          LOG.info(\"Evaluating completely on Iceberg side: {}\", predicate);\n        }\n\n      } catch (Exception e) {\n        LOG.warn(\"Failed to check if {} can be pushed down: {}\", predicate, e.getMessage());\n        postScanFilters.add(predicate);\n      }\n    }\n\n    this.filterExpressions = expressions;\n    this.pushedPredicates = pushableFilters.toArray(new Predicate[0]);\n\n    return postScanFilters.toArray(new Predicate[0]);\n  }\n\n  private boolean unpartitioned() {\n    return table.specs().values().stream().noneMatch(PartitionSpec::isPartitioned);\n  }\n\n  @Override\n  public Predicate[] pushedPredicates() {\n    return pushedPredicates;\n  }","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java#L168-L204","documentation":"During scan planning SparkScanBuilder.pushPredicates evaluates each Spark filter to decide whether it can be translated to an Iceberg Predicate and pushed into the scan. If translation/evaluation throws for any reason, this warning is logged and the filter is kept as a post-scan (residual) filter instead — the query still runs correctly, just without pushdown.","triggerScenarios":"Calling a DataFrame/SQL query with a filter expression that SparkExpressionConverter cannot map (e.g. unusual UDF-derived functions, non-deterministic expressions, struct/map edge cases, incompatible type casts) while Iceberg attempts pushdown in pushPredicates.","commonSituations":"Queries with custom Catalyst expressions or third-party functions on Iceberg tables; version mismatches between Spark and Iceberg where expression converters lag new Spark functions; filters over columns whose types don't match the schema after evolution.","solutions":["Read the logged exception message to identify which predicate failed and why.","Rewrite the offending filter using supported expressions/operators that Iceberg can push down.","Check Iceberg version — upgrade if your Spark functions were added after the connector's converter support.","Verify data source is still correct: the filter is applied post-scan, so results are right but may read more data; check scanned-file metrics to assess impact."],"exampleFix":"// before: unpushable filter keeps whole-table scan\ndf.filter(callUDF(\"myfunc\", col(\"ts\")).gt(lit(0)))\n\n// after: pushable native predicate\nlong ts = ...;\ndf.filter(functions.col(\"ts\").gt(functions.lit(ts)));","handlingStrategy":"validation","validationCode":"// Pre-check whether a filter is pushable\nExpression expr = SparkExpressionConverter.convertToExpression(table.schema(), sparkPredicate); // throws if unsupported\nif (expr == null) {\n  LOG.info(\"Filter {} will not be pushed down; consider rewriting\", sparkPredicate);\n}","typeGuard":null,"tryCatchPattern":"try {\n  converter.convertToExpression(schema, sparkPredicate);\n} catch (Exception e) {\n  LOG.warn(\"Unpushable filter, keeping as post-scan: {}\", sparkPredicate, e);\n}","preventionTips":["Use built-in Spark operators on schema columns for filters","Avoid UDFs and non-deterministic expressions in WHERE clauses on Iceberg tables","Keep Spark connector and Spark versions matched","Check scanned-file metrics after queries to detect lost pushdown"],"tags":["spark","predicate-pushdown","fallback"],"backgroundTag":"predicate-pushdown-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}