{"record":{"id":"5527ac862ed53960","repo":"apache/druid","slug":"iceberg-filter-produced-residual-expression-that-r","errorCode":null,"errorMessage":"Iceberg filter produced residual expression that requires row-level filtering. This typically means the filter is on a non-partition column. Residual rows may be ingested unless filtered by transformSpec. Residual filter: [%s]","messagePattern":"Iceberg filter produced residual expression that requires row-level filtering\\. This typically means the filter is on a non-partition column\\. Residual rows may be ingested unless filtered by transformSpec\\. Residual filter: \\[(.+?)\\]","errorType":"console","errorClass":"org.apache.druid.error.DruidException","httpStatus":null,"severity":"error","filePath":"extensions-contrib/druid-iceberg-extensions/src/main/java/org/apache/druid/iceberg/input/IcebergCatalog.java","lineNumber":139,"sourceCode":"        }\n      }\n\n      // Handle residual filter based on mode\n      if (detectedResidual != null) {\n        String message = StringUtils.format(\n            \"Iceberg filter produced residual expression that requires row-level filtering. \"\n            + \"This typically means the filter is on a non-partition column. \"\n            + \"Residual rows may be ingested unless filtered by transformSpec. \"\n            + \"Residual filter: [%s]\",\n            detectedResidual\n        );\n\n        if (residualFilterMode == ResidualFilterMode.FAIL) {\n          throw DruidException.forPersona(DruidException.Persona.DEVELOPER)\n                              .ofCategory(DruidException.Category.RUNTIME_FAILURE)\n                              .build(message);\n        }\n        log.warn(message);\n      }\n\n      long duration = System.currentTimeMillis() - start;\n      log.info(\"Data file scan and fetch took [%d ms] time for [%d] paths\", duration, dataFilePaths.size());\n    }\n    catch (DruidException e) {\n      throw e;\n    }\n    catch (Exception e) {\n      throw new RE(e, \"Failed to load iceberg table with identifier [%s]\", tableIdentifier);\n    }\n    finally {\n      Thread.currentThread().setContextClassLoader(currCtxClassloader);\n    }\n    return dataFilePaths;\n  }\n}\n","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/druid-iceberg-extensions/src/main/java/org/apache/druid/iceberg/input/IcebergCatalog.java#L121-L157","documentation":"In IcebergCatalog.extractSnapshotDataFiles, after planning Iceberg file scans a non-trivial residual() expression was found on a FileScanTask, meaning the supplied Iceberg filter could not be fully pushed down to partition/file pruning and requires row-level evaluation. When druid.iceberg.residual-filter-mode is FAIL, this raises a DruidException (DEVELOPER persona, RUNTIME_FAILURE) instructing that unfiltered residual rows may be ingested unless a transformSpec filter is added; otherwise it is only logged as a warning.","triggerScenarios":"extractSnapshotDataFiles is called with an icebergFilter whose predicate targets a non-partition column (or a column with a transform Iceberg cannot prune on), so tableScan.planFiles() yields FileScanTask.residual() != alwaysTrue, and residualFilterMode == ResidualFilterMode.FAIL.","commonSituations":"Filtering on a regular (non-partition) column like 'country' when the table is partitioned by 'date'; using expressions Iceberg can't prune (e.g. complex transforms, non-identity partitions); upgrading Druid and setting residual filter mode to FAIL to catch silently-unpruned scans; users expecting Iceberg pushdown semantics identical to Druid-native filters.","solutions":["Add a matching filter in the ingestion transformSpec so residual rows are filtered at Druid level, then keep mode=warn (or safe for FAIL)","Re-partition the Iceberg table on the column you filter on (identity partitioning) so the filter can be fully pruned","Set residual filter mode to WARN/accept if full accuracy is guaranteed downstream and you just want the warning suppressed","Rewrite the filter to target partition columns, or split the query so partition-prunable predicates are pushed down and only small residual work remains"],"exampleFix":"// before: filter only in Iceberg scan, non-partition column\n\"filter\": \"country = 'US'\"            // residual -> FAIL\n// after: also filter in transformSpec\n\"transformSpec\": {\"filterExpr\": \"country = 'US'\"}\n// or change mode:\n// druid.iceberg.residual-filter.mode = WARN","handlingStrategy":"validation","validationCode":"// preflight: ensure the Iceberg filter only references partition columns\njava.util.Set<String> partitionCols = table.spec().identitySourceIds().keySet(); // plus transform columns\njava.util.Set<String> filterCols = org.apache.druid.java.util.common.StringUtils\n    // extract referenced columns from your filter spec before submission\n// if any filter column is not a partition column, add a matching transformSpec filter first","typeGuard":null,"tryCatchPattern":"try {\n  catalog.extractSnapshotDataFiles(...);\n} catch (DruidException e) {\n  if (e.getCategory() == DruidException.Category.RUNTIME_FAILURE\n      && e.getMessage().contains(\"residual expression\")) {\n    // add matching transformSpec filter, or switch residual-filter-mode to WARN, then retry\n  } else {\n    throw e;\n  }\n}","preventionTips":["Only filter on identity-partition columns if you rely on full pushdown","Always mirror Iceberg filters in the Druid transformSpec filterExpr so residual rows are filtered regardless of mode","Keep residual-filter-mode=FAIL in CI to catch non-prunable filters before production ingestion","Re-partition tables used heavily with point filters (e.g. by the filtered dimension)"],"tags":["iceberg","pushdown","residual-filter","ingestion"],"backgroundTag":"unsupported-operation","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}