apache/druid · error · org.apache.druid.error.DruidException

Iceberg filter produced residual expression that requires ro

Error message

Iceberg filter produced residual expression that requires row-level filtering. This typically means the filter is on a non-partition column. Residual rows may be ingested unless filtered by transformSpec. Residual filter: [%s]

What it means

In IcebergCatalog.extractSnapshotDataFiles, after planning Iceberg file scans a non-trivial residual() expression was found on a FileScanTask, meaning the supplied Iceberg filter could not be fully pushed down to partition/file pruning and requires row-level evaluation. When druid.iceberg.residual-filter-mode is FAIL, this raises a DruidException (DEVELOPER persona, RUNTIME_FAILURE) instructing that unfiltered residual rows may be ingested unless a transformSpec filter is added; otherwise it is only logged as a warning.

Source

Thrown at extensions-contrib/druid-iceberg-extensions/src/main/java/org/apache/druid/iceberg/input/IcebergCatalog.java:139

        }
      }

      // Handle residual filter based on mode
      if (detectedResidual != null) {
        String message = StringUtils.format(
            "Iceberg filter produced residual expression that requires row-level filtering. "
            + "This typically means the filter is on a non-partition column. "
            + "Residual rows may be ingested unless filtered by transformSpec. "
            + "Residual filter: [%s]",
            detectedResidual
        );

        if (residualFilterMode == ResidualFilterMode.FAIL) {
          throw DruidException.forPersona(DruidException.Persona.DEVELOPER)
                              .ofCategory(DruidException.Category.RUNTIME_FAILURE)
                              .build(message);
        }
        log.warn(message);
      }

      long duration = System.currentTimeMillis() - start;
      log.info("Data file scan and fetch took [%d ms] time for [%d] paths", duration, dataFilePaths.size());
    }
    catch (DruidException e) {
      throw e;
    }
    catch (Exception e) {
      throw new RE(e, "Failed to load iceberg table with identifier [%s]", tableIdentifier);
    }
    finally {
      Thread.currentThread().setContextClassLoader(currCtxClassloader);
    }
    return dataFilePaths;
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Add a matching filter in the ingestion transformSpec so residual rows are filtered at Druid level, then keep mode=warn (or safe for FAIL)
2. Re-partition the Iceberg table on the column you filter on (identity partitioning) so the filter can be fully pruned
3. Set residual filter mode to WARN/accept if full accuracy is guaranteed downstream and you just want the warning suppressed
4. Rewrite the filter to target partition columns, or split the query so partition-prunable predicates are pushed down and only small residual work remains

Example fix

// before: filter only in Iceberg scan, non-partition column
"filter": "country = 'US'"            // residual -> FAIL
// after: also filter in transformSpec
"transformSpec": {"filterExpr": "country = 'US'"}
// or change mode:
// druid.iceberg.residual-filter.mode = WARN

// preflight: ensure the Iceberg filter only references partition columns
java.util.Set<String> partitionCols = table.spec().identitySourceIds().keySet(); // plus transform columns
java.util.Set<String> filterCols = org.apache.druid.java.util.common.StringUtils
    // extract referenced columns from your filter spec before submission
// if any filter column is not a partition column, add a matching transformSpec filter first

try {
  catalog.extractSnapshotDataFiles(...);
} catch (DruidException e) {
  if (e.getCategory() == DruidException.Category.RUNTIME_FAILURE
      && e.getMessage().contains("residual expression")) {
    // add matching transformSpec filter, or switch residual-filter-mode to WARN, then retry
  } else {
    throw e;
  }
}

When it happens

Trigger: extractSnapshotDataFiles is called with an icebergFilter whose predicate targets a non-partition column (or a column with a transform Iceberg cannot prune on), so tableScan.planFiles() yields FileScanTask.residual() != alwaysTrue, and residualFilterMode == ResidualFilterMode.FAIL.

Common situations: Filtering on a regular (non-partition) column like 'country' when the table is partitioned by 'date'; using expressions Iceberg can't prune (e.g. complex transforms, non-identity partitions); upgrading Druid and setting residual filter mode to FAIL to catch silently-unpruned scans; users expecting Iceberg pushdown semantics identical to Druid-native filters.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Failed to initialize Glue catalog
• ColumnCapacityExceededException
• Bloom filter aggregators are query-time only
• Unsupported keyFormat. KafkaInputformat only supports input
• ORC flattener does not support JQ

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/5527ac862ed53960. Report an issue: GitHub.