apache/iceberg · error · IllegalArgumentException

Could not resolve using columns

Error message

Could not resolve %s using columns: %s

What it means

SparkTableUtil's partition-filter resolution resolves UnresolvedAttribute column references from a user-provided expression against the plan's output attributes using the configured case-sensitivity resolver. When a referenced column name cannot be resolved among the plan's output columns, this IllegalArgumentException is thrown, listing the attribute and the available columns.

Solutions

1. Correct the column name in the filter expression to match the table's output columns exactly.
2. Match column case: with caseSensitive mode enabled, use the exact case from the schema, or disable case sensitivity in the resolver.
3. Drop table qualifiers/aliases from the attribute nameParts; pass bare column names.
4. Print plan.output() (the columns listed in the error message) and validate filter columns against it before calling the util.

Example fix

// before
SparkTableUtil.partitionFilter(spark, table, functions.col("partition_date"), ...) // unresolvable
// after
SparkTableUtil.partitionFilter(spark, table, functions.col("date"), ...) // matches plan.output()

Dataset<Row> df = spark.table(tableName);
java.util.Set<String> cols = java.util.Arrays.stream(df.schema().fieldNames())
    .collect(java.util.stream.Collectors.toSet());
if (!cols.contains("date")) { throw new IllegalArgumentException("unknown filter column: date, available: " + cols); }

boolean columnExists = java.util.Arrays.asList(spark.table(tableName).schema().fieldNames())
    .contains(columnName);

try {
  SparkTableUtil.partitionFilter(spark, table, expr, ...);
} catch (IllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Could not resolve")) { /* fix column name or relax case sensitivity */ }
  else throw e;
}

When it happens

Trigger: Calling SparkTableUtil.partitionFilter / filterPartitions with an expression referencing a column name that does not exist (or differs in case with case-sensitive resolution) in the loaded table's schema.

Common situations: Typos in partition column names; case mismatch ('Date' vs 'date') when spark.sql.caseSensitive=true; passing Spark column names qualified with table alias not present in plan.output(); using columns from the data schema that are not part of the resolved plan output.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Failed to check if can be pushed down
• Already closed files for partition:
• ALTER TABLE contains multiple distribution clauses
• ALTER TABLE contains multiple distribution clauses
• ALTER TABLE contains multiple distribution clauses

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/ed25dc5f30858204. Report an issue: GitHub.

Appendix: source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java:350

    Map<String, String> partitionSpec =
        JavaConverters.mapAsJavaMapConverter(partition.spec()).asJava();
    return new SparkPartition(partitionSpec, uri, format);
  }

  private static Expression resolveAttrs(SparkSession spark, String table, Expression expr) {
    Function2<String, String, Object> resolver = spark.sessionState().analyzer().resolver();
    LogicalPlan plan = spark.table(table).queryExecution().analyzed();
    return expr.transform(
        new AbstractPartialFunction<Expression, Expression>() {
          @Override
          public Expression apply(Expression attr) {
            UnresolvedAttribute unresolvedAttribute = (UnresolvedAttribute) attr;
            Option<NamedExpression> namedExpressionOption =
                plan.resolve(unresolvedAttribute.nameParts(), resolver);
            if (namedExpressionOption.isDefined()) {
              return (Expression) namedExpressionOption.get();
            } else {
              throw new IllegalArgumentException(
                  String.format("Could not resolve %s using columns: %s", attr, plan.output()));
            }
          }

          @Override
          public boolean isDefinedAt(Expression attr) {
            return attr instanceof UnresolvedAttribute;
          }
        });
  }

  private static Iterator<ManifestFile> buildManifest(
      int formatVersion,
      Long snapshotId,
      SerializableConfiguration conf,
      PartitionSpec spec,
      String basePath,
      Iterator<Tuple2<String, DataFile>> fileTuples) {

View on GitHub (pinned to 86d9c8fc54)