{"record":{"id":"ed25dc5f30858204","repo":"apache/iceberg","slug":"could-not-resolve-s-using-columns-s","errorCode":null,"errorMessage":"Could not resolve %s using columns: %s","messagePattern":"Could not resolve (.+?) using columns: (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java","lineNumber":350,"sourceCode":"    Map<String, String> partitionSpec =\n        JavaConverters.mapAsJavaMapConverter(partition.spec()).asJava();\n    return new SparkPartition(partitionSpec, uri, format);\n  }\n\n  private static Expression resolveAttrs(SparkSession spark, String table, Expression expr) {\n    Function2<String, String, Object> resolver = spark.sessionState().analyzer().resolver();\n    LogicalPlan plan = spark.table(table).queryExecution().analyzed();\n    return expr.transform(\n        new AbstractPartialFunction<Expression, Expression>() {\n          @Override\n          public Expression apply(Expression attr) {\n            UnresolvedAttribute unresolvedAttribute = (UnresolvedAttribute) attr;\n            Option<NamedExpression> namedExpressionOption =\n                plan.resolve(unresolvedAttribute.nameParts(), resolver);\n            if (namedExpressionOption.isDefined()) {\n              return (Expression) namedExpressionOption.get();\n            } else {\n              throw new IllegalArgumentException(\n                  String.format(\"Could not resolve %s using columns: %s\", attr, plan.output()));\n            }\n          }\n\n          @Override\n          public boolean isDefinedAt(Expression attr) {\n            return attr instanceof UnresolvedAttribute;\n          }\n        });\n  }\n\n  private static Iterator<ManifestFile> buildManifest(\n      int formatVersion,\n      Long snapshotId,\n      SerializableConfiguration conf,\n      PartitionSpec spec,\n      String basePath,\n      Iterator<Tuple2<String, DataFile>> fileTuples) {","sourceCodeStart":332,"sourceCodeEnd":368,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java#L332-L368","documentation":"SparkTableUtil's partition-filter resolution resolves UnresolvedAttribute column references from a user-provided expression against the plan's output attributes using the configured case-sensitivity resolver. When a referenced column name cannot be resolved among the plan's output columns, this IllegalArgumentException is thrown, listing the attribute and the available columns.","triggerScenarios":"Calling SparkTableUtil.partitionFilter / filterPartitions with an expression referencing a column name that does not exist (or differs in case with case-sensitive resolution) in the loaded table's schema.","commonSituations":"Typos in partition column names; case mismatch ('Date' vs 'date') when spark.sql.caseSensitive=true; passing Spark column names qualified with table alias not present in plan.output(); using columns from the data schema that are not part of the resolved plan output.","solutions":["Correct the column name in the filter expression to match the table's output columns exactly.","Match column case: with caseSensitive mode enabled, use the exact case from the schema, or disable case sensitivity in the resolver.","Drop table qualifiers/aliases from the attribute nameParts; pass bare column names.","Print plan.output() (the columns listed in the error message) and validate filter columns against it before calling the util."],"exampleFix":"// before\nSparkTableUtil.partitionFilter(spark, table, functions.col(\"partition_date\"), ...) // unresolvable\n// after\nSparkTableUtil.partitionFilter(spark, table, functions.col(\"date\"), ...) // matches plan.output()","handlingStrategy":"validation","validationCode":"Dataset<Row> df = spark.table(tableName);\njava.util.Set<String> cols = java.util.Arrays.stream(df.schema().fieldNames())\n    .collect(java.util.stream.Collectors.toSet());\nif (!cols.contains(\"date\")) { throw new IllegalArgumentException(\"unknown filter column: date, available: \" + cols); }","typeGuard":"boolean columnExists = java.util.Arrays.asList(spark.table(tableName).schema().fieldNames())\n    .contains(columnName);","tryCatchPattern":"try {\n  SparkTableUtil.partitionFilter(spark, table, expr, ...);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Could not resolve\")) { /* fix column name or relax case sensitivity */ }\n  else throw e;\n}","preventionTips":["Validate filter column names against table schema before building expressions.","Account for spark.sql.caseSensitive when mixing column cases.","Use bare (unqualified) column names matching plan output; avoid aliases.","Copy the available-columns list from the error message to correct the name."],"tags":["unresolved-attribute","column-resolution","spark","filter"],"backgroundTag":"resource-not-found","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}