apache/iceberg · error · IllegalArgumentException

ID column not found

Error message

ID column not found: ${colName}

What it means

When building the table writer with identifier (ID) columns, each configured ID column is looked up in the target Iceberg table schema. If schema.findField returns null the column does not exist and an IllegalArgumentException is thrown.

Solutions

1. Fix the configured column name to exactly match a field in the Iceberg table schema (names are case-sensitive).
2. Verify with SHOW CREATE TABLE / Spark DESCRIBE which columns exist; enable auto-create/evolution (iceberg.tables.auto-evolution.enabled=true) if upstream added new columns.
3. Ensure the key-field config aligns with fields produced by the converter.

Example fix

// before
"iceberg.columns": "user_id,ordertime"
// after (schema field is actually 'userId')
"iceberg.columns": "userId,ordertime"

for (String col : idCols) {
  if (table.schema().findField(col) == null)
    throw new ConfigException("ID column missing from table schema: " + col);
}

try { writer = createTableWriter(...); }
catch (IllegalArgumentException e) { LOG.error("config/schema mismatch: {}", e.getMessage()); throw e; }

When it happens

Trigger: Creating/updating a table from the Kafka Connect sink with ID columns (iceberg.columns config / identifier fields, usually for upserts) where a configured column name is misspelled or absent from the schema.

Common situations: Case-sensitivity mismatches (schema uses 'UserID', config says 'userid'); column added later upstream but table auto-evolution off; typos in iceberg.columns for key fields.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Unsupported type
• An error occurred closing catalog instance, ignoring...
• An error occurred converting record, topic
• Cannot convert date
• Cannot convert java.util.Date to variant without a…

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d84ea2f001e7fa7f. Report an issue: GitHub.

Appendix: source

Thrown at kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordUtils.java:125

    long targetFileSize =
        PropertyUtil.propertyAsLong(
            tableProps,
            TableProperties.WRITE_TARGET_FILE_SIZE_BYTES,
            TableProperties.WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT);

    Set<Integer> identifierFieldIds = table.schema().identifierFieldIds();

    // override the identifier fields if the config is set
    List<String> idCols = config.tableConfig(tableReference.identifier().toString()).idColumns();
    if (!idCols.isEmpty()) {
      identifierFieldIds =
          idCols.stream()
              .map(
                  colName -> {
                    NestedField field = table.schema().findField(colName);
                    if (field == null) {
                      throw new IllegalArgumentException("ID column not found: " + colName);
                    }
                    return field.fieldId();
                  })
              .collect(Collectors.toSet());
    }

    FileWriterFactory<Record> writerFactory;
    if (identifierFieldIds == null || identifierFieldIds.isEmpty()) {
      writerFactory =
          new GenericFileWriterFactory.Builder(table)
              .dataSchema(table.schema())
              .dataFileFormat(format)
              .writerProperties(tableProps)
              .build();
    } else {
      writerFactory =
          new GenericFileWriterFactory.Builder(table)
              .dataSchema(table.schema())

View on GitHub (pinned to 86d9c8fc54)