What causes "Failed to find field '${key}' in schema ${schema.name()}" in apache/iceberg?

Ensure all documents in the collection share the same field set, or remove/null out inconsistent sub-documents.

apache/iceberg · error · DataException

Failed to find field

Error message

Failed to find field '${key}' in schema ${schema.name()}

What it means

MongoDataConverter.convertFieldValue throws DataException when converting a BSON DOCUMENT field: the matching Field cannot be found by key in the target Kafka Connect struct schema. The schema was built from one document but a record contains a document with fields absent from that schema.

Solutions

Ensure all documents in the collection share the same field set, or remove/null out inconsistent sub-documents.
Re-run schema inference so the Connect schema includes the new field.
Pre-create a schema covering all expected document keys.

Defensive patterns

Strategy: type-guard

Validate before calling

// Before converting, verify every document key exists in the schema
BsonDocument doc = ...;
for (String k : doc.keySet()) {
  if (schema.field(k) == null) throw new IllegalStateException("document key missing from schema: " + k);
}

Type guard

boolean schemaCovers(org.apache.kafka.connect.data.Schema schema, BsonDocument doc) { return doc.keySet().stream().allMatch(k -> schema.field(k) != null); }

Try / catch

try { ... } catch (org.apache.kafka.connect.errors.DataException e) { log.error("Schema drift detected: {}", e.getMessage()); /* re-infer schema or DLQ */ }

Prevention

Keep MongoDB documents homogeneous within a collection
Version schema changes and re-run schema inference before migrating new fields
Monitor for new fields with a schema validation rule in MongoDB

When it happens

Trigger: Calling convertRecord/convertFieldValue on a record whose BSON document contains a sub-document key that is not present in the pre-built Connect schema for that struct (schema.name() lookup returns null).

Common situations: MongoDB collections with heterogeneous documents — new fields added to some documents after the schema was inferred from the first sample; schema drift between documents in the same collection.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/2e23db0fb2b7939c. Report an issue: GitHub.

Appendix: source

Thrown at kafka-connect/kafka-connect-transforms/src/main/java/org/debezium/connector/mongodb/transforms/MongoDataConverter.java:143

      case REGULAR_EXPRESSION:
        Struct regexStruct = new Struct(schema.field(key).schema());
        regexStruct.put("regex", keyValueForStruct.getValue().asRegularExpression().getPattern());
        regexStruct.put("options", keyValueForStruct.getValue().asRegularExpression().getOptions());
        colValue = regexStruct;
        break;

      case TIMESTAMP:
        colValue = new Date(1000L * keyValueForStruct.getValue().asTimestamp().getTime());
        break;

      case DECIMAL128:
        colValue = keyValueForStruct.getValue().asDecimal128().getValue().toString();
        break;

      case DOCUMENT:
        Field field = schema.field(key);
        if (field == null) {
          throw new DataException("Failed to find field '" + key + "' in schema " + schema.name());
        }
        Schema documentSchema = field.schema();
        Struct documentStruct = new Struct(documentSchema);
        BsonDocument docs = keyValueForStruct.getValue().asDocument();

        for (Entry<String, BsonValue> doc : docs.entrySet()) {
          convertFieldValue(doc, documentStruct, documentSchema);
        }

        colValue = documentStruct;
        break;

      case ARRAY:
        if (keyValueForStruct.getValue().asArray().isEmpty()) {
          switch (arrayEncoding) {
            case ARRAY:
              colValue = Lists.newArrayList();
              break;

View on GitHub (pinned to 86d9c8fc54)