apache/beam · warning

Ignoring unmatched input fields: {}

Error message

Ignoring unmatched input fields: {}

What it means

SqsIO's SchemaEntryMapper maps input elements to SQS SendMessageBatchRequestEntry fields via a Beam schema. Input fields that don't match any target schema field are not errors; they are collected as 'ignored' and logged with this warning so users know those fields are silently dropped from each message.

Source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sqs/SqsIO.java:537

            // make sure field types match
            if (!sourceField.typesEqual(targetSchema.getField(targetIdx))) {
              invalid.add(sourceField.getName());
            }
            fieldMapping[targetIdx] = i;
          } else {
            ignored.add(sourceField.getName());
          }
        }
        checkState(
            ignored.size() < sourceSchema.getFieldCount(),
            "No fields matched, expected %s but got %s",
            schema.getFieldNames(),
            ignored);

        checkState(invalid.isEmpty(), "Detected incompatible types for input fields: {}", invalid);

        if (!ignored.isEmpty()) {
          LOG.warn("Ignoring unmatched input fields: {}", ignored);
        }
      }

      @Override
      public SendMessageBatchRequestEntry apply(String entryId, T input) {
        Row row = toRow.apply(input);
        Object[] values = new Object[fieldMapping.length];
        values[0] = entryId;
        for (int i = 0; i < values.length; i++) {
          if (fieldMapping[i] >= 0) {
            values[i] = row.getValue(fieldMapping[i]);
          }
        }
        return fromRow.apply(Row.withSchema(schema).attachValues(values));
      }
    }

    /** Result of {@link #writeBatches}. */

View on GitHub (pinned to 12126d8942)

Solutions

1. Add mappings for the extra fields if their values must reach SQS
2. Or narrow the input schema to only fields the mapper consumes (e.g. via select()/Row projection)
3. Ignore safely if the fields are intentionally unused — the message is informational
4. Check the companion 'Detected incompatible types' error if fields are matched but mistyped

Example fix

// before: input has fields 'payload,ts,extra' but only payload,ts mapped
// after: project out the unmapped field
PCollection<Row> mapped = input.getSchema().match();
mapped = input.apply(Select.fieldNames("payload", "ts"));

// ensure input schema fields are all consumed by the mapper
Set<String> mappedFields = mapperConfig.consumedFieldNames();
Set<String> inputFields = new HashSet<>(input.getSchema().getFieldNames());
Set<String> unmatched = new HashSet<>(inputFields);
unmatched.removeAll(mappedFields);
if (!unmatched.isEmpty()) throw new IllegalStateException("Unmapped fields: " + unmatched);

When it happens

Trigger: Using SqsIO.write() with .withEntryMapper/schema mapping on a PCollection whose schema contains fields not referenced by any SQS entry attribute/body mapping — e.g. a Row with extra columns after an upstream schema evolution.

Common situations: Upstream producers added new fields; reusing a mapper config across pipeline versions; accidental extra columns from a JOIN in SQL-generated PCollections.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• SqlTransform can only be applied to schema'd transforms. Ple
• Cannot provide a coder for a Beam Row. Please provide a sche
• Unable to generate coder for schema {schema}
• Expecting exactly one field, found
• The input schema must have exactly one field of type byte.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b1f460a97b942841. Report an issue: GitHub.