apache/beam · error · IllegalArgumentException

Expecting exactly one field, found

Error message

Expecting exactly one field, found %s

What it means

In expand(), when the configured format is RAW, the input PCollection's schema must contain exactly one field, since RAW writes the single field's bytes verbatim to Kafka. The provider throws IllegalArgumentException stating how many fields were actually found.

Solutions

1. Use a different format (JSON/AVRO/PROTO) that supports multi-field schemas.
2. Map/combine your Row into a single BYTES field before the sink (e.g. serialize your payload yourself).
3. Verify the PCollection's schema via pc.getSchema() before applying the transform.

Example fix

// before
rows.apply("write", kafkaWrite.withFormat("RAW")); // schema has 3 fields
// after
rows.apply(MapElements.into(Schema.FieldType.BYTES).via(row -> serialize(row)))
    .setSchema(Schema.of(Schema.Field.of("payload", Schema.FieldType.BYTES)))
    .apply("write", kafkaWrite.withFormat("RAW"));

if (format.equals("RAW") && pc.getSchema().getFields().size() != 1) { throw new IllegalArgumentException("RAW format requires exactly one field"); }

When it happens

Trigger: Applying the Kafka write SchemaTransform with format RAW on a PCollection whose schema has 0 or 2+ fields.

Common situations: Pipelines emitting multi-field Rows (e.g. key+value) directly to a RAW-format Kafka sink; forgetting to pre-map the record to a single BYTES field.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• The input schema must have exactly one field of type byte.
• Expecting messageName to be non-null.
• Format is not supported. Supported formats are
• Cast isn't compatible using +validator()+: +reason
• : closing producer after unrecoverable error. The work…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9c023c504b96f53d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaWriteSchemaTransformProvider.java:203

          Schema errorSchema,
          boolean handleErrors) {
        super(name, toGenericRecordsFn, errorSchema, handleErrors, RECORD_OUTPUT_TAG);
      }
    }

    @SuppressWarnings({
      "nullness" // TODO(https://github.com/apache/beam/issues/20497)
    })
    @Override
    public PCollectionRowTuple expand(PCollectionRowTuple input) {
      Schema inputSchema = input.get("input").getSchema();
      org.apache.avro.Schema avroSchema = AvroUtils.toAvroSchema(inputSchema);
      final SerializableFunction<Row, byte[]> toBytesFn;
      SerializableFunction<Row, GenericRecord> toGenericRecordsFn = null;
      if (configuration.getFormat().equals("RAW")) {
        int numFields = inputSchema.getFields().size();
        if (numFields != 1) {
          throw new IllegalArgumentException("Expecting exactly one field, found " + numFields);
        }
        if (!inputSchema.getField(0).getType().equals(Schema.FieldType.BYTES)) {
          throw new IllegalArgumentException(
              "The input schema must have exactly one field of type byte.");
        }
        toBytesFn = getRowToRawBytesFunction(inputSchema.getField(0).getName());
      } else if (configuration.getFormat().equals("JSON")) {
        toBytesFn = JsonUtils.getRowToJsonBytesFunction(inputSchema);
      } else if (configuration.getFormat().equals("PROTO")) {
        String descriptorPath = configuration.getFileDescriptorPath();
        String schema = configuration.getSchema();
        String messageName = configuration.getMessageName();
        if (messageName == null) {
          throw new IllegalArgumentException("Expecting messageName to be non-null.");
        }
        if (descriptorPath != null && schema != null) {
          throw new IllegalArgumentException(
              "You must include a descriptorPath or a proto Schema but not both.");

View on GitHub (pinned to 12126d8942)