apache/beam · error · IllegalStateException

Field `{keyField}` should of type `VARBINARY`. Please change

Error message

Field `{keyField}` should of type `VARBINARY`. Please change the type or specify a field to write the KEY value from via TableProperties.

What it means

RowToEntity converts Beam Rows back into Datastore Entities. When a keyField is specified, the input schema's field with that name must be of Beam type BYTES, because the Entity KEY is taken from raw bytes. expand() throws this IllegalStateException when the field exists but is declared with any other type (VARBINARY in the message is a legacy label for non-BYTES).

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/RowToEntity.java:69

  private static final Logger LOG = LoggerFactory.getLogger(RowToEntity.class);

  private RowToEntity(Supplier<String> keySupplier, String kind, String keyField) {
    this.keySupplier = keySupplier;
    this.kind = kind;
    this.keyField = keyField;
  }

  @Override
  public PCollection<Entity> expand(PCollection<Row> input) {
    boolean isFieldPresent = input.getSchema().getFieldNames().contains(keyField);
    if (isFieldPresent) {
      if (!input
          .getSchema()
          .getField(keyField)
          .getType()
          .getTypeName()
          .equals(Schema.TypeName.BYTES)) {
        throw new IllegalStateException(
            "Field `"
                + keyField
                + "` should of type `VARBINARY`. Please change the type or specify a field to"
                + " write the KEY value from via TableProperties.");
      }
      LOG.info("Field to use as Entity KEY is set to: `{}`.", keyField);
    }
    return input.apply(ParDo.of(new RowToEntity.RowToEntityConverter(isFieldPresent)));
  }

  /**
   * Create a PTransform instance.
   *
   * @param keyField Row field containing a serialized {@code Key}, must be set when using user
   *     specified keys.
   * @param kind DataStore `Kind` data will be written to (required when generating random {@code
   *     Key}s).
   * @return {@code PTransform} instance for Row to Entity conversion.

View on GitHub (pinned to 12126d8942)

Solutions

Change the keyField column type to Schema.FieldType.BYTES in the input PCollection's schema
Or select a different BYTES-typed field as the keyField, or remove withKeyField so keys are not taken from the row
If the key is a string, encode it to bytes yourself (e.g. key.getBytes(UTF_8)) and store it in a BYTES field

Example fix

// before
Schema schema = Schema.of(Schema.Field.of("key", Schema.FieldType.STRING));
// after
Schema schema = Schema.of(Schema.Field.of("key", Schema.FieldType.BYTES));

Defensive patterns

Strategy: validation

Validate before calling

if (pc.getSchema().getFieldNames().contains(keyField)
    && !pc.getSchema().getField(keyField).getType().getTypeName().equals(Schema.TypeName.BYTES)) {
  throw new IllegalArgumentException("keyField `" + keyField + "` must be BYTES");
}

Type guard

boolean hasBytesKeyField(PCollection<Row> pc, String keyField) {
  return pc.getSchema().getFieldNames().contains(keyField)
      && pc.getSchema().getField(keyField).getType().getTypeName().equals(Schema.TypeName.BYTES);
}

Prevention

Keep the KEY field BYTES end-to-end, including upstream transform output schemas
If the key is textual, encode it to bytes before it enters the PCollection
Call pc.setSchema(...) with an explicit schema rather than relying on inferred field types

When it happens

Trigger: Applying DatastoreIO.v1().write().withKeyField(...).expand() (RowToEntity.expand) on a PCollection whose schema declares keyField as STRING/INT64/etc. rather than BYTES.

Common situations: Output schema from an upstream transform (e.g. BigQuery read) typed the key as STRING; changing keyField name without updating the schema; writing a row field where the user assumed the string key would be auto-encoded.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0b0aaa28f176c1b8. Report an issue: GitHub.