apache/beam · error · IllegalStateException

Failed to parse DataStore key from bytes.

Error message

Failed to parse DataStore key from bytes.

What it means

DatastoreV1 IO's RowToEntity can build a Datastore Key from a Beam Row either from kind/name fields or from a raw protobuf-serialized Key stored in a byte[] column. When the byte[] column is used, the bytes are parsed with Key.parseFrom; if the bytes are not a valid serialized com.google.datastore.v1.Key protobuf, an IllegalStateException with this message is thrown, aborting the element.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/RowToEntity.java:154

    }

    /**
     * Create a random key for a {@code Row} without a keyField or use a user-specified key by
     * parsing it from byte array when keyField is set.
     *
     * @param row {@code Row} to construct a key for.
     * @return resulting {@code Key}.
     */
    private com.google.datastore.v1.Key constructKeyFromRow(Row row) {
      if (!useNonRandomKey) {
        // When key field is not present - use key supplier to generate a random one.
        return makeKey(kind, keySupplier.get()).build();
      }
      byte[] keyBytes = row.getBytes(keyField);
      try {
        return com.google.datastore.v1.Key.parseFrom(keyBytes);
      } catch (InvalidProtocolBufferException e) {
        throw new IllegalStateException("Failed to parse DataStore key from bytes.");
      }
    }

    /**
     * Converts a {@code Row} value to an appropriate DataStore {@code Value} object.
     *
     * @param value {@code Row} value to convert.
     * @return resulting {@code Value}.
     * @throws IllegalStateException when no mapping function for object of given type exists.
     */
    private Value mapObjectToValue(Object value) {
      if (value == null) {
        return Value.newBuilder().build();
      }

      if (Boolean.class.equals(value.getClass())) {
        return makeValue((Boolean) value).build();
      } else if (Byte.class.equals(value.getClass())) {

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the key byte[] column contains exactly com.google.datastore.v1.Key.toByteString() output from a datastore client, not a raw string or another protobuf type
2. In the upstream pipeline, serialize keys with makeKey(...).build().toByteString().toByteArray() before writing the row
3. Check that no encoding/compression step between writer and reader corrupted the bytes (e.g. re-encoding as UTF-8 string)
4. As a workaround, use the kind + key-field name mode of RowToEntity instead of raw key bytes

Example fix

// before
row.getString("id") // stored UTF-8 string in byte[] key column -> parseFrom fails
// after
byte[] keyBytes = com.google.datastore.v1.Key.newBuilder()
    .addPartitionId(...).build().toByteString().toByteArray(); // write valid Key protobuf bytes

if (keyBytes == null || keyBytes.length == 0) throw new IllegalArgumentException("empty key bytes");
try { com.google.datastore.v1.Key.parseFrom(keyBytes); } catch (InvalidProtocolBufferException e) { throw new IllegalArgumentException("column is not a datastore Key protobuf"); }

boolean isValidDatastoreKey(byte[] b) {
  if (b == null || b.length == 0) return false;
  try { com.google.datastore.v1.Key.parseFrom(b); return true; }
  catch (InvalidProtocolBufferException e) { return false; }
}

try {
  Key key = com.google.datastore.v1.Key.parseFrom(row.getBytes(keyField));
} catch (InvalidProtocolBufferException | IllegalStateException e) {
  LOG.error("Invalid key bytes in row", e); // route to dead-letter / skip
}

When it happens

Trigger: The schema field designated as the key field contains bytes that are not a valid datastore.v1.Key protobuf message — e.g. the column holds a plain string key, a differently-serialized key, truncated bytes, or a key serialized with a different protobuf type/version.

Common situations: Migrating from another sink where the key column was written as a UTF-8 string rather than Key.toByteString(); pipeline version mismatch where the writer serialized keys differently than the reader expects; corrupted or hand-crafted test rows.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Could not encode message as bytes
• Failed to convert PipelineOptions to JSON
• Unexpected error while serializing PubsubMessage to a byte a
• Failed to print type: {row}
• Failed to print mod: {row}

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/54ceb36363be11ef. Report an issue: GitHub.