apache/beam · error · RuntimeException

Could not decode Pubsub message

Error message

Could not decode Pubsub message

What it means

PubsubMessages' PubsubMessage decode function parses incoming bytes as a com.google.pubsub.v1.PubsubMessage protobuf. If parseFrom fails with InvalidProtocolBufferException, the bytes are not a valid encoded PubsubMessage and a RuntimeException 'Could not decode Pubsub message' is thrown with the cause attached.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessages.java:114

  public static class ParsePayloadAsPubsubMessageProto
      implements SerializableFunction<PubsubMessage, byte[]> {
    @Override
    public byte[] apply(PubsubMessage input) {
      return toSerializedPubsubMessageProto(input);
    }
  }

  // Convert the serialized PubsubMessage proto to PubsubMessage.
  public static class ParsePubsubMessageProtoAsPayload
      implements SerializableFunction<byte[], PubsubMessage> {
    @Override
    public PubsubMessage apply(byte[] input) {
      try {
        com.google.pubsub.v1.PubsubMessage message =
            com.google.pubsub.v1.PubsubMessage.parseFrom(input);
        return fromProto(message);
      } catch (InvalidProtocolBufferException e) {
        throw new RuntimeException("Could not decode Pubsub message", e);
      }
    }
  }

  public static class DeserializeBytesIntoPubsubMessagePayloadOnly
      implements SerializableFunction<byte[], PubsubMessage> {

    @Override
    public PubsubMessage apply(byte[] value) {
      return new PubsubMessage(value, ImmutableMap.of());
    }
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Use the correct deserializer for your input: if the bytes are just the message payload, use DeserializeBytesIntoPubsubMessagePayloadOnly; only full proto-encoded messages work with the default one.
2. Verify the producer: Beam's read path expects protobuf-encoded PubsubMessage for this function; align publisher serialization accordingly.
3. Wrap the decode in a DLQ/fallback path to inspect offending bytes instead of crashing the pipeline.

Example fix

// before
SerializableFunction<byte[], PubsubMessage> f = new PubsubMessages.DeserializeBytesIntoPubsubMessage();
f.apply(rawJsonBytes); // throws
// after
SerializableFunction<byte[], PubsubMessage> f = new PubsubMessages.DeserializeBytesIntoPubsubMessagePayloadOnly(); // or publish protobuf-encoded messages

// quick sniff before decode
boolean looksLikeProto = input != null && input.length > 0; // full validation requires parse attempt
try { com.google.pubsub.v1.PubsubMessage.parseFrom(input); } catch (Exception e) { /* not a full message proto */ }

try {
  PubsubMessage m = deserializer.apply(bytes);
} catch (RuntimeException e) {
  deadLetter(bytes, e.getCause()); // InvalidProtocolBufferException is the cause
}

When it happens

Trigger: Feeding a PubsubIO message-deserialization function byte arrays that are not protobuf-encoded PubsubMessage protos — e.g. raw JSON, Avro, or plain text payloads decoded with the wrong deserializer.

Common situations: Mixing up payload-only vs full-message deserializers (DeserializeBytesIntoPubsubMessage vs payload-only variants), reading from a topic published by non-Beam producers, or corrupted/mangled messages.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• Could not parse Pub/Sub message
• Could not decode bytes as message
• Unexpected type_info: +protoFieldType.getTypeInfoCase()
• Unable to infer data schema from configuration proto.
• Could not decode Pubsub message

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/756bac5da25bd8f9. Report an issue: GitHub.