apache/beam · error · SizeLimitExceededException

Pubsub message contains {attributes.size()} attributes which

Error message

Pubsub message contains {attributes.size()} attributes which exceeds the maximum of {PUBSUB_MESSAGE_MAX_ATTRIBUTES}. See https://cloud.google.com/pubsub/quotas#resource_limits

What it means

Pub/Sub limits a message to at most PUBSUB_MESSAGE_MAX_ATTRIBUTES attribute pairs; validatePubsubMessage throws SizeLimitExceededException when the attribute map exceeds this count. This enforces the documented quota before the publish request is sent.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java:98

        throw new SizeLimitExceededException(
            "Pubsub message ordering key of length "
                + orderingKeySize
                + " exceeds maximum of "
                + ORDERING_KEY_MAX_BYTE_SIZE
                + " bytes. See https://cloud.google.com/pubsub/quotas#resource_limits");
      }
      totalSize += orderingKeySize;
    }

    final @Nullable Map<String, String> attributes = message.getAttributeMap();
    if (payloadSize == 0 && (attributes == null || attributes.isEmpty())) {
      throw new IllegalArgumentException(
          "Pubsub message must contain a non-empty payload or at least one attribute.");
    }

    if (attributes != null) {
      if (attributes.size() > PUBSUB_MESSAGE_MAX_ATTRIBUTES) {
        throw new SizeLimitExceededException(
            "Pubsub message contains "
                + attributes.size()
                + " attributes which exceeds the maximum of "
                + PUBSUB_MESSAGE_MAX_ATTRIBUTES
                + ". See https://cloud.google.com/pubsub/quotas#resource_limits");
      }

      // Consider attribute encoding overhead, so it doesn't go over the request limits
      totalSize += attributes.size() * PUBSUB_MESSAGE_ATTRIBUTE_ENCODE_ADDITIONAL_BYTES;

      for (Map.Entry<String, String> attribute : attributes.entrySet()) {
        String key = attribute.getKey();
        int keySize = key.getBytes(StandardCharsets.UTF_8).length;
        if (keySize > PUBSUB_MESSAGE_ATTRIBUTE_MAX_KEY_BYTES) {
          throw new SizeLimitExceededException(
              "Pubsub message attribute key '"
                  + key
                  + "' exceeds the maximum of "

View on GitHub (pinned to 12126d8942)

Solutions

1. Move bulk metadata into the message payload (e.g. JSON body) and keep only routing attributes
2. Filter attributes to the required subset before publishing
3. Aggregate overflow metadata as a single serialized attribute or GCS reference
4. Validate attribute count at construction time

Example fix

// before
PubsubMessage msg = new PubsubMessage(payload, allRecordFieldsAsAttributes); // 200 attrs
// after
Map<String, String> attrs = ImmutableMap.of("type", "record", "id", recordId);
PubsubMessage msg = new PubsubMessage(toJson(record), attrs);

java
if (attrs != null && attrs.size() > 100) {
  throw new IllegalArgumentException("Too many attributes: " + attrs.size());
}

java
try {
  validatePubsubMessage(msg, maxBatchSize);
} catch (SizeLimitExceededException e) {
  msg = new PubsubMessage(toJson(msg.getAttributeMap()).getBytes(UTF_8),
      ImmutableMap.of("type", "overflow"));
}

When it happens

Trigger: Publishing a PubsubMessage with more than PUBSUB_MESSAGE_MAX_ATTRIBUTES entries in its attribute map through PreparePubsubWriteDoFn.process — often when entire record fields are dumped into attributes.

Common situations: Mapping every column of a wide table row to message attributes; forwarding upstream HTTP headers as attributes; fan-out logic that accumulates attributes per hop.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• Pubsub message attribute key '{key}' exceeds the maximum of
• Pubsub message attribute value for key '{key}' starting with
• Pubsub message data field of length {payloadSize} exceeds ma
• Pubsub message ordering key of length {orderingKeySize} exce
• Pubsub message of length {totalSize} exceeds maximum of {max

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6586517d1fe2429f. Report an issue: GitHub.