apache/beam · error · SizeLimitExceededException

Pubsub message attribute value for key '{key}' starting with

Error message

Pubsub message attribute value for key '{key}' starting with '{value.substring(0, Math.min(256, value.length()))}' exceeds the maximum of {PUBSUB_MESSAGE_ATTRIBUTE_MAX_VALUE_BYTES} bytes. See https://cloud.google.com/pubsub/quotas#resource_limits

What it means

PreparePubsubWriteDoFn validates each Pub/Sub message attribute value before publishing. Google Cloud Pub/Sub limits a single attribute value to 1024 bytes (PUBSUB_MESSAGE_ATTRIBUTE_MAX_VALUE_BYTES); when the UTF-8 encoded value exceeds this, a SizeLimitExceededException is thrown with a 256-char preview of the offending value so the developer can locate it.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java:125

      totalSize += attributes.size() * PUBSUB_MESSAGE_ATTRIBUTE_ENCODE_ADDITIONAL_BYTES;

      for (Map.Entry<String, String> attribute : attributes.entrySet()) {
        String key = attribute.getKey();
        int keySize = key.getBytes(StandardCharsets.UTF_8).length;
        if (keySize > PUBSUB_MESSAGE_ATTRIBUTE_MAX_KEY_BYTES) {
          throw new SizeLimitExceededException(
              "Pubsub message attribute key '"
                  + key
                  + "' exceeds the maximum of "
                  + PUBSUB_MESSAGE_ATTRIBUTE_MAX_KEY_BYTES
                  + " bytes. See https://cloud.google.com/pubsub/quotas#resource_limits");
        }
        totalSize += keySize;

        String value = attribute.getValue();
        int valueSize = value.getBytes(StandardCharsets.UTF_8).length;
        if (valueSize > PUBSUB_MESSAGE_ATTRIBUTE_MAX_VALUE_BYTES) {
          throw new SizeLimitExceededException(
              "Pubsub message attribute value for key '"
                  + key
                  + "' starting with '"
                  + value.substring(0, Math.min(256, value.length()))
                  + "' exceeds the maximum of "
                  + PUBSUB_MESSAGE_ATTRIBUTE_MAX_VALUE_BYTES
                  + " bytes. See https://cloud.google.com/pubsub/quotas#resource_limits");
        }
        totalSize += valueSize;
      }
    }

    if (totalSize > maxPublishBatchSize) {
      throw new SizeLimitExceededException(
          "Pubsub message of length "
              + totalSize
              + " exceeds maximum of "
              + maxPublishBatchSize

View on GitHub (pinned to 12126d8942)

Solutions

1. Move the large data into the message payload and keep only small metadata in attributes
2. Truncate or hash the attribute value before publishing (e.g. SHA-256 digest of a large field)
3. Split the value across multiple attributes if it must be carried as attributes
4. Shorten the value by removing redundant data such as query parameters or whitespace

Example fix

// before
message.putAttributes("payload", bigJsonString);
// after
message.putAttributes("payloadHash", sha256Hex(bigJsonString));

for (Map.Entry<String,String> e : attributes.entrySet()) {
  if (e.getValue().getBytes(java.nio.charset.StandardCharsets.UTF_8).length > 1024) {
    throw new IllegalArgumentException("Attribute '" + e.getKey() + "' value exceeds 1024 bytes");
  }
}

boolean validAttrValue(String v) { return v != null && v.getBytes(java.nio.charset.StandardCharsets.UTF_8).length <= 1024; }

try { pipeline.run(); } catch (SizeLimitExceededException e) { log.error("Pub/Sub attribute too large: {}", e.getMessage()); }

When it happens

Trigger: process() calls validatePubsubMessage(); the byte length of any attribute value (value.getBytes(StandardCharsets.UTF_8).length) exceeds PUBSUB_MESSAGE_ATTRIBUTE_MAX_VALUE_BYTES during a PubsubIO.write with attributes.

Common situations: Putting large payloads (JSON blobs, stack traces, URLs with query strings) into message attributes instead of the message body; multi-byte UTF-8 content pushing a seemingly short string over 1024 bytes.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• Pubsub message contains {attributes.size()} attributes which
• Pubsub message attribute key '{key}' exceeds the maximum of
• Pubsub message of length {totalSize} exceeds maximum of {max
• Pubsub message data field of length {payloadSize} exceeds ma
• Pubsub message ordering key of length {orderingKeySize} exce

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d9fe255752435186. Report an issue: GitHub.