apache/beam · error · SizeLimitExceededException

Pubsub message attribute key '{key}' exceeds the maximum of

Error message

Pubsub message attribute key '{key}' exceeds the maximum of {PUBSUB_MESSAGE_ATTRIBUTE_MAX_KEY_BYTES} bytes. See https://cloud.google.com/pubsub/quotas#resource_limits

What it means

Pub/Sub limits each attribute key length; validatePubsubMessage throws SizeLimitExceededException when a single attribute key's UTF-8 byte length exceeds PUBSUB_MESSAGE_ATTRIBUTE_MAX_KEY_BYTES. Enforced client-side before publish since the Pub/Sub API would reject it.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java:113

    if (attributes != null) {
      if (attributes.size() > PUBSUB_MESSAGE_MAX_ATTRIBUTES) {
        throw new SizeLimitExceededException(
            "Pubsub message contains "
                + attributes.size()
                + " attributes which exceeds the maximum of "
                + PUBSUB_MESSAGE_MAX_ATTRIBUTES
                + ". See https://cloud.google.com/pubsub/quotas#resource_limits");
      }

      // Consider attribute encoding overhead, so it doesn't go over the request limits
      totalSize += attributes.size() * PUBSUB_MESSAGE_ATTRIBUTE_ENCODE_ADDITIONAL_BYTES;

      for (Map.Entry<String, String> attribute : attributes.entrySet()) {
        String key = attribute.getKey();
        int keySize = key.getBytes(StandardCharsets.UTF_8).length;
        if (keySize > PUBSUB_MESSAGE_ATTRIBUTE_MAX_KEY_BYTES) {
          throw new SizeLimitExceededException(
              "Pubsub message attribute key '"
                  + key
                  + "' exceeds the maximum of "
                  + PUBSUB_MESSAGE_ATTRIBUTE_MAX_KEY_BYTES
                  + " bytes. See https://cloud.google.com/pubsub/quotas#resource_limits");
        }
        totalSize += keySize;

        String value = attribute.getValue();
        int valueSize = value.getBytes(StandardCharsets.UTF_8).length;
        if (valueSize > PUBSUB_MESSAGE_ATTRIBUTE_MAX_VALUE_BYTES) {
          throw new SizeLimitExceededException(
              "Pubsub message attribute value for key '"
                  + key
                  + "' starting with '"
                  + value.substring(0, Math.min(256, value.length()))
                  + "' exceeds the maximum of "
                  + PUBSUB_MESSAGE_ATTRIBUTE_MAX_VALUE_BYTES

View on GitHub (pinned to 12126d8942)

Solutions

1. Shorten the offending attribute key (use a short canonical name like 'src' instead of the full path)
2. Sanitize/normalize incoming keys to a max byte length before building the message
3. Move long key information into the payload or a shorter key's value
4. Validate each key's UTF-8 length at construction time

Example fix

// before
attrs.put("/very/long/source/system/path/from/external/integration/...", value);
// after
attrs.put("source-path", value); // key well under PUBSUB_MESSAGE_ATTRIBUTE_MAX_KEY_BYTES

java
for (Map.Entry<String, String> a : attrs.entrySet()) {
  if (a.getKey().getBytes(StandardCharsets.UTF_8).length > 256) {
    throw new IllegalArgumentException("Attribute key too long: " + a.getKey());
  }
}

java
try {
  validatePubsubMessage(msg, maxBatchSize);
} catch (SizeLimitExceededException e) {
  if (e.getMessage().contains("attribute key")) {
    msg = new PubsubMessage(msg.getPayload(), shortenKeys(msg.getAttributeMap()));
  } else throw e;
}

When it happens

Trigger: Publishing a PubsubMessage whose attribute map contains a key whose UTF-8 byte length exceeds PUBSUB_MESSAGE_ATTRIBUTE_MAX_KEY_BYTES (256) via PreparePubsubWriteDoFn.process.

Common situations: Using long header names, full file paths, or concatenated key strings as attribute keys; non-ASCII keys inflating byte count; forwarding arbitrary external metadata keys unfiltered.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• Pubsub message contains {attributes.size()} attributes which
• Pubsub message attribute value for key '{key}' starting with
• Pubsub message data field of length {payloadSize} exceeds ma
• Pubsub message ordering key of length {orderingKeySize} exce
• Pubsub message of length {totalSize} exceeds maximum of {max

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/48f90bbdc85ddd62. Report an issue: GitHub.