{"record":{"id":"0b5d9e006b3e3a2d","repo":"apache/beam","slug":"pubsub-message-of-length-totalsize-exceeds-maximum-of","errorCode":null,"errorMessage":"Pubsub message of length {totalSize} exceeds maximum of {maxPublishBatchSize} bytes, when considering the payload and attributes. See https://cloud.google.com/pubsub/quotas#resource_limits","messagePattern":"Pubsub message of length (.+?) exceeds maximum of (.+?) bytes, when considering the payload and attributes\\. See https://cloud\\.google\\.com/pubsub/quotas#resource_limits","errorType":"validation","errorClass":"SizeLimitExceededException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java","lineNumber":139,"sourceCode":"\n        String value = attribute.getValue();\n        int valueSize = value.getBytes(StandardCharsets.UTF_8).length;\n        if (valueSize > PUBSUB_MESSAGE_ATTRIBUTE_MAX_VALUE_BYTES) {\n          throw new SizeLimitExceededException(\n              \"Pubsub message attribute value for key '\"\n                  + key\n                  + \"' starting with '\"\n                  + value.substring(0, Math.min(256, value.length()))\n                  + \"' exceeds the maximum of \"\n                  + PUBSUB_MESSAGE_ATTRIBUTE_MAX_VALUE_BYTES\n                  + \" bytes. See https://cloud.google.com/pubsub/quotas#resource_limits\");\n        }\n        totalSize += valueSize;\n      }\n    }\n\n    if (totalSize > maxPublishBatchSize) {\n      throw new SizeLimitExceededException(\n          \"Pubsub message of length \"\n              + totalSize\n              + \" exceeds maximum of \"\n              + maxPublishBatchSize\n              + \" bytes, when considering the payload and attributes. \"\n              + \"See https://cloud.google.com/pubsub/quotas#resource_limits\");\n    }\n    return totalSize;\n  }\n\n  PreparePubsubWriteDoFn(\n      SerializableFunction<ValueInSingleWindow<InputT>, PubsubMessage> formatFunction,\n      @Nullable SerializableFunction<ValueInSingleWindow<InputT>, PubsubIO.PubsubTopic>\n          topicFunction,\n      boolean usesOrderingKey,\n      int maxPublishBatchSize,\n      BadRecordRouter badRecordRouter,\n      Coder<InputT> inputCoder,","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java#L121-L157","documentation":"Before writing, PreparePubsubWriteDoFn sums the payload plus all attribute keys and values and rejects messages whose total size exceeds maxPublishBatchSize, enforcing the Pub/Sub per-message request size quota (10MB by default). This fails fast client-side instead of getting a rejected publish from the service.","triggerScenarios":"process() calls validatePubsubMessage(); computed totalSize (payload + attribute keys + attribute values) exceeds maxPublishBatchSize when writing to Pub/Sub via PubsubIO.","commonSituations":"Publishing very large files or serialized objects as a single message; unbounded accumulation of attributes; misconfigured or reduced maxPublishBatchSize while sending large batches.","solutions":["Reduce the message payload size (compress, chunk, or write large data to GCS and send a reference)","Trim attribute keys and values to the minimum needed","Enable Pub/Sub topic message storage / use a dead-letter style pattern with a reference pointer","Increase batching configuration only if within actual service quota, or raise maxPublishBatchSize explicitly to match the topic quota"],"exampleFix":"// before\nPCollection<String> big = ...; // 15MB strings\nbig.apply(PubsubIO.writeStrings().to(topic));\n// after\nPCollection<String> refs = big.apply(\"toGcs\", new WriteRefsToGcs());\nrefs.apply(PubsubIO.writeStrings().to(topic));","handlingStrategy":"validation","validationCode":"long total = payload.length + attributes.keySet().stream().mapToLong(k -> k.getBytes(StandardCharsets.UTF_8).length).sum()\n    + attributes.values().stream().mapToLong(v -> v.getBytes(StandardCharsets.UTF_8).length).sum();\nif (total > 10_000_000) throw new IllegalArgumentException(\"Message total \" + total + \" bytes exceeds Pub/Sub limit\");","typeGuard":null,"tryCatchPattern":"try { pipeline.run(); } catch (SizeLimitExceededException e) { log.error(\"Pub/Sub message too large: {}\", e.getMessage()); }","preventionTips":["Chunk or compress large payloads before publishing","Store big blobs in GCS and publish a reference","Keep the attribute set minimal","Test with production-sized messages before launch"],"tags":["java","pubsub","message-size","quota","payload"],"backgroundTag":"payload-too-large","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-15T02:17:10.978Z"}