apache/beam · error · CoderException

Message is larger than the max size supported by the coder

Error message

Message is larger than the max size supported by the coder

What it means

AmqpMessageCoder.encode tries serializing the AMQP message at progressively smaller candidate sizes; if every attempt fails (message too large for the allowed max size), it throws CoderException('Message is larger than the max size supported by the coder'). The message cannot be encoded within the coder's size limit.

Solutions

1. Reduce message payload size — store large data externally and send a reference.
2. Increase the coder's max message size setting if the runner/broker allows larger messages.
3. Enable compression of the payload before publishing.
4. Split large logical records into multiple AMQP messages.

Example fix

// before
byte[] data = hugePayload; // > maxMessageSize
sender.send(message);
// after
byte[] data = compress(hugePayload);
if (data.length > maxMessageSize) { storeExternallyAndSendRef(data); }

if (payload.length > MAX_AMQP_MESSAGE_BYTES) {
  throw new IllegalArgumentException("payload too large: " + payload.length);
}

try {
  sender.send(message);
} catch (org.apache.beam.sdk.coders.CoderException e) {
  log.error("AMQP message exceeds coder max size; reduce payload", e);
}

When it happens

Trigger: Encoding an AMQP Message whose encoded body exceeds maxMessageSize; all internal encode attempts throw and the loop falls through to the CoderException.

Common situations: Publishing very large payloads (big blobs embedded in message bodies), broker limits lower than the message size, maxMessageSize misconfigured for the pipeline.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• bad decoding function
• bad encoding function
• cannot encode a null BitSet
• cannot encode a null byte[]
• cannot encode a null Count-min Sketch

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d835f3a5789430e4. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoder.java:50

  private static final int[] MESSAGE_SIZES =
      new int[] {8 * 1024, 64 * 1024, 1 * 1024 * 1024, 64 * 1024 * 1024};

  static AmqpMessageCoder of() {
    return new AmqpMessageCoder();
  }

  @Override
  public void encode(Message value, OutputStream outStream) throws CoderException, IOException {
    for (int maxMessageSize : MESSAGE_SIZES) {
      try {
        encode(value, outStream, maxMessageSize);
        return;
      } catch (Exception e) {
        continue;
      }
    }
    throw new CoderException("Message is larger than the max size supported by the coder");
  }

  private void encode(Message value, OutputStream outStream, int messageSize)
      throws IOException, BufferOverflowException {
    byte[] data = new byte[messageSize];
    int bytesWritten = value.encode(data, 0, data.length);
    VarInt.encode(bytesWritten, outStream);
    outStream.write(data, 0, bytesWritten);
  }

  @Override
  public Message decode(InputStream inStream) throws CoderException, IOException {
    Message message = Message.Factory.create();
    int bytesToRead = VarInt.decodeInt(inStream);
    byte[] encodedMessage = new byte[bytesToRead];
    ByteStreams.readFully(inStream, encodedMessage);
    message.decode(encodedMessage, 0, encodedMessage.length);
    return message;

View on GitHub (pinned to 12126d8942)