What causes "msg:{msg}" in apache/incubator-seata?

Reduce payload size per message (batch limits, smaller undo logs per branch) so a single slow peer cannot fill the send buffer.

apache/incubator-seata · critical · FrameworkException

0109

0109

Error message

msg:{msg}

What it means

channelWritableCheck blocks (on a lock + condition) while the Netty channel's send buffer is full; after NettyClientConfig.max-not-writeable-retry await cycles it destroys the channel and throws FrameworkException with error code ChannelIsNotWritable. It is backpressure: the peer is not reading fast enough, so Seata refuses to grow the outbound buffer unboundedly and drops the connection instead.

Solutions

Reduce payload size per message (batch limits, smaller undo logs per branch) so a single slow peer cannot fill the send buffer.
Address the slow reader: tune TC store throughput (db pool, disk), check peer GC/CPU throttling.
Increase netty write-buffer high-water mark / max-not-writeable-retry only as a buffer, not a fix; also verify client and TC bandwidth.
If recurrent, scale out TC nodes and rebalance transaction groups so no single channel carries the full burst.

Example fix

# before
transport.max-not-writeable-retry default; huge batch undo in one tx
# after
smaller branch batches +
transport.max-not-writeable-retry=64000 (buy time while fixing tc store)

Defensive patterns

Strategy: fallback

Validate before calling

// before enqueuing a burst, check channel health/backpressure
if (!channel.isActive() || !channel.isWritable()) {
    routeToHealthyChannelOrSpool(); // do not pile onto a saturated channel
}

Type guard

boolean canAcceptNow(Channel ch) { return ch.isActive() && ch.isWritable(); }

Try / catch

catch (FrameworkException e) {
    if (e.getCode() == FrameworkErrorCode.ChannelIsNotWritable) {
        // channel already destroyed: rebuild connection and replay idempotent messages with backoff
        reconnectAndReplayIdempotent();
    } else { throw e; }
}

Prevention

Keep per-transaction undo payload small; batch branch operations
Monitor channel writability and outbound queue depth as leading indicators
Provision TC store/CPU for peak commit bursts; scale TC horizontally
Never disable the writable check — fix the slow peer instead

When it happens

Trigger: Sending a large burst of RPC messages (batch undo logs, big branch registrations, merge-able requests) to a TC or client that stalls reading — GC pause on the peer, slow dbstore on TC, or a network path with bandwidth asymmetry; writable-buffer bytes high-water mark is exceeded and never recovers within the retry window.

Common situations: Large transactions generating multi-MB undo log payloads, TC db store slow (connection pool exhausted, lock contention), k8s network policy/traffic shaping throttling egress, or a misbehaving consumer holding the channel.

Related errors

AI-assisted analysis of apache/incubator-seata@e01f97c6db (2026-08-14). Data as JSON: /api/errors/c0cec47b3e474d2f. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/apache/seata/core/rpc/netty/AbstractNettyRemoting.java:390

        SocketAddress socketAddress = channel.remoteAddress();
        String address = socketAddress.toString();
        if (socketAddress.toString().indexOf(NettyClientConfig.getSocketAddressStartChar()) == 0) {
            address = socketAddress
                    .toString()
                    .substring(NettyClientConfig.getSocketAddressStartChar().length());
        }
        return address;
    }

    private void channelWritableCheck(Channel channel, Object msg) {
        int tryTimes = 0;
        writabilityLock.lock();
        try {
            while (!channel.isWritable()) {
                tryTimes++;
                if (tryTimes > NettyClientConfig.getMaxNotWriteableRetry()) {
                    destroyChannel(channel);
                    throw new FrameworkException(
                            "msg:" + ((msg == null) ? "null" : msg.toString()),
                            FrameworkErrorCode.ChannelIsNotWritable);
                }
                try {
                    writabilityCondition.await(NOT_WRITEABLE_CHECK_MILLS, TimeUnit.MILLISECONDS);
                } catch (InterruptedException exx) {
                    LOGGER.error(exx.getMessage());
                    Thread.currentThread().interrupt();
                    throw new FrameworkException(exx);
                }
            }
        } finally {
            writabilityLock.unlock();
        }
    }

    /**
     * Destroy channel.

View on GitHub (pinned to e01f97c6db)