apache/incubator-seata · critical · FrameworkException

0109

0109

Error message

msg:{msg}

What it means

channelWritableCheck blocks (on a lock + condition) while the Netty channel's send buffer is full; after NettyClientConfig.max-not-writeable-retry await cycles it destroys the channel and throws FrameworkException with error code ChannelIsNotWritable. It is backpressure: the peer is not reading fast enough, so Seata refuses to grow the outbound buffer unboundedly and drops the connection instead.

Source

Thrown at core/src/main/java/org/apache/seata/core/rpc/netty/AbstractNettyRemoting.java:390

        SocketAddress socketAddress = channel.remoteAddress();
        String address = socketAddress.toString();
        if (socketAddress.toString().indexOf(NettyClientConfig.getSocketAddressStartChar()) == 0) {
            address = socketAddress
                    .toString()
                    .substring(NettyClientConfig.getSocketAddressStartChar().length());
        }
        return address;
    }

    private void channelWritableCheck(Channel channel, Object msg) {
        int tryTimes = 0;
        writabilityLock.lock();
        try {
            while (!channel.isWritable()) {
                tryTimes++;
                if (tryTimes > NettyClientConfig.getMaxNotWriteableRetry()) {
                    destroyChannel(channel);
                    throw new FrameworkException(
                            "msg:" + ((msg == null) ? "null" : msg.toString()),
                            FrameworkErrorCode.ChannelIsNotWritable);
                }
                try {
                    writabilityCondition.await(NOT_WRITEABLE_CHECK_MILLS, TimeUnit.MILLISECONDS);
                } catch (InterruptedException exx) {
                    LOGGER.error(exx.getMessage());
                    Thread.currentThread().interrupt();
                    throw new FrameworkException(exx);
                }
            }
        } finally {
            writabilityLock.unlock();
        }
    }

    /**
     * Destroy channel.

View on GitHub (pinned to e01f97c6db)

Solutions

1. Reduce payload size per message (batch limits, smaller undo logs per branch) so a single slow peer cannot fill the send buffer.
2. Address the slow reader: tune TC store throughput (db pool, disk), check peer GC/CPU throttling.
3. Increase netty write-buffer high-water mark / max-not-writeable-retry only as a buffer, not a fix; also verify client and TC bandwidth.
4. If recurrent, scale out TC nodes and rebalance transaction groups so no single channel carries the full burst.

Example fix

# before
transport.max-not-writeable-retry default; huge batch undo in one tx
# after
smaller branch batches +
transport.max-not-writeable-retry=64000 (buy time while fixing tc store)

// before enqueuing a burst, check channel health/backpressure
if (!channel.isActive() || !channel.isWritable()) {
    routeToHealthyChannelOrSpool(); // do not pile onto a saturated channel
}

boolean canAcceptNow(Channel ch) { return ch.isActive() && ch.isWritable(); }

catch (FrameworkException e) {
    if (e.getCode() == FrameworkErrorCode.ChannelIsNotWritable) {
        // channel already destroyed: rebuild connection and replay idempotent messages with backoff
        reconnectAndReplayIdempotent();
    } else { throw e; }
}

When it happens

Trigger: Sending a large burst of RPC messages (batch undo logs, big branch registrations, merge-able requests) to a TC or client that stalls reading — GC pause on the peer, slow dbstore on TC, or a network path with bandwidth asymmetry; writable-buffer bytes high-water mark is exceeded and never recovers within the retry window.

Common situations: Large transactions generating multi-MB undo log payloads, TC db store slow (connection pool exhausted, lock contention), k8s network policy/traffic shaping throttling egress, or a misbehaving consumer holding the channel.

Related errors

• 0109
• connect cancelled, can not connect to services-server.
• Server start failed, the listen port: %s
• ip and port string cannot be empty!
• "Invalid endpoint format: " + address + ". Endpoint should b

AI-assisted analysis of apache/incubator-seata@e01f97c6db (2026-08-14). Data as JSON: /api/errors/c0cec47b3e474d2f. Report an issue: GitHub.