alibaba/canal · error · IOException

socket read timeout occured ! readSize = {}, readableBytes =

Error message

socket read timeout occured ! readSize = {}, readableBytes = {}, timeout = {}

What it means

Thrown by NettySocketChannel.read(int readSize, int timeout) when the netty read cache has fewer bytes than requested (readSize > cache.readableBytes()) AND the cumulative wait time (accumulated in 10ms WAIT_PERIOD ticks) exceeds the configured timeout. The message reports the requested size, currently buffered bytes, and the timeout so you can tell a slow producer from a dead one. Canal polls the in-memory ByteBuf cache rather than blocking on the socket directly, so this fires when MySQL stops streaming binlog events for longer than the configured read timeout.

Source

Thrown at driver/src/main/java/com/alibaba/otter/canal/parse/driver/mysql/socket/NettySocketChannel.java:185

        // 若读取内容较长，则自动扩充超时时间，以初始缓存大小为基准计算倍数
        if (timeout > 0 && readSize > DEFAULT_INIT_BUFFER_SIZE) {
            timeout *= (readSize / DEFAULT_INIT_BUFFER_SIZE + 1);
        }
        do {
            if (readSize > cache.readableBytes()) {
                if (null == channel) {
                    throw new IOException("socket has Interrupted !");
                }

                if (timeout > 0) {
                    accumulatedWaitTime += WAIT_PERIOD;
                    if (accumulatedWaitTime > timeout) {
                        StringBuilder sb = new StringBuilder("socket read timeout occured !");
                        sb.append(" readSize = ").append(readSize);
                        sb.append(", readableBytes = ").append(cache.readableBytes());
                        sb.append(", timeout = ").append(timeout);
                        throw new IOException(sb.toString());
                    }
                }

                synchronized (this) {
                    try {
                        wait(WAIT_PERIOD);
                    } catch (InterruptedException e) {
                        throw new IOException("socket has Interrupted !");
                    }
                }
            } else {
                byte[] back = new byte[readSize];
                synchronized (lock) {
                    cache.readBytes(back);
                }
                return back;
            }
        } while (true);

View on GitHub (pinned to 87be50e876)

Solutions

1. Enable canal.instance.detecting.enable=true with a detecting SQL (e.g. SELECT 1) and a short detectingIntervalInSeconds so MySQL keeps the connection alive during idle periods.
2. Increase the read timeout (canal.instance.network.readTimeout / the timeout arg passed to read()) to exceed expected idle windows.
3. Check network path between canal server and MySQL: firewall idle timeouts, NAT reaping, TLS middleboxes — verify with a long-lived tcpdump or mysql client session.
4. Confirm the MySQL user has REPLICATION SLAVE/CLIENT privileges and the binlog dump thread is actually running (SHOW PROCESSLIST) — a silently killed dump thread produces exactly this stall.
5. If using a load balancer / RDS proxy in front of MySQL, raise its idle timeout above the canal read timeout.

Example fix

# before (instance.properties)
canal.instance.network.readTimeout = 30000
canal.instance.detecting.enable = false

# after
canal.instance.network.readTimeout = 90000
canal.instance.detecting.enable = true
canal.instance.detecting.sql = SELECT 1
canal.instance.detecting.intervalInSeconds = 3

// Before reading, sanity-check that the channel is alive and that a heartbeat is configured
if (!socketChannel.isConnected()) {
    throw new ConnectException("channel not connected before read");
}
// Ensure canal.instance.detecting.enable=true + detecting.sql are set in instance.properties
// so MySQL keeps the link alive during idle periods (prevents the stall that causes the timeout).

try {
    byte[] data = socketChannel.read(readSize, readTimeoutMs);
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().startsWith("socket read timeout occured")) {
        // transient: log readSize/readableBytes/timeout from the message and let the
        // parser-level reconnect/retry logic kick in (HeartBeatHAController + MysqlEventParser retry)
        log.warn("canal read timeout, will retry: {}", e.getMessage());
        throw e; // surfaced & retried upstream
    }
    throw e;
}

When it happens

Trigger: Calling SocketChannel.read(readSize, timeout) with a positive timeout while the upstream MySQL/replication source stops sending data. The loop at NettySocketChannel.java:172-203 waits WAIT_PERIOD=10ms per iteration, adds to accumulatedWaitTime, and throws once it exceeds `timeout`. The timeout is auto-scaled (line 169-171) when readSize > 1MB.

Common situations: MySQL primary is idle (no writes) and no heartbeat/detecting SQL is configured; network partition or firewall dropping the replication connection silently; MySQL binlog dump thread killed server-side; the canal.instance.network.readTimeout property set too low for the workload; replica lag causing the dump thread to stall.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Received error packet: errno = {}, sqlstate = {} errmsg = {}
• Timeout occurred, failed to read total {} bytes in {} millis
• can't create socket!
• Start RabbitMQ producer error
• Unexpected response {} while fetching binlog: packet #{}, le

AI-assisted analysis of alibaba/canal@87be50e876 (2026-08-14). Data as JSON: /api/errors/896270c706707773. Report an issue: GitHub.