{"record":{"id":"896270c706707773","repo":"alibaba/canal","slug":"socket-read-timeout-occured-readsize-reada","errorCode":null,"errorMessage":"socket read timeout occured ! readSize = {}, readableBytes = {}, timeout = {}","messagePattern":"socket read timeout occured ! readSize = (.+?), readableBytes = (.+?), timeout = (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"driver/src/main/java/com/alibaba/otter/canal/parse/driver/mysql/socket/NettySocketChannel.java","lineNumber":185,"sourceCode":"\r\n        // 若读取内容较长，则自动扩充超时时间，以初始缓存大小为基准计算倍数\r\n        if (timeout > 0 && readSize > DEFAULT_INIT_BUFFER_SIZE) {\r\n            timeout *= (readSize / DEFAULT_INIT_BUFFER_SIZE + 1);\r\n        }\r\n        do {\r\n            if (readSize > cache.readableBytes()) {\r\n                if (null == channel) {\r\n                    throw new IOException(\"socket has Interrupted !\");\r\n                }\r\n\r\n                if (timeout > 0) {\r\n                    accumulatedWaitTime += WAIT_PERIOD;\r\n                    if (accumulatedWaitTime > timeout) {\r\n                        StringBuilder sb = new StringBuilder(\"socket read timeout occured !\");\r\n                        sb.append(\" readSize = \").append(readSize);\r\n                        sb.append(\", readableBytes = \").append(cache.readableBytes());\r\n                        sb.append(\", timeout = \").append(timeout);\r\n                        throw new IOException(sb.toString());\r\n                    }\r\n                }\r\n\r\n                synchronized (this) {\r\n                    try {\r\n                        wait(WAIT_PERIOD);\r\n                    } catch (InterruptedException e) {\r\n                        throw new IOException(\"socket has Interrupted !\");\r\n                    }\r\n                }\r\n            } else {\r\n                byte[] back = new byte[readSize];\r\n                synchronized (lock) {\r\n                    cache.readBytes(back);\r\n                }\r\n                return back;\r\n            }\r\n        } while (true);\r","sourceCodeStart":167,"sourceCodeEnd":203,"githubUrl":"https://github.com/alibaba/canal/blob/87be50e87686a3e8af08c368d0e1ffd1f59eb04a/driver/src/main/java/com/alibaba/otter/canal/parse/driver/mysql/socket/NettySocketChannel.java#L167-L203","documentation":"Thrown by NettySocketChannel.read(int readSize, int timeout) when the netty read cache has fewer bytes than requested (readSize > cache.readableBytes()) AND the cumulative wait time (accumulated in 10ms WAIT_PERIOD ticks) exceeds the configured timeout. The message reports the requested size, currently buffered bytes, and the timeout so you can tell a slow producer from a dead one. Canal polls the in-memory ByteBuf cache rather than blocking on the socket directly, so this fires when MySQL stops streaming binlog events for longer than the configured read timeout.","triggerScenarios":"Calling SocketChannel.read(readSize, timeout) with a positive timeout while the upstream MySQL/replication source stops sending data. The loop at NettySocketChannel.java:172-203 waits WAIT_PERIOD=10ms per iteration, adds to accumulatedWaitTime, and throws once it exceeds `timeout`. The timeout is auto-scaled (line 169-171) when readSize > 1MB.","commonSituations":"MySQL primary is idle (no writes) and no heartbeat/detecting SQL is configured; network partition or firewall dropping the replication connection silently; MySQL binlog dump thread killed server-side; the canal.instance.network.readTimeout property set too low for the workload; replica lag causing the dump thread to stall.","solutions":["Enable canal.instance.detecting.enable=true with a detecting SQL (e.g. SELECT 1) and a short detectingIntervalInSeconds so MySQL keeps the connection alive during idle periods.","Increase the read timeout (canal.instance.network.readTimeout / the timeout arg passed to read()) to exceed expected idle windows.","Check network path between canal server and MySQL: firewall idle timeouts, NAT reaping, TLS middleboxes — verify with a long-lived tcpdump or mysql client session.","Confirm the MySQL user has REPLICATION SLAVE/CLIENT privileges and the binlog dump thread is actually running (SHOW PROCESSLIST) — a silently killed dump thread produces exactly this stall.","If using a load balancer / RDS proxy in front of MySQL, raise its idle timeout above the canal read timeout."],"exampleFix":"# before (instance.properties)\ncanal.instance.network.readTimeout = 30000\ncanal.instance.detecting.enable = false\n\n# after\ncanal.instance.network.readTimeout = 90000\ncanal.instance.detecting.enable = true\ncanal.instance.detecting.sql = SELECT 1\ncanal.instance.detecting.intervalInSeconds = 3","handlingStrategy":"retry","validationCode":"// Before reading, sanity-check that the channel is alive and that a heartbeat is configured\nif (!socketChannel.isConnected()) {\n    throw new ConnectException(\"channel not connected before read\");\n}\n// Ensure canal.instance.detecting.enable=true + detecting.sql are set in instance.properties\n// so MySQL keeps the link alive during idle periods (prevents the stall that causes the timeout).","typeGuard":null,"tryCatchPattern":"try {\n    byte[] data = socketChannel.read(readSize, readTimeoutMs);\n} catch (IOException e) {\n    if (e.getMessage() != null && e.getMessage().startsWith(\"socket read timeout occured\")) {\n        // transient: log readSize/readableBytes/timeout from the message and let the\n        // parser-level reconnect/retry logic kick in (HeartBeatHAController + MysqlEventParser retry)\n        log.warn(\"canal read timeout, will retry: {}\", e.getMessage());\n        throw e; // surfaced & retried upstream\n    }\n    throw e;\n}","preventionTips":["Always enable canal.instance.detecting.enable with a short detecting interval to keep idle MySQL connections alive.","Size canal.instance.network.readTimeout above the worst-case idle/replication-lag window.","Raise intermediary (firewall/NAT/LB/RDS) idle timeouts above canal's read timeout.","Monitor SHOW PROCESSLIST for the canal dump thread; a missing dump thread precedes this error."],"tags":["network","netty","socket","timeout","binlog","replication"],"backgroundTag":null,"analyzedSha":"87be50e87686a3e8af08c368d0e1ffd1f59eb04a","analyzedAt":"2026-08-14T04:30:11.918Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}