alibaba/canal · error · SocketTimeoutException

Timeout occurred, failed to read total

Error message

Timeout occurred, failed to read total {} bytes in {} milliseconds, actual read only {} bytes

What it means

Thrown by read(int,int) when the accumulated wait reaches the requested timeout without reading all bytes (remain > 0 && accTimeout >= timeout). The message reports readSize, the budget in ms, and how many bytes were actually read. The loop accumulates SO_TIMEOUT (1s) increments per SocketTimeoutException catch, so the budget is consumed in roughly 1-second granularity.

Solutions

1. Increase the read timeout budget passed to read(readSize, timeout) for large payloads.
2. Scale the timeout with the expected payload size (the Netty path does this; the BIO path does not).
3. Reduce concurrent load on the MySQL server or move the binlog client closer (network latency).
4. Check for slow queries/long transactions on the master that delay binlog delivery.
5. Verify SO_TIMEOUT granularity (1s) is acceptable for the use case.

Example fix

// before
byte[] b = channel.read(bodyLen, 3000); // too small for big rows

// after
int budget = Math.max(3000, bodyLen / 1024); // ~1ms per KB floor
byte[] b = channel.read(bodyLen, budget);

// choose a budget scaled to payload size before calling read
int budget = Math.max(defaultTimeout, readSize / 1024); // ~1ms/KB floor
byte[] b = channel.read(readSize, budget);

try {
    byte[] b = channel.read(readSize, budget);
} catch (java.net.SocketTimeoutException e) {
    // back off and retry; if persistent, investigate server load/network
}

When it happens

Trigger: Calling read(readSize, timeout) when the server stalls or sends bytes too slowly to fill readSize within timeout milliseconds. Common for large packets or a slow/overloaded MySQL server, or when the network is throttled.

Common situations: Server under heavy load; reading a very large binlog event/transaction; saturated network; the readSize requested is larger than what the server sends before a stall; misconfigured too-small timeout relative to expected packet size.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• EOF encountered.
• Socket already closed.
• socket read timeout occured ! readSize =
• can't create socket!
• command : show variables like 'server_id' has an error!

AI-assisted analysis of alibaba/canal@87be50e876 (2026-08-14). Data as JSON: /api/errors/a159016f1cee7ab0. Report an issue: GitHub.

Appendix: source

Thrown at driver/src/main/java/com/alibaba/otter/canal/parse/driver/mysql/socket/BioSocketChannel.java:95

            throw new SocketException("Socket already closed.");
        }
        while (remain > 0 && accTimeout < timeout) {
            try {
                int read = input.read(data, readSize - remain, remain);
                if (read > -1) {
                    remain -= read;
                } else {
                    throw new IOException("EOF encountered.");
                }
            } catch (SocketTimeoutException te) {
                if (Thread.interrupted()) {
                    throw new ClosedByInterruptException();
                }
                accTimeout += SO_TIMEOUT;
            }
        }
        if (remain > 0 && accTimeout >= timeout) {
            throw new SocketTimeoutException(
                "Timeout occurred, failed to read total " + readSize + " bytes in " + timeout
                                             + " milliseconds, actual read only " + (readSize - remain) + " bytes");
        }
        return data;
    }

    @Override
    public void read(byte[] data, int off, int len, int timeout) throws IOException {
        InputStream input = this.input;
        int accTimeout = 0;
        if (input == null) {
            throw new SocketException("Socket already closed.");
        }

        int n = 0;
        while (n < len && accTimeout < timeout) {
            try {
                int read = input.read(data, off + n, len - n);

View on GitHub (pinned to 87be50e876)