VictoriaMetrics/VictoriaMetrics · error

cannot flush %q to server: %w

Error message

cannot flush %q to server: %w

What it means

writeMessage in lib/handshake sends a handshake message over a net.Conn and, if the connection implements a Flush method (buffered/borrowed conn), flushes it. This error wraps the underlying Flush failure, meaning the buffered data could not be pushed to the peer.

Source

Thrown at lib/handshake/handshake.go:292

	return writeMessage(c, string(buf[:]))
}

func readIsCompressed(c net.Conn) (bool, error) {
	buf, err := readData(c, 1)
	if err != nil {
		return false, err
	}
	isCompressed := buf[0] != 0
	return isCompressed, nil
}

func writeMessage(c net.Conn, msg string) error {
	if _, err := io.WriteString(c, msg); err != nil {
		return fmt.Errorf("cannot write %q to server: %w", msg, err)
	}
	if fc, ok := c.(flusher); ok {
		if err := fc.Flush(); err != nil {
			return fmt.Errorf("cannot flush %q to server: %w", msg, err)
		}
	}
	return nil
}

type flusher interface {
	Flush() error
}

func readMessage(c net.Conn, msg string) error {
	buf, err := readData(c, len(msg))
	if err != nil {
		return err
	}
	if string(buf) != msg {
		return fmt.Errorf("unexpected message obtained; got %q; want %q", buf, msg)
	}
	return nil

View on GitHub (pinned to 5079fb58f1)

Solutions

1. Check connectivity/firewall between the two endpoints (the wrapped error names the root cause).
2. Add retries with backoff around the connection/handshake on the client side.
3. Verify the peer service is running and accepting connections (logs, liveness probes).
4. Ensure read/write deadlines on the conn are generous enough for slow networks.

Example fix

// before
if err := writeMessage(conn, msg); err != nil {
    return err
}
// after
if err := writeMessage(conn, msg); err != nil {
    logger.Printf("handshake flush failed, retrying: %v", err)
    conn.Close()
    conn, err = dialWithRetry(addr, 3, time.Second)
    if err != nil { return err }
    return writeMessage(conn, msg)
}

// Before using the conn, verify it is alive:
if tcp, ok := c.(*net.TCPConn); ok {
    if err := tcp.SetWriteDeadline(time.Now().Add(5 * time.Second)); err != nil {
        return fmt.Errorf("conn unusable: %w", err)
    }
}
if n, err := c.Write([]byte{}); err != nil || n < 0 {
    return fmt.Errorf("conn already broken: %w", err)
}

err := writeMessage(conn, msg)
if err != nil {
    // wrapped Flush failure is unrecoverable on this conn
    conn.Close()
    conn, err = redial(addr) // retry on a fresh connection
}

When it happens

Trigger: Calling writeMessage (via genericServer, genericClient, or writeIsCompressed) when the connection's Flush() returns an error, typically because the underlying TCP conn is broken or the peer closed it mid-handshake.

Common situations: Peer process crashed or restarted during handshake; network drop; connection deadline expired; load balancer terminated an idle connection just as the handshake started.

Related errors

• cannot write %q to server: %w
• cannot read message with size %d: %w; read only %d bytes
• cannot read uint32: %w
• cannot read uint64: %w
• cannot read int64: %w

AI-assisted analysis of VictoriaMetrics/VictoriaMetrics@5079fb58f1 (2026-09-03). Data as JSON: /api/errors/d4dbfd319cb67e80. Report an issue: GitHub.