VictoriaMetrics/VictoriaMetrics · error

cannot read prometheus remote_write data from client in %d s

Error message

cannot read prometheus remote_write data from client in %d seconds: %w

What it means

Parse for the Prometheus remote_write protocol wraps any failure from reading and decompressing the client request body, including elapsed seconds in the message. The underlying cause (oversized body, decompression failure, read timeout, malformed chunked stream) is chained via %w. The elapsed-time figure helps distinguish slow/stalled uploads from instant failures.

Source

Thrown at lib/protoparser/promremotewrite/stream/streamparser.go:31

	"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/protoparserutil"
	"github.com/VictoriaMetrics/metrics"
)

var maxInsertRequestSize = flagutil.NewBytes("maxInsertRequestSize", 32*1024*1024, "The maximum size in bytes of a single Prometheus remote_write API request")

// Parse parses Prometheus remote_write message from reader and calls callback for the parsed timeseries.
//
// callback shouldn't hold tss after returning.
func Parse(r io.Reader, isVMRemoteWrite bool, callback func(tss []prompb.TimeSeries, mms []prompb.MetricMetadata) error) error {
	startTime := fasttime.UnixTimestamp()

	readCalls.Inc()
	err := protoparserutil.ReadUncompressedData(r, "", maxInsertRequestSize, func(data []byte) error {
		return parseRequestBody(data, isVMRemoteWrite, callback)
	})
	if err != nil {
		readErrors.Inc()
		return fmt.Errorf("cannot read prometheus remote_write data from client in %d seconds: %w", fasttime.UnixTimestamp()-startTime, err)
	}
	return nil
}

func parseRequestBody(data []byte, isVMRemoteWrite bool, callback func(tss []prompb.TimeSeries, mms []prompb.MetricMetadata) error) error {
	// Synchronously process the request in order to properly return errors to Parse caller,
	// so it could properly return HTTP 503 status code in response.
	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896
	bb := bodyBufferPool.Get()
	defer bodyBufferPool.Put(bb)
	if isVMRemoteWrite {
		var err error
		bb.B, err = encoding.DecompressZSTDLimited(bb.B[:0], data, maxInsertRequestSize.IntN())
		if err != nil {
			// Fall back to Snappy decompression, since vmagent may send snappy-encoded messages
			// with 'Content-Encoding: zstd' header if they were put into persistent queue before vmagent restart.
			// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301
			//

View on GitHub (pinned to 5079fb58f1)

Solutions

1. Raise -maxInsertRequestSize on the server if bodies legitimately exceed it
2. Check the wrapped cause for the exact read/decompression failure
3. Tune vmagent's -remoteWrite.maxBlockSize / queues to produce smaller requests
4. Verify proxy/LB timeouts allow the full upload duration

Example fix

// before (server)
./victoria-metrics -maxInsertRequestSize=33554432
// after
./victoria-metrics -maxInsertRequestSize=268435456

// Go: client-side size check before remote_write
if int64(len(compressedBody)) > maxInsertRequestSize {
    return errors.New("request too large; reduce batch or raise -maxInsertRequestSize")
}

err := rwstreamparser.Parse(r, isVMRemoteWrite, callback)
if err != nil {
    var netErr net.Error
    if errors.As(err, &netErr) && netErr.Timeout() {
        return retryWithBackoff()
    }
    log.Printf("remote_write read failed after N s, cause: %v", errors.Unwrap(err))
    return err
}

When it happens

Trigger: POSTs to /api/v1/write (remote_write) where ReadUncompressedData fails: body larger than -maxInsertRequestSize, broken gzip/snappy/zstd stream, client disconnect mid-upload, or read timeout after N seconds.

Common situations: vmagent configured with a larger max block size than the server's -maxInsertRequestSize; network drops during large remote_write pushes; proxies buffering/timing out; payload compression mismatch.

Related errors

• cannot read isCompressed flag: %w
• cannot write isCompressed flag: %w
• cannot read the next field: %w
• cannot create client for %q: %w
• cannot query %q: %w

AI-assisted analysis of VictoriaMetrics/VictoriaMetrics@5079fb58f1 (2026-09-03). Data as JSON: /api/errors/59266fd494b865d7. Report an issue: GitHub.