{"record":{"id":"59266fd494b865d7","repo":"VictoriaMetrics/VictoriaMetrics","slug":"cannot-read-prometheus-remote-write-data-from-clie","errorCode":null,"errorMessage":"cannot read prometheus remote_write data from client in %d seconds: %w","messagePattern":"cannot read prometheus remote_write data from client in (.+?) seconds: %w","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"lib/protoparser/promremotewrite/stream/streamparser.go","lineNumber":31,"sourceCode":"\t\"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/protoparserutil\"\n\t\"github.com/VictoriaMetrics/metrics\"\n)\n\nvar maxInsertRequestSize = flagutil.NewBytes(\"maxInsertRequestSize\", 32*1024*1024, \"The maximum size in bytes of a single Prometheus remote_write API request\")\n\n// Parse parses Prometheus remote_write message from reader and calls callback for the parsed timeseries.\n//\n// callback shouldn't hold tss after returning.\nfunc Parse(r io.Reader, isVMRemoteWrite bool, callback func(tss []prompb.TimeSeries, mms []prompb.MetricMetadata) error) error {\n\tstartTime := fasttime.UnixTimestamp()\n\n\treadCalls.Inc()\n\terr := protoparserutil.ReadUncompressedData(r, \"\", maxInsertRequestSize, func(data []byte) error {\n\t\treturn parseRequestBody(data, isVMRemoteWrite, callback)\n\t})\n\tif err != nil {\n\t\treadErrors.Inc()\n\t\treturn fmt.Errorf(\"cannot read prometheus remote_write data from client in %d seconds: %w\", fasttime.UnixTimestamp()-startTime, err)\n\t}\n\treturn nil\n}\n\nfunc parseRequestBody(data []byte, isVMRemoteWrite bool, callback func(tss []prompb.TimeSeries, mms []prompb.MetricMetadata) error) error {\n\t// Synchronously process the request in order to properly return errors to Parse caller,\n\t// so it could properly return HTTP 503 status code in response.\n\t// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896\n\tbb := bodyBufferPool.Get()\n\tdefer bodyBufferPool.Put(bb)\n\tif isVMRemoteWrite {\n\t\tvar err error\n\t\tbb.B, err = encoding.DecompressZSTDLimited(bb.B[:0], data, maxInsertRequestSize.IntN())\n\t\tif err != nil {\n\t\t\t// Fall back to Snappy decompression, since vmagent may send snappy-encoded messages\n\t\t\t// with 'Content-Encoding: zstd' header if they were put into persistent queue before vmagent restart.\n\t\t\t// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301\n\t\t\t//","sourceCodeStart":13,"sourceCodeEnd":49,"githubUrl":"https://github.com/VictoriaMetrics/VictoriaMetrics/blob/5079fb58f1e8e62113f90c945ad71586c797d770/lib/protoparser/promremotewrite/stream/streamparser.go#L13-L49","documentation":"Parse for the Prometheus remote_write protocol wraps any failure from reading and decompressing the client request body, including elapsed seconds in the message. The underlying cause (oversized body, decompression failure, read timeout, malformed chunked stream) is chained via %w. The elapsed-time figure helps distinguish slow/stalled uploads from instant failures.","triggerScenarios":"POSTs to /api/v1/write (remote_write) where ReadUncompressedData fails: body larger than -maxInsertRequestSize, broken gzip/snappy/zstd stream, client disconnect mid-upload, or read timeout after N seconds.","commonSituations":"vmagent configured with a larger max block size than the server's -maxInsertRequestSize; network drops during large remote_write pushes; proxies buffering/timing out; payload compression mismatch.","solutions":["Raise -maxInsertRequestSize on the server if bodies legitimately exceed it","Check the wrapped cause for the exact read/decompression failure","Tune vmagent's -remoteWrite.maxBlockSize / queues to produce smaller requests","Verify proxy/LB timeouts allow the full upload duration"],"exampleFix":"// before (server)\n./victoria-metrics -maxInsertRequestSize=33554432\n// after\n./victoria-metrics -maxInsertRequestSize=268435456","handlingStrategy":"validation","validationCode":"// Go: client-side size check before remote_write\nif int64(len(compressedBody)) > maxInsertRequestSize {\n    return errors.New(\"request too large; reduce batch or raise -maxInsertRequestSize\")\n}","typeGuard":null,"tryCatchPattern":"err := rwstreamparser.Parse(r, isVMRemoteWrite, callback)\nif err != nil {\n    var netErr net.Error\n    if errors.As(err, &netErr) && netErr.Timeout() {\n        return retryWithBackoff()\n    }\n    log.Printf(\"remote_write read failed after N s, cause: %v\", errors.Unwrap(err))\n    return err\n}","preventionTips":["Set vmagent -remoteWrite.maxBlockSize below server -maxInsertRequestSize","Keep LB timeouts longer than slowest expected upload","Monitor readErrors counter for trending failures"],"tags":["remote-write","network","compression","http-client"],"backgroundTag":"remote-write-read-failure","analyzedSha":"5079fb58f1e8e62113f90c945ad71586c797d770","analyzedAt":"2026-09-03T18:10:26.153Z","contentChangedAt":"2026-09-03T18:10:26.153Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}