{"record":{"id":"27303f443b1df243","repo":"thanos-io/thanos","slug":"request-at","errorCode":null,"errorMessage":"request.At","messagePattern":"request\\.At","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/capnproto_writer.go","lineNumber":75,"sourceCode":"\t}\n\tgetRef := app.(storage.GetRef)\n\tvar (\n\t\tref          storage.SeriesRef\n\t\terrorTracker = &writeErrorTracker{}\n\t)\n\tapp = &ReceiveAppender{\n\t\ttLogger:        tLogger,\n\t\ttooFarInFuture: r.opts.TooFarInFutureTimeWindow,\n\t\tAppender:       app,\n\t}\n\n\tvar (\n\t\tseries  writecapnp.Series\n\t\tbuilder labels.ScratchBuilder\n\t)\n\tfor wreq.Next() {\n\t\tif err := wreq.At(&series); err != nil {\n\t\t\treturn errors.Wrap(err, \"request.At\")\n\t\t}\n\n\t\t// Check if time series labels are valid. If not, skip the time series\n\t\t// and report the error.\n\t\tif err := validateLabels(series.Labels); err != nil {\n\t\t\tlset := &labelpb.ZLabelSet{Labels: labelpb.ZLabelsFromPromLabels(series.Labels)}\n\t\t\terrorTracker.addLabelsError(err, lset, tLogger)\n\t\t\tcontinue\n\t\t}\n\n\t\tvar lset labels.Labels\n\t\t// Check if the TSDB has cached reference for those labels.\n\t\tref, lset = getRef.GetRef(series.Labels, series.Labels.Hash())\n\t\tif ref == 0 {\n\t\t\t// NOTE(GiedriusS): do a deep copy because the labels are reused in the capnp message.\n\t\t\t// Creation of new series is much rarer compared to adding extra samples\n\t\t\t// to an existing series.\n\t\t\tbuilder.Reset()","sourceCodeStart":57,"sourceCodeEnd":93,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/capnproto_writer.go#L57-L93","documentation":"This error wraps a failure to decode the next series from a Cap'n Proto write request inside CapNProtoWriter.Write. The receive writer streams WriteRequest messages; wreq.At(&series) fails when the wire payload is malformed or does not match the expected capnp schema. It means the incoming replication/forwarding message is corrupt or schema-incompatible.","triggerScenarios":"wreq.Next() yields a message and wreq.At(&series) fails during decode — corrupt frames on the wire, truncated request, or sender/receiver capnp schema version mismatch.","commonSituations":"Mixed Thanos versions in a hashring where the capnp write protocol differs; network middleware truncating or mangling frames; a buggy sender serializing invalid label/sample data.","solutions":["Upgrade/align Thanos versions across all Receive peers so the capnp schema matches","Check network proxies/load balancers for frame truncation or corruption","Log the raw failing request and compare against the expected writecapnp schema","Retry the replication from the sender; the error is per-series and the writer skips to the next series after reporting"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// sender side: verify request size and non-nil before sending\nif req == nil || len(req.Timeseries) == 0 {\n\treturn errors.New(\"empty write request\")\n}","typeGuard":null,"tryCatchPattern":"err := writer.Write(ctx, wreq)\nvar wrapped interface{ Cause() error }\nif errors.As(err, &wrapped) {\n\tlog.Printf(\"capnp decode failed: %v\", errors.Cause(err))\n\t// request sender re-replicate this batch\n}","preventionTips":["Run identical Thanos versions across the whole hashring","Avoid proxies that re-buffer/re-frame raw capnp connections","Add integration tests replicating batches between peer versions"],"tags":["capnp","deserialization","receive"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}