{"record":{"id":"75f620c36b0a5c2f","repo":"thanos-io/thanos","slug":"commit-samples","errorCode":null,"errorMessage":"commit samples","messagePattern":"commit samples","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/receive/capnproto_writer.go","lineNumber":146,"sourceCode":"\t\t\t\t\texlset := &labelpb.ZLabelSet{Labels: labelpb.ZLabelsFromPromLabels(copiedLabels)}\n\t\t\t\t\terrorTracker.addLabelsError(err, exlset, exLogger)\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tif _, err = app.AppendExemplar(ref, lset, exemplar.Exemplar{\n\t\t\t\t\tLabels: copiedLabels,\n\t\t\t\t\tValue:  ex.Value,\n\t\t\t\t\tTs:     ex.Ts,\n\t\t\t\t\tHasTs:  true,\n\t\t\t\t}); err != nil {\n\t\t\t\t\terrorTracker.addExemplarError(err, exLogger)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\terrs := errorTracker.collectErrors(tLogger)\n\tif err := app.Commit(); err != nil {\n\t\terrs.Add(errors.Wrap(err, \"commit samples\"))\n\t}\n\treturn errs.ErrOrNil()\n}\n\n// ValidateLabels validates label names and values (checks for empty\n// names and values, out of order labels and duplicate label names)\n// Returns appropriate error if validation fails on a label.\nfunc validateLabels(lbls labels.Labels) error {\n\tif lbls.Len() == 0 {\n\t\treturn labelpb.ErrEmptyLabels\n\t}\n\n\tvar (\n\t\tisFirst  = true\n\t\tprevName string\n\t)\n\treturn lbls.Validate(func(l labels.Label) error {\n\t\tif l.Name == \"\" || l.Value == \"\" {","sourceCodeStart":128,"sourceCodeEnd":164,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/capnproto_writer.go#L128-L164","documentation":"This error wraps a failure from app.Commit() at the end of CapNProtoWriter.Write. Commit flushes all appended series into the TSDB head; a failure here means none of the batch's samples were durably accepted. It is collected into the writeErrorTracker along with any per-series validation errors and returned to the caller.","triggerScenarios":"app.Commit() returns an error — head storage failure, too old/out-of-bounds sample timestamps rejected at commit, or the appender was invalidated mid-batch (e.g. head truncation).","commonSituations":"Samples outside the allowed time window (min/max block duration) arriving from misaligned senders; disk full or I/O errors on the TSDB directory; head compaction racing with writes.","solutions":["Inspect the wrapped cause for timestamp-out-of-bounds vs storage I/O errors","Align sender clocks and enforce ingestion time windowing (out-of-order/time-window flags) so samples fall within the head's bounds","Free disk space / fix storage backend health","Retry the replication; commit failures are batch-level so the sender must resend"],"exampleFix":"// before\nif err := app.Commit(); err != nil {\n\treturn err\n}\n// after\nif err := app.Commit(); err != nil {\n\terrs.Add(errors.Wrap(err, \"commit samples\"))\n\tlogger.Error(errs, \"commit failed, requester should resend batch\")\n}\nreturn errs.ErrOrNil()","handlingStrategy":"try-catch","validationCode":"// sender: clamp timestamps to the receiver's acceptance window before writing\nnow := time.Now().UnixMilli()\nwindow := int64((13 * 24 * time.Hour) / time.Millisecond)\nfor i := range samples {\n\tif samples[i].T > now || samples[i].T < now-window { drop(i) }\n}","typeGuard":null,"tryCatchPattern":"err := writer.Write(ctx, wreq)\nif err != nil {\n\tif strings.Contains(err.Error(), \"commit samples\") {\n\t\t// batch-level failure: re-queue the entire batch\n\t}\n}","preventionTips":["Keep receiver clocks NTP-synced and time windows aligned cluster-wide","Alert on TSDB disk usage before the head fills","Enable out-of-order/windowed ingestion flags consistently on all receivers"],"tags":["tsdb","commit","data-loss"],"backgroundTag":"database-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}