{"record":{"id":"a091b814004e05e7","repo":"thanos-io/thanos","slug":"add-d-series","errorCode":null,"errorMessage":"add %d series","messagePattern":"add (.+?) series","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/receive/writer_errors.go","lineNumber":122,"sourceCode":"\tcase errors.Is(err, storage.ErrOutOfOrderExemplar):\n\t\ta.numExemplarsOutOfOrder++\n\t\tlevel.Debug(exLogger).Log(\"msg\", \"Out of order exemplar\")\n\tcase errors.Is(err, storage.ErrDuplicateExemplar):\n\t\ta.numExemplarsDuplicate++\n\t\tlevel.Debug(exLogger).Log(\"msg\", \"Duplicate exemplar\")\n\tcase errors.Is(err, storage.ErrExemplarLabelLength):\n\t\ta.numExemplarsLabelLength++\n\t\tlevel.Debug(exLogger).Log(\"msg\", \"Label length for exemplar exceeds max limit\", \"limit\", exemplar.ExemplarMaxLabelSetLength)\n\tdefault:\n\t\tlevel.Debug(exLogger).Log(\"msg\", \"Error ingesting exemplar\", \"err\", err)\n\t}\n}\n\nfunc (a *writeErrorTracker) collectErrors(tLogger log.Logger) writeErrors {\n\tvar errs writeErrors\n\tif a.numLabelsOutOfOrder > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on series with out-of-order labels\", \"numDropped\", a.numLabelsOutOfOrder)\n\t\terrs.Add(errors.Wrapf(labelpb.ErrOutOfOrderLabels, \"add %d series\", a.numLabelsOutOfOrder))\n\t}\n\tif a.numLabelsDuplicates > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on series with duplicate labels\", \"numDropped\", a.numLabelsDuplicates)\n\t\terrs.Add(errors.Wrapf(labelpb.ErrDuplicateLabels, \"add %d series\", a.numLabelsDuplicates))\n\t}\n\tif a.numLabelsEmpty > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on series with empty label name or value\", \"numDropped\", a.numLabelsEmpty)\n\t\terrs.Add(errors.Wrapf(labelpb.ErrEmptyLabels, \"add %d series\", a.numLabelsEmpty))\n\t}\n\n\tif a.numSamplesOutOfOrder > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting out-of-order samples\", \"numDropped\", a.numSamplesOutOfOrder)\n\t\terrs.Add(errors.Wrapf(storage.ErrOutOfOrderSample, \"add %d samples\", a.numSamplesOutOfOrder))\n\t}\n\tif a.numSamplesDuplicates > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting samples with different value but same timestamp\", \"numDropped\", a.numSamplesDuplicates)\n\t\terrs.Add(errors.Wrapf(storage.ErrDuplicateSampleForTimestamp, \"add %d samples\", a.numSamplesDuplicates))\n\t}","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/writer_errors.go#L104-L140","documentation":"Aggregated error from Writer.Write produced by writeErrorTracker.collectErrors: N series were dropped because their label sets were out of order (labelpb.ErrOutOfOrderLabels wrapped as 'add %d series'). Labels are out of order when not sorted lexicographically by name as Prometheus requires, so the whole series is rejected before appending.","triggerScenarios":"Remote-write payload contains time series whose Labels slice is not sorted by label name; ValidateLabels returns ErrOutOfOrderLabels for one or more series and the count is wrapped at the end of Write.","commonSituations":"Custom senders/SDKs building label sets manually without sorting; exporters emitting unsorted labels; middleware rewriting labels and breaking sort order.","solutions":["Sort the series' labels by name before sending (labels.Labels.Sort or sort.Slice by Name).","Check which series are dropped via the receiver's warn log 'Error on series with out-of-order labels'.","Fix the producing exporter/library to emit sorted, unique labels per Prometheus write spec.","Upgrade the sending Prometheus - it sorts labels before remote write."],"exampleFix":"// before\nlbls := []labelpb.ZLabel{{Name: \"z\", Value: \"1\"}, {Name: \"a\", Value: \"2\"}}\n// after\nsort.Slice(lbls, func(i, j int) bool { return lbls[i].Name < lbls[j].Name })","handlingStrategy":"validation","validationCode":"sort.Slice(series[i].Labels, func(a, b int) bool { return series[i].Labels[a].Name < series[i].Labels[b].Name })\nif err := labelpb.ValidateLabels(series[i].Labels); err != nil {\n    return err // drop or fix before sending\n}","typeGuard":null,"tryCatchPattern":"if err := writer.Write(ctx, tenant, series); err != nil {\n    var we receive.WriteErrors // inspect aggregated errors\n    if strings.Contains(err.Error(), labelpb.ErrOutOfOrderLabels.Error()) {\n        // fix sender labeling; series were dropped\n    }\n}","preventionTips":["Always sort labels by name before remote write","Validate with labelpb.ValidateLabels client-side","Fix custom exporters to emit sorted labels","Watch receiver 'out-of-order labels' warn logs"],"tags":["thanos","receive","labels","validation","remote-write","go"],"backgroundTag":"schema-validation-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}