{"record":{"id":"b0fa6a84f3047a26","repo":"thanos-io/thanos","slug":"add-d-samples","errorCode":null,"errorMessage":"add %d samples","messagePattern":"add (.+?) samples","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/writer_errors.go","lineNumber":135,"sourceCode":"\nfunc (a *writeErrorTracker) collectErrors(tLogger log.Logger) writeErrors {\n\tvar errs writeErrors\n\tif a.numLabelsOutOfOrder > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on series with out-of-order labels\", \"numDropped\", a.numLabelsOutOfOrder)\n\t\terrs.Add(errors.Wrapf(labelpb.ErrOutOfOrderLabels, \"add %d series\", a.numLabelsOutOfOrder))\n\t}\n\tif a.numLabelsDuplicates > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on series with duplicate labels\", \"numDropped\", a.numLabelsDuplicates)\n\t\terrs.Add(errors.Wrapf(labelpb.ErrDuplicateLabels, \"add %d series\", a.numLabelsDuplicates))\n\t}\n\tif a.numLabelsEmpty > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on series with empty label name or value\", \"numDropped\", a.numLabelsEmpty)\n\t\terrs.Add(errors.Wrapf(labelpb.ErrEmptyLabels, \"add %d series\", a.numLabelsEmpty))\n\t}\n\n\tif a.numSamplesOutOfOrder > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting out-of-order samples\", \"numDropped\", a.numSamplesOutOfOrder)\n\t\terrs.Add(errors.Wrapf(storage.ErrOutOfOrderSample, \"add %d samples\", a.numSamplesOutOfOrder))\n\t}\n\tif a.numSamplesDuplicates > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting samples with different value but same timestamp\", \"numDropped\", a.numSamplesDuplicates)\n\t\terrs.Add(errors.Wrapf(storage.ErrDuplicateSampleForTimestamp, \"add %d samples\", a.numSamplesDuplicates))\n\t}\n\tif a.numSamplesOutOfBounds > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting samples that are too old or are too far into the future\", \"numDropped\", a.numSamplesOutOfBounds)\n\t\terrs.Add(errors.Wrapf(storage.ErrOutOfBounds, \"add %d samples\", a.numSamplesOutOfBounds))\n\t}\n\tif a.numSamplesTooOld > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting samples that are outside of the allowed out-of-order time window\", \"numDropped\", a.numSamplesTooOld)\n\t\terrs.Add(errors.Wrapf(storage.ErrTooOldSample, \"add %d samples\", a.numSamplesTooOld))\n\t}\n\n\tif a.numExemplarsOutOfOrder > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting out-of-order exemplars\", \"numDropped\", a.numExemplarsOutOfOrder)\n\t\terrs.Add(errors.Wrapf(storage.ErrOutOfOrderExemplar, \"add %d exemplars\", a.numExemplarsOutOfOrder))\n\t}","sourceCodeStart":117,"sourceCodeEnd":153,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/writer_errors.go#L117-L153","documentation":"Thanos Receive wraps storage.ErrOutOfOrderSample ('out of order sample') reporting N dropped samples. The TSDB head rejected a sample because its timestamp is older than the most recent sample already appended for that series (outside the allowed out-of-order window, if configured).","triggerScenarios":"A remote-write request contains a sample for a series whose timestamp is earlier than the series' current head max time and beyond the out-of-order acceptance window; append returns ErrOutOfOrderSample, the tracker counts it, collectErrors emits 'add N samples: out of order sample'.","commonSituations":"Replaying old data after a receiver outage or failover; clock skew between producers; multiple scrapers of the same target ingesting with different delays; backfilled historical data sent through live remote-write.","solutions":["Enable and size the out-of-order window (--tsdb.out-of-order-time-window) on the receiver if delayed data is expected.","Fix producer-side clock synchronization (NTP) or the ordering bug in the sender.","Use backfilling (tsdb block tooling) for historical data instead of live remote-write.","Check prometheus_tsdb_head_max_time on the receiver to confirm which samples fall outside the head."],"exampleFix":"// before: receiver rejects all late samples\n// after: allow a 5m out-of-order window\nr := receive.New(...)\nopts.OutOfOrderTimeWindow = 5 * time.Minute","handlingStrategy":"retry","validationCode":"if sample.Ts < lastKnownHeadMaxTimeMs {\n    return fmt.Errorf(\"sample ts %d older than head max %d\", sample.Ts, lastKnownHeadMaxTimeMs)\n}","typeGuard":null,"tryCatchPattern":"if errors.Is(err, storage.ErrOutOfOrderSample) { // buffer for backfill or drop with metric increment; do not hot-retry }","preventionTips":["Run NTP on all producers.","Use block backfill tooling for historical data instead of live remote-write.","Size the receiver's out-of-order window to expected producer lateness."],"tags":["prometheus","remote-write","tsdb","ingestion"],"backgroundTag":"invalid-state-transition","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}