{"record":{"id":"d1635cfb2437acd1","repo":"thanos-io/thanos","slug":"add-d-exemplars","errorCode":null,"errorMessage":"add %d exemplars","messagePattern":"add (.+?) exemplars","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/receive/writer_errors.go","lineNumber":152,"sourceCode":"\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting out-of-order samples\", \"numDropped\", a.numSamplesOutOfOrder)\n\t\terrs.Add(errors.Wrapf(storage.ErrOutOfOrderSample, \"add %d samples\", a.numSamplesOutOfOrder))\n\t}\n\tif a.numSamplesDuplicates > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting samples with different value but same timestamp\", \"numDropped\", a.numSamplesDuplicates)\n\t\terrs.Add(errors.Wrapf(storage.ErrDuplicateSampleForTimestamp, \"add %d samples\", a.numSamplesDuplicates))\n\t}\n\tif a.numSamplesOutOfBounds > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting samples that are too old or are too far into the future\", \"numDropped\", a.numSamplesOutOfBounds)\n\t\terrs.Add(errors.Wrapf(storage.ErrOutOfBounds, \"add %d samples\", a.numSamplesOutOfBounds))\n\t}\n\tif a.numSamplesTooOld > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting samples that are outside of the allowed out-of-order time window\", \"numDropped\", a.numSamplesTooOld)\n\t\terrs.Add(errors.Wrapf(storage.ErrTooOldSample, \"add %d samples\", a.numSamplesTooOld))\n\t}\n\n\tif a.numExemplarsOutOfOrder > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting out-of-order exemplars\", \"numDropped\", a.numExemplarsOutOfOrder)\n\t\terrs.Add(errors.Wrapf(storage.ErrOutOfOrderExemplar, \"add %d exemplars\", a.numExemplarsOutOfOrder))\n\t}\n\tif a.numExemplarsDuplicate > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting duplicate exemplars\", \"numDropped\", a.numExemplarsDuplicate)\n\t\terrs.Add(errors.Wrapf(storage.ErrDuplicateExemplar, \"add %d exemplars\", a.numExemplarsDuplicate))\n\t}\n\tif a.numExemplarsLabelLength > 0 {\n\t\tlevel.Warn(tLogger).Log(\"msg\", \"Error on ingesting exemplars with label length exceeding maximum limit\", \"numDropped\", a.numExemplarsLabelLength)\n\t\terrs.Add(errors.Wrapf(storage.ErrExemplarLabelLength, \"add %d exemplars\", a.numExemplarsLabelLength))\n\t}\n\treturn errs\n}\n","sourceCodeStart":134,"sourceCodeEnd":164,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/writer_errors.go#L134-L164","documentation":"Thanos Receive wraps storage.ErrOutOfOrderExemplar ('out of order exemplar') reporting N dropped exemplars. Exemplars are rejected when their timestamp is older than the last exemplar already stored for that series; the exemplar ring buffer enforces per-series ordering.","triggerScenarios":"A remote-write request includes exemplars whose timestamps precede previously accepted exemplars for the same series; appendExemplar returns storage.ErrOutOfOrderExemplar, the tracker counts it, collectErrors emits 'add N exemplars: out of order exemplar'.","commonSituations":"Sender retrying exemplar batches after a partial failure; multiple producers scraping the same instrumented target and delivering exemplars at different rates; clock skew on the application emitting OpenTelemetry/Prometheus exemplars.","solutions":["Ensure exemplar timestamps are monotonically increasing per series at the producer.","Remove duplicate scrapers of the same target that emit interleaved exemplar streams.","Sync producer clocks (NTP) to avoid skewed timestamps across replicas.","Drop or reorder stale exemplar buffers before re-sending after failures."],"exampleFix":"// before: buffered exemplars replayed in arbitrary order after retry\nsendExemplars(shuffledBuffer)\n// after: sort by timestamp before sending\nsort.SliceStable(buf, func(i, j int) bool { return buf[i].Ts < buf[j].Ts })\nsendExemplars(buf)","handlingStrategy":"validation","validationCode":"if ex.Ts <= lastExemplarTsPerSeries[seriesKey] {\n    return fmt.Errorf(\"exemplar ts %d not newer than last %d\", ex.Ts, lastExemplarTsPerSeries[seriesKey])\n}","typeGuard":null,"tryCatchPattern":"if errors.Is(err, storage.ErrOutOfOrderExemplar) { // sort/drop stale exemplars; never hot-retry or the same drop repeats }","preventionTips":["Guarantee per-series monotonic exemplar timestamps at the producer.","Avoid multiple producers scraping the same instrumented target.","NTP-sync hosts emitting exemplars."],"tags":["prometheus","remote-write","exemplars","ingestion"],"backgroundTag":"invalid-state-transition","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}