{"record":{"id":"fce7c5f7b4e5e0f4","repo":"thanos-io/thanos","slug":"series-v-out-of-order-previous-v","errorCode":null,"errorMessage":"series %v out of order; previous %v","messagePattern":"series (.+?) out of order; previous (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/block/index.go","lineNumber":286,"sourceCode":"\t\tprevLset.CopyFrom(lset)\n\n\t\tid := p.At()\n\t\tif prevId != 0 {\n\t\t\t// Approximate size.\n\t\t\tseriesSize.Add(int64(id-prevId) * int64(offsetMultiplier))\n\t\t}\n\t\tprevId = id\n\t\tstats.TotalSeries++\n\n\t\tif err := r.Series(id, &builder, &chks); err != nil {\n\t\t\treturn stats, errors.Wrap(err, \"read series\")\n\t\t}\n\t\tlset = builder.Labels()\n\t\tif lset.IsEmpty() {\n\t\t\treturn stats, errors.Errorf(\"empty label set detected for series %d\", id)\n\t\t}\n\t\tif !prevLset.IsEmpty() && labels.Compare(prevLset, lset) >= 0 {\n\t\t\treturn stats, errors.Errorf(\"series %v out of order; previous %v\", lset, prevLset)\n\t\t}\n\t\tvar l0 *labels.Label\n\t\tlset.Range(func(l labels.Label) {\n\t\t\tif l0 != nil {\n\t\t\t\tif l.Name < l0.Name {\n\t\t\t\t\tstats.OutOfOrderLabels++\n\t\t\t\t\tlevel.Warn(logger).Log(\"msg\",\n\t\t\t\t\t\t\"out-of-order label set: known bug in Prometheus 2.8.0 and below\",\n\t\t\t\t\t\t\"labelset\", lset.String(),\n\t\t\t\t\t\t\"series\", fmt.Sprintf(\"%d\", id),\n\t\t\t\t\t)\n\t\t\t\t}\n\t\t\t}\n\t\t\tl0 = &l\n\t\t})\n\n\t\tif len(chks) == 0 {\n\t\t\treturn stats, errors.Errorf(\"empty chunks for series %d\", id)","sourceCodeStart":268,"sourceCodeEnd":304,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/index.go#L268-L304","documentation":"The index requires series to be stored in sorted label-set order (that is how postings lookups work). GatherIndexHealthStats compares each series' label set with the previous one and errors with \"series %v out of order; previous %v\" when the order regresses. This indicates a corrupt or improperly written index, not a data problem in the chunks.","triggerScenarios":"GatherIndexHealthStats iterates postings and encounters series whose labels compare >= 0 against the previous series — from index corruption or indexes written by versions with sorting bugs (e.g. the tsdb issue fixed around symbols/sort ordering).","commonSituations":"Old blocks written by buggy Prometheus/Thanos versions, corrupted uploads, verification after migrating TSDB versions, blocks assembled by external tooling.","solutions":["Rebuild the block index with offline compaction (tsdb tooling) to restore sorted order","Repair or drop the block via block.Repair for known historical issues","Check Prometheus/Thanos versions that wrote the block and upgrade before re-writing data","Restore the block from a known-good backup"],"exampleFix":"// before\nstats, err := block.GatherIndexHealthStats(ctx, bdir) // series {foo=\"bar\"} out of order; previous {foo=\"baz\"}\n// after\n// rebuild index via offline compaction\nb, err := tsdb.OpenBlock(logger, bdir, nil, nil)\nif err != nil { return err }\n_, err = tsdb.CompactBlock(logger, bdir, b) // rewrites sorted index","handlingStrategy":"validation","validationCode":"meta, err := metadata.ReadFromDir(bdir)\nif err != nil { return err }\nif meta.PrometheusVersion() == \"\" || versionKnownBuggy(meta) {\n    return fmt.Errorf(\"block %s written by suspect version %s\", bdir, meta.Version)\n}","typeGuard":null,"tryCatchPattern":"stats, err := block.GatherIndexHealthStats(ctx, bdir)\nif err != nil && strings.Contains(err.Error(), \"out of order\") {\n    return rebuildIndexViaCompaction(logger, bdir)\n}","preventionTips":["Upgrade Prometheus/Thanos before writing new blocks so indexes use fixed sorting","Re-compact old blocks migrated across versions","Verify index ordering (VerifyIndex) after migrations","Reject blocks from untrusted/external writers"],"tags":["corruption","index","ordering"],"backgroundTag":"internal-invariant-violation","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}