{"record":{"id":"ddb7027f72df73e8","repo":"thanos-io/thanos","slug":"empty-label-set-detected-for-series-d","errorCode":null,"errorMessage":"empty label set detected for series %d","messagePattern":"empty label set detected for series (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/block/index.go","lineNumber":283,"sourceCode":"\t// Per series.\n\tvar prevId storage.SeriesRef\n\tfor p.Next() {\n\t\tprevLset.CopyFrom(lset)\n\n\t\tid := p.At()\n\t\tif prevId != 0 {\n\t\t\t// Approximate size.\n\t\t\tseriesSize.Add(int64(id-prevId) * int64(offsetMultiplier))\n\t\t}\n\t\tprevId = id\n\t\tstats.TotalSeries++\n\n\t\tif err := r.Series(id, &builder, &chks); err != nil {\n\t\t\treturn stats, errors.Wrap(err, \"read series\")\n\t\t}\n\t\tlset = builder.Labels()\n\t\tif lset.IsEmpty() {\n\t\t\treturn stats, errors.Errorf(\"empty label set detected for series %d\", id)\n\t\t}\n\t\tif !prevLset.IsEmpty() && labels.Compare(prevLset, lset) >= 0 {\n\t\t\treturn stats, errors.Errorf(\"series %v out of order; previous %v\", lset, prevLset)\n\t\t}\n\t\tvar l0 *labels.Label\n\t\tlset.Range(func(l labels.Label) {\n\t\t\tif l0 != nil {\n\t\t\t\tif l.Name < l0.Name {\n\t\t\t\t\tstats.OutOfOrderLabels++\n\t\t\t\t\tlevel.Warn(logger).Log(\"msg\",\n\t\t\t\t\t\t\"out-of-order label set: known bug in Prometheus 2.8.0 and below\",\n\t\t\t\t\t\t\"labelset\", lset.String(),\n\t\t\t\t\t\t\"series\", fmt.Sprintf(\"%d\", id),\n\t\t\t\t\t)\n\t\t\t\t}\n\t\t\t}\n\t\t\tl0 = &l\n\t\t})","sourceCodeStart":265,"sourceCodeEnd":301,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/index.go#L265-L301","documentation":"GatherIndexHealthStats detects a series whose decoded label set is empty (no labels at all). Since every TSDB series must have at least a __name__ label, this is a hard invariant violation indicating a corrupt or malformed index. The error message includes the offending series reference ID.","triggerScenarios":"r.Series(id, ...) decodes successfully but yields an empty label set — typically a damaged series section, mispadded offsets (v2 index 16-byte padding), or an index written by buggy code.","commonSituations":"Blocks affected by historical TSDB index bugs, corrupted uploads, blocks whose index was partially overwritten, forensic verification after storage incidents.","solutions":["Treat the block as corrupt; restore it from object storage or backup","Identify the offending series ref from the error message to confirm offset corruption","Re-run offline compaction to rebuild the index if chunk data is intact","Delete the block so it is re-fetched or tombstoned"],"exampleFix":"// before\nstats, err := block.GatherIndexHealthStats(ctx, bdir) // empty label set detected for series 12345\n// after\nmeta, err := metadata.ReadFromDir(bdir)\nif err != nil { return err }\nif _, err := block.Repair(ctx, logger, filepath.Dir(bdir), meta.ULID, meta.Thanos.Source); err != nil {\n    return fmt.Errorf(\"block %s unusable, remove manually: %w\", meta.ULID, err)\n}","handlingStrategy":"validation","validationCode":"if err := block.VerifyIndex(ctx, bdir, 1); err != nil {\n    return fmt.Errorf(\"block %s failed index verification: %w\", bdir, err)\n}","typeGuard":null,"tryCatchPattern":"stats, err := block.GatherIndexHealthStats(ctx, bdir)\nvar emptyLsetErr *errors.errorString\nif err != nil && strings.Contains(err.Error(), \"empty label set detected\") {\n    return block.Repair(ctx, logger, filepath.Dir(bdir), id, source, block.IgnoreIssue347OutsideChunks)\n}","preventionTips":["Check block health before repair/downsampling operations","Restore from verified backups when index invariants fail","Track known historical TSDB index bugs for blocks written by old versions","Tombstone irreparable blocks quickly to keep store-gateway healthy"],"tags":["corruption","index","invariant"],"backgroundTag":"internal-invariant-violation","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}