{"record":{"id":"a4f8a3b246e5b740","repo":"thanos-io/thanos","slug":"postings-offsets-for-s","errorCode":null,"errorMessage":"postings offsets for %s","messagePattern":"postings offsets for (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/store/lazy_postings.go","lineNumber":63,"sourceCode":"\tseriesMaxSize int64,\n\tseriesMatchRatio float64,\n\tpostingGroupMaxKeySeriesRatio float64,\n\tlazyExpandedPostingSizeBytes prometheus.Counter,\n\tlazyExpandedPostingGroupsByReason *prometheus.CounterVec,\n) ([]*postingGroup, bool, error) {\n\tif len(postingGroups) <= 1 {\n\t\treturn postingGroups, false, nil\n\t}\n\t// Collect posting cardinality of each posting group.\n\tfor _, pg := range postingGroups {\n\t\t// A posting group can have either add keys or remove keys but not both the same time.\n\t\tvals := pg.addKeys\n\t\tif len(pg.removeKeys) > 0 {\n\t\t\tvals = pg.removeKeys\n\t\t}\n\t\trngs, err := r.block.indexHeaderReader.PostingsOffsets(pg.name, vals...)\n\t\tif err != nil {\n\t\t\treturn nil, false, errors.Wrapf(err, \"postings offsets for %s\", pg.name)\n\t\t}\n\n\t\texistentKeys := 0\n\t\tfor _, rng := range rngs {\n\t\t\tif rng == indexheader.NotFoundRange {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tif rng.End <= rng.Start {\n\t\t\t\tlevel.Error(r.logger).Log(\"msg\", \"invalid index range, fallback to non lazy posting optimization\")\n\t\t\t\treturn postingGroups, false, nil\n\t\t\t}\n\t\t\texistentKeys++\n\t\t\t// Each range starts from the #entries field which is 4 bytes.\n\t\t\t// Need to subtract it when calculating number of postings.\n\t\t\t// https://github.com/prometheus/prometheus/blob/v2.46.0/tsdb/docs/format/index.md.\n\t\t\tpg.cardinality += (rng.End - rng.Start - 4) / 4\n\t\t}\n\t\tpg.existentKeys = existentKeys","sourceCodeStart":45,"sourceCodeEnd":81,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/lazy_postings.go#L45-L81","documentation":"optimizePostingsFetchByDownloadedBytes resolves posting list offsets via the block index header reader (PostingsOffsets) for each postings group it wants to fetch. This error wraps a failure of that index lookup for the given label name, meaning posting offsets for that name could not be read from the block's index (corrupt/partial index, IO error, or unsupported index format).","triggerScenarios":"A store-gateway lazy postings fetch (fetchLazyExpandedPostings path) where r.block.indexHeaderReader.PostingsOffsets(name, vals...) fails for a label name — typically because the block's index is truncated/corrupted, the index-header file is missing/stale relative to the index, or a decode error occurs reading the postings offset table.","commonSituations":"Blocks uploaded incompletely (missing or mismatched .idx versus index-header), object-store flakiness during index read, Thanos version reading blocks written by an incompatible Prometheus index format, corrupted local cache of block index files.","solutions":["Inspect the wrapped underlying error to distinguish decode vs IO failure.","Verify the block in object storage is complete (all index files, matching index-header version) and re-upload if truncated.","Delete local cached/extracted copies of the block so the gateway re-downloads it.","Check that Prometheus index format version is supported by your Thanos build (upgrade if needed)."],"exampleFix":"// before: stale index-header left from interrupted upload\n// fix by re-syncing/removing the bad block\nthanos tools bucket verify --objstore.config-file=bucket.yml --id=<block-id>\nthanos tools bucket mark --marker=deletion-mark.json --id=<corrupt-block-id> --objstore.config-file=bucket.yml","handlingStrategy":"retry","validationCode":"// Pre-check before issuing lazy postings query\nif err := q.BlockMeta(); err != nil {\n\treturn fmt.Errorf(\"block unreadable, skip postings optimization: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"rngs, err := r.block.indexHeaderReader.PostingsOffsets(pg.name, pg.addKeys...)\nif err != nil {\n\tif isTransient(err) { // object-store timeout / 5xx\n\t\treturn nil, false, backoff.Retry(ctx, func() error {\n\t\t\t_, err = r.block.indexHeaderReader.PostingsOffsets(pg.name, pg.addKeys...)\n\t\t\treturn err\n\t\t})\n\t}\n\tlogger.Warn(\"falling back to non-optimized postings fetch\", \"err\", err)\n}","preventionTips":["Run periodic 'thanos tools bucket verify' to catch incomplete/corrupt blocks.","Use atomic block uploads so index and index-header always match.","Configure retry/backoff on object-store clients.","Mark and ignore blocks with corrupt indexes instead of failing queries repeatedly."],"tags":["storage","index","blocks"],"backgroundTag":"resource-not-found","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}