thanos-io/thanos · error
Lookup labels symbols
Error message
Lookup labels symbols
What it means
Wraps IndexReader.LookupLabelsSymbols failure in blockSeriesClient.series. After loading a series' label set in symbolized form, this resolves symbol references to actual label strings. Failure means symbol table lookups against the block index failed (fetch/decode error or out-of-range symbol reference).
Solutions
- Verify the block's index with 'thanos tools bucket verify'; remove/restore corrupted blocks.
- Check the wrapped root error for object-store connectivity/throttling and retry the query.
- Ensure blocks are not deleted while store gateways still sync/serve them (retain long enough, coordinate compactor deletion).
- Rebuild the index-header if symbol table metadata is stale.
Defensive patterns
Strategy: try-catch
Try / catch
if err := query(ctx); err != nil && strings.Contains(err.Error(), "Lookup labels symbols") {
if isNotFound(err) {
// block likely deleted mid-query: refresh store list and retry once
if retryErr := query(ctx); retryErr != nil {
return retryErr
}
return nil
}
return err
} Prevention
- Coordinate compactor deletions with store gateways (sufficient block retention/sync delay).
- Verify blocks to catch corrupted symbol tables before serving queries.
- Monitor object-store fetch errors from store gateways.
- Rebuild index-headers when symbol-table metadata is suspect.
When it happens
Trigger: Series with matched chunks (or lazy expanded posting) call indexr.LookupLabelsSymbols(ctx, symbolizedLset, b) and the symbol table lookup or fetch fails; also occurs when a series references symbols outside the symbol table (index corruption).
Common situations: Corrupted or truncated symbol tables in uploaded blocks, object-store fetch failures under load, block deleted from the bucket while a query was streaming it.
Understand the failure class
Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.
Related errors
AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07).
Data as JSON: /api/errors/63797e8a31529453.
Report an issue: GitHub.
Appendix: source
Thrown at pkg/store/bucket.go:1327
seriesMatched := 0
b.entries = b.entries[:0]
OUTER:
for i := range postingsBatch {
if err := b.ctx.Err(); err != nil {
return err
}
hasMatchedChunks, err := b.indexr.LoadSeriesForTime(postingsBatch[i], &b.symbolizedLset, &b.chkMetas, b.skipChunks, b.mint, b.maxt)
if err != nil {
return errors.Wrap(err, "read series")
}
// Skip getting series symbols if we know there is no matched chunks
// and lazy expanded posting not enabled.
if !lazyExpandedPosting && !hasMatchedChunks {
continue
}
if err := b.indexr.LookupLabelsSymbols(b.ctx, b.symbolizedLset, b.b); err != nil {
return errors.Wrap(err, "Lookup labels symbols")
}
b.lset = b.b.Labels()
for _, matcher := range b.lazyPostings.matchers {
val := b.lset.Get(matcher.Name)
if !matcher.Matches(val) {
// Series not matched means series we overfetched due to lazy posting expansion.
seriesBytes := b.indexr.loadedSeries[postingsBatch[i]]
b.lazyExpandedPostingSeriesOverfetchedSizeBytes.Add(float64(len(seriesBytes)))
continue OUTER
}
}
if lazyExpandedPosting {
b.expandedPostings = append(b.expandedPostings, postingsBatch[i])
}
// If lazy expanded postings enabled, due to expanded postings cache, we need to
// make sure we check lazy posting matchers and update expanded postings before
// going to next series.View on GitHub (pinned to 35b8b99117)