thanos-io/thanos · error

Internal

Internal

Error message

chunk iter

What it means

After streaming series/chunks to the client, the Series handler checks the chunk querier iterator's error and, if non-nil, wraps it with 'chunk iter' and returns an Internal gRPC status. It signals the underlying TSDB chunk query failed mid-iteration, not a client mistake.

Solutions

1. Inspect the wrapped cause in the Internal status for the root error
2. Verify block integrity (thanos tools bucket verify) and re-download/repair corrupt blocks
3. Check disk health and free space on the store node
4. Retry the query; if persistent, remove or repair the affected block

if status.Code(err) == codes.Internal && strings.Contains(err.Error(), "chunk iter") { log.Error(err); /* inspect wrapped cause, repair blocks or retry */ }

When it happens

Trigger: The ChunkQuerier's chunk iterator (chIter.Err() != nil) errors during Series — e.g. corrupt index blocks, failed block reads, memory pressure closing the querier, or head compaction during the query.

Common situations: Corrupted or partially downloaded TSDB blocks in a store node, disk I/O failures, querying while compaction/deletion runs, or out-of-memory eviction killing iterators.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• retrieving targets
• error retrieving rules
• retrieving exemplars
• Aborted
• res.GetWarning()

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/c8f4015e777e7ea2. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/tsdb.go:375

			frameBytesLeft -= c.Size()
			seriesChunks = append(seriesChunks, c)

			// We are fine with minor inaccuracy of max bytes per frame. The inaccuracy will be max of full chunk size.
			isNext = chIter.Next()
			if frameBytesLeft > 0 && isNext {
				continue
			}
			if err := srv.Send(storepb.NewSeriesResponse(&storepb.Series{Labels: storeSeries.Labels, Chunks: seriesChunks})); err != nil {
				return status.Error(codes.Aborted, err.Error())
			}

			if isNext {
				frameBytesLeft = bytesLeftForChunks
				seriesChunks = make([]storepb.AggrChunk, 0, len(seriesChunks))
			}
		}
		if err := chIter.Err(); err != nil {
			return status.Error(codes.Internal, errors.Wrap(err, "chunk iter").Error())
		}

	}
	if err := set.Err(); err != nil {
		return status.Error(codes.Internal, err.Error())
	}
	for _, w := range set.Warnings() {
		if err := srv.Send(storepb.NewWarnSeriesResponse(w)); err != nil {
			return status.Error(codes.Aborted, err.Error())
		}
	}
	return srv.Flush()
}

// LabelNames returns all known label names constrained with the given matchers.
func (s *TSDBStore) LabelNames(ctx context.Context, r *storepb.LabelNamesRequest) (
	*storepb.LabelNamesResponse, error,
) {

View on GitHub (pinned to 35b8b99117)