thanos-io/thanos · error

panic in server iterator

Error message

panic %v in server iterator: %s

What it means

The server-side iterator sequence (iter.Seq2) for in-process Series calls recovers from any panic raised while yielding SeriesResponses and re-panics with the original value plus a stack trace, prefixed 'panic %v in server iterator'. This converts an opaque panic into one that identifies it came from the store server's streaming iterator.

Solutions

1. Read the embedded stack trace in the panic message to find the panicking frame in the Series server path
2. Fix the underlying panic cause in the server iterator (nil guard, bounds check)
3. Update Thanos/pinnacle of the in-process store to a version with the panic fix
4. Wrap consumer-side iteration with recover only as a last-resort safety net

Example fix

// before: panics propagate cryptically
for s, err := range client.Series(ctx, req) { ... }
// after: recover and log full stack
func safeIter() { defer func(){ if r := recover(); r != nil { log.Errorf("series iter panic: %v\n%s", r, debug.Stack()) } }(); for s, err := range client.Series(ctx, req) { _ = s; _ = err } }

defer func() { if r := recover(); r != nil { log.Errorf("series iter panic: %v\n%s", r, debug.Stack()) } }()

When it happens

Trigger: Any panic inside the series streaming callback — e.g. nil pointer while building SeriesResponse, index out of range in chunk conversion, or a panic in srv.Series — while consuming the iterator returned by the in-process store client.

Common situations: Upstream storepb server implementation panics on malformed or unexpected series data; concurrent map access or nil logger in the server; a bug triggered by specific series/chunk shapes during StoreAPI iteration.

Related errors

• panicked while compacting
• symbols
• chunk iter
• iterate chunk while re-encoding
• unexpected result aggregate type

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/765f16801017bfaf. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/storepb/inprocess.go:147

}

func (r *readOnlySeriesClient) SendMsg(m interface{}) error {
	return nil
}

func (r *readOnlySeriesClient) RecvMsg(m interface{}) error {
	return io.EOF
}

func (s serverAsClient) Series(ctx context.Context, in *SeriesRequest, _ ...grpc.CallOption) (Store_SeriesClient, error) {
	if s.readOnly.Load() {
		return &readOnlySeriesClient{ctx: ctx}, nil
	}
	var srvIter iter.Seq2[*SeriesResponse, error] = func(yield func(*SeriesResponse, error) bool) {
		defer func() {
			if r := recover(); r != nil {
				st := debug.Stack()
				panic(fmt.Sprintf("panic %v in server iterator: %s", r, st))
			}
		}()
		srv := newInProcessServer(ctx, yield)
		err := s.srv.Series(in, srv)
		if err != nil {
			yield(nil, err)
			return
		}
	}

	clientIter, stop := iter.Pull2(srvIter)
	return newInProcessClient(ctx, clientIter, stop), nil
}

View on GitHub (pinned to 35b8b99117)