thanos-io/thanos · warning

NOT OK

Error message

NOT OK

What it means

The HTTP probe handler writes the literal body "NOT OK" with HTTP 503 Service Unavailable when the readiness check function returns false. It is not a Go error; it is the readiness endpoint's failure response, meaning the process (e.g. Thanos component) is not ready to serve traffic.

Solutions

1. Wait for the component to complete startup; check startup logs for readiness progress
2. Increase the readiness probe's initialDelaySeconds/failureThreshold in Kubernetes
3. If permanently not-ready, investigate the blocking dependency (storage, Prometheus, peers)
4. Use the /-/healthy liveness endpoint to distinguish liveness from readiness

Example fix

// kubernetes probe tuning
// before
readinessProbe: { httpGet: { path: /-/ready }, failureThreshold: 3 }
// after
readinessProbe: { httpGet: { path: /-/ready }, initialDelaySeconds: 30, periodSeconds: 10, failureThreshold: 12 }

// orchestrator-side: tolerate transient 503 from /-/ready
if resp.StatusCode == http.StatusServiceUnavailable {
    time.Sleep(10 * time.Second)
    // re-probe; only alert after sustained failures
}

When it happens

Trigger: GET to the /-/ready (probe) endpoint while p.IsReady returns false — the component has not finished startup or has been marked not ready.

Common situations: Kubernetes readiness probes failing during slow startup (index loading, store sync); orchestrator rerouting traffic away from a warming-up instance; monitoring alerting on 503s from /-/ready.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• creating request to downstream URL
• Not ready
• Not ready
• not ready
• error starting web server

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/a65aa2406dad7261. Report an issue: GitHub.

Appendix: source

Thrown at pkg/prober/http.go:41

// NewHTTP returns HTTPProbe representing readiness and healthiness of given component.
func NewHTTP() *HTTPProbe {
	return &HTTPProbe{}
}

// HealthyHandler returns a HTTP Handler which responds health checks.
func (p *HTTPProbe) HealthyHandler(logger log.Logger) http.HandlerFunc {
	return p.handler(logger, p.isHealthy)
}

// ReadyHandler returns a HTTP Handler which responds readiness checks.
func (p *HTTPProbe) ReadyHandler(logger log.Logger) http.HandlerFunc {
	return p.handler(logger, p.IsReady)
}

func (p *HTTPProbe) handler(logger log.Logger, c check) http.HandlerFunc {
	return func(w http.ResponseWriter, _ *http.Request) {
		if !c() {
			http.Error(w, "NOT OK", http.StatusServiceUnavailable)
			return
		}
		if _, err := io.WriteString(w, "OK"); err != nil {
			level.Error(logger).Log("msg", "failed to write probe response", "err", err)
		}
	}
}

// IsReady returns true if component is ready.
func (p *HTTPProbe) IsReady() bool {
	ready := p.ready.Load()
	return ready > 0
}

// isHealthy returns true if component is healthy.
func (p *HTTPProbe) isHealthy() bool {
	healthy := p.healthy.Load()
	return healthy > 0

View on GitHub (pinned to 35b8b99117)