{"record":{"id":"dbf02f4a265f6e6b","repo":"thanos-io/thanos","slug":"target-not-ready","errorCode":null,"errorMessage":"target not ready","messagePattern":"target not ready","errorType":"error_code","errorClass":null,"httpStatus":503,"severity":"warning","filePath":"pkg/receive/handler.go","lineNumber":92,"sourceCode":"\tLimitStatsQueryParam = \"limit\"\n\t// Labels for metrics.\n\tlabelSuccess = \"success\"\n\tlabelError   = \"error\"\n)\n\ntype ReplicationProtocol string\n\nconst (\n\tProtobufReplication  ReplicationProtocol = \"protobuf\"\n\tCapNProtoReplication ReplicationProtocol = \"capnproto\"\n)\n\nvar (\n\t// errConflict is returned whenever an operation fails due to any conflict-type error.\n\terrConflict = errors.New(\"conflict\")\n\n\terrBadReplica  = errors.New(\"request replica exceeds receiver replication factor\")\n\terrNotReady    = errors.New(\"target not ready\")\n\terrUnavailable = errors.New(\"target not available\")\n\n\terrValidation = errors.New(\"validation error\")\n)\n\ntype WriteableStoreAsyncClient interface {\n\tstorepb.WriteableStoreClient\n\tRemoteWriteAsync(context.Context, *storepb.WriteRequest, endpointReplica, []int, chan writeResponse, func(error))\n\t// TryRemoteWriteAsync submits the request without blocking. Returns false if the peer's\n\t// worker pool is at capacity; the caller should fall back to RemoteWriteAsync.\n\tTryRemoteWriteAsync(context.Context, *storepb.WriteRequest, endpointReplica, []int, chan writeResponse, func(error)) bool\n}\n\n// Options for the web Handler.\ntype Options struct {\n\tWriter                  *Writer\n\tListenAddress           string\n\tRegistry                *prometheus.Registry","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L74-L110","documentation":"errNotReady is the sentinel returned by Write and receiveOTLPHTTP when the target receiver is not ready to accept writes (e.g. it is still starting up or refreshing its hashring). The HTTP handler maps it to http.StatusServiceUnavailable (503) and capnp maps it to WriteError_unavailable. Clients are expected to retry, ideally honoring Retry-After once supported.","triggerScenarios":"Write or receiveOTLPHTTP forwards a write to a target whose receiver responds/behaves as not ready — the root cause of the returned error is errNotReady per errors.Cause.","commonSituations":"Writes hitting receivers during startup before the hashring is loaded, a receiver that hasn't joined the ring yet during scaling events, load balancers routing to pods not yet fully initialized.","solutions":["Retry the write with backoff; the condition is transient by design","Wait for the receiver to report ready before routing traffic (readiness probes / load balancer health checks)","Check why the target is not ready: hashring file load failures or slow startup in its logs","Harden the sender with bounded retries on 503 so temporary unavailability doesn't drop samples"],"exampleFix":"// before\nerr := r.Write(ctx, req)\n// after\nerr := r.Write(ctx, req)\nif err != nil && errors.Is(errors.Cause(err), errNotReady) {\n    time.Sleep(backoff); goto retry // bounded retry loop\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":"func isNotReady(err error) bool {\n    return errors.Is(errors.Cause(err), errNotReady)\n}","tryCatchPattern":"err := r.Write(ctx, req)\nif err != nil && errors.Is(errors.Cause(err), errNotReady) {\n    select {\n    case <-time.After(backoff):\n        // retry with jitter, bounded attempts\n    case <-ctx.Done():\n        return ctx.Err()\n    }\n}","preventionTips":["Gate traffic on receiver readiness probes before routing writes","Wait for hashring loading to complete at startup before accepting writes","Use bounded exponential backoff with jitter for transient unavailability","Scale receivers ahead of traffic surges so they are ready when needed"],"tags":["availability","retry","replication"],"backgroundTag":"request-timeout","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}