{"record":{"id":"04ca0c526e08a03a","repo":"thanos-io/thanos","slug":"target-not-available","errorCode":null,"errorMessage":"target not available","messagePattern":"target not available","errorType":"error_code","errorClass":null,"httpStatus":503,"severity":"warning","filePath":"pkg/receive/handler.go","lineNumber":93,"sourceCode":"\t// Labels for metrics.\n\tlabelSuccess = \"success\"\n\tlabelError   = \"error\"\n)\n\ntype ReplicationProtocol string\n\nconst (\n\tProtobufReplication  ReplicationProtocol = \"protobuf\"\n\tCapNProtoReplication ReplicationProtocol = \"capnproto\"\n)\n\nvar (\n\t// errConflict is returned whenever an operation fails due to any conflict-type error.\n\terrConflict = errors.New(\"conflict\")\n\n\terrBadReplica  = errors.New(\"request replica exceeds receiver replication factor\")\n\terrNotReady    = errors.New(\"target not ready\")\n\terrUnavailable = errors.New(\"target not available\")\n\n\terrValidation = errors.New(\"validation error\")\n)\n\ntype WriteableStoreAsyncClient interface {\n\tstorepb.WriteableStoreClient\n\tRemoteWriteAsync(context.Context, *storepb.WriteRequest, endpointReplica, []int, chan writeResponse, func(error))\n\t// TryRemoteWriteAsync submits the request without blocking. Returns false if the peer's\n\t// worker pool is at capacity; the caller should fall back to RemoteWriteAsync.\n\tTryRemoteWriteAsync(context.Context, *storepb.WriteRequest, endpointReplica, []int, chan writeResponse, func(error)) bool\n}\n\n// Options for the web Handler.\ntype Options struct {\n\tWriter                  *Writer\n\tListenAddress           string\n\tRegistry                *prometheus.Registry\n\tTenantHeader            string","sourceCodeStart":75,"sourceCodeEnd":111,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler.go#L75-L111","documentation":"errUnavailable is the sentinel returned by Write and receiveOTLPHTTP when the target receiver is unavailable (unreachable or not accepting connections). The HTTP handler and capnp server both map it to an unavailable status (http.StatusServiceUnavailable / WriteError_unavailable). Like errNotReady it is transient and matched via errors.Cause.","triggerScenarios":"Write or receiveOTLPHTTP attempts to replicate/forward a write to a target that is down, unreachable, or refuses the connection; the resulting error's root cause is errUnavailable.","commonSituations":"A receiver pod is crash-looping or being restarted during a deploy, network partition between receive nodes, DNS/name resolution failure for a hashring endpoint, endpoint removed from the ring but still cached.","solutions":["Verify the target endpoint is running and reachable (ping/DNS/port check on the given host:port)","Update the hashring configuration to remove dead endpoints and confirm all receivers use the same config","Retry the write; the receiver will re-route per the current hashring","Check network policies / service discovery between the receive nodes"],"exampleFix":"// before (hashring endpoint pointing at removed instance)\n- endpoints: ['thanos-receive-9.thanos-receive:10901']\n// after\n- endpoints: ['thanos-receive-0.thanos-receive:10901']","handlingStrategy":"retry","validationCode":"conn, err := net.DialTimeout(\"tcp\", targetAddr, 2*time.Second)\nif err != nil { return fmt.Errorf(\"target %s unreachable before write: %w\", targetAddr, err) }\nconn.Close()","typeGuard":"func isUnavailable(err error) bool {\n    return errors.Is(errors.Cause(err), errUnavailable)\n}","tryCatchPattern":"err := r.Write(ctx, req)\nif err != nil && errors.Is(errors.Cause(err), errUnavailable) {\n    // re-route via hashring or retry after checking target health\n    return http.StatusServiceUnavailable\n}","preventionTips":["Monitor endpoint liveness and remove dead endpoints from the hashring promptly","Deploy a service mesh/DNS with health checking between receive nodes","Confirm network policies allow receiver-to-receiver traffic on the replication port","Prune stale hashring endpoints during scaling events to avoid routing to dead nodes"],"tags":["availability","network","replication"],"backgroundTag":"connection-refused","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}