{"record":{"id":"90da089ddf42708e","repo":"grafana/grafana","slug":"watch-request-failed-with-status-response-status","errorCode":null,"errorMessage":"Watch request failed with status ${response.status}: ${response.statusText}","messagePattern":"Watch request failed with status (.+?): (.+?)","errorType":"http","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"public/app/features/apiserver/client.ts","lineNumber":89,"sourceCode":"          map((event) => event.message),\n          catchError((error) => {\n            console.warn('Live channel watch failed, falling back to polling:', error);\n            return this.createPollingFallback(params, error);\n          })\n        );\n    }\n\n    const decoder = new TextDecoder();\n    return getBackendSrv()\n      .chunked({\n        url: this.url,\n        params: requestParams,\n        method: 'GET',\n      })\n      .pipe(\n        map((response) => {\n          if (!response.ok) {\n            throw new Error(`Watch request failed with status ${response.status}: ${response.statusText}`);\n          }\n          return response;\n        }),\n        filter((response) => response.data instanceof Uint8Array),\n        map((response) => {\n          const text = decoder.decode(response.data);\n          return text.split('\\n');\n        }),\n        mergeMap((text) => from(text)),\n        filter((line) => line.length > 0),\n        map((line) => {\n          try {\n            return JSON.parse(line);\n          } catch (e) {\n            console.warn('Invalid JSON in watch stream:', e, line);\n            return null;\n          }\n        }),","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/grafana/grafana/blob/ae3104e3690e28e72c8f0c97efc12647a70f75c3/public/app/features/apiserver/client.ts#L71-L107","documentation":"Thrown inside the apiserver ResourceClient watch pipeline when a streamed HTTP chunk arrives with response.ok === false. The watch endpoint streams Kubernetes-style watch events over chunked transfer; a non-ok status (5xx, 4xx, gateway errors) means the watch cannot continue, so the operator throws inside the RxJS map and the stream errors out (it has a retry({count:3,delay:1000}) upstream and a final catchError).","triggerScenarios":"Watching a k8s/apiserver resource (e.g. via /api/apiserver/... watch) when the backend returns a non-2xx chunk: server error, auth failure mid-stream, gateway timeout, or the apiserver is unreachable.","commonSituations":"Grafana backend restarted mid-watch; session expired during a long watch; proxy/load-balancer cut the chunked connection with a 502/504; the watched resource type was removed in a backend upgrade; rate limiting returns 429.","solutions":["Let the existing retry({count:3,delay:1000}) handle transient failures; for persistent ones, re-authenticate / reload.","Check backend health and the watched resource's GVR still exists on this Grafana version.","Verify auth/session is still valid (re-login if expired).","If behind a proxy, ensure it permits long-lived chunked responses and does not buffer/timeout them."],"exampleFix":"// before\nclient.watch(params).subscribe((e) => handle(e));\n// after\nclient.watch(params).subscribe({\n  next: (e) => handle(e),\n  error: (err) => {\n    console.error('watch failed', err);\n    scheduleReconnect();\n  },\n});","handlingStrategy":"retry","validationCode":"const healthy = await getBackendSrv().get('/api/health').catch(() => null);\nif (!healthy) {\n  scheduleReconnect();\n  return;\n}","typeGuard":"const isRecoverableStatus = (status: number): boolean =>\n  status === 408 || status === 425 || status === 429 || status >= 500;","tryCatchPattern":"client.watch(params).subscribe({\n  next: (e) => handle(e),\n  error: (err) => {\n    if (/Watch request failed/.test(String(err))) {\n      scheduleReconnect();\n    } else { throw err; }\n  },\n});","preventionTips":["Rely on the built-in retry({count:3,delay:1000}) for transient failures.","Re-authenticate when sessions expire mid-watch.","Ensure proxies permit long-lived chunked responses without buffering."],"tags":["apiserver","watch","streaming","network","kubernetes"],"backgroundTag":null,"analyzedSha":"ae3104e3690e28e72c8f0c97efc12647a70f75c3","analyzedAt":"2026-08-12T12:48:38.752Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}