grafana/grafana · error · Error

Watch request failed with status ${response.status}: ${respo

Error message

Watch request failed with status ${response.status}: ${response.statusText}

What it means

Thrown inside the apiserver ResourceClient watch pipeline when a streamed HTTP chunk arrives with response.ok === false. The watch endpoint streams Kubernetes-style watch events over chunked transfer; a non-ok status (5xx, 4xx, gateway errors) means the watch cannot continue, so the operator throws inside the RxJS map and the stream errors out (it has a retry({count:3,delay:1000}) upstream and a final catchError).

Source

Thrown at public/app/features/apiserver/client.ts:89

          map((event) => event.message),
          catchError((error) => {
            console.warn('Live channel watch failed, falling back to polling:', error);
            return this.createPollingFallback(params, error);
          })
        );
    }

    const decoder = new TextDecoder();
    return getBackendSrv()
      .chunked({
        url: this.url,
        params: requestParams,
        method: 'GET',
      })
      .pipe(
        map((response) => {
          if (!response.ok) {
            throw new Error(`Watch request failed with status ${response.status}: ${response.statusText}`);
          }
          return response;
        }),
        filter((response) => response.data instanceof Uint8Array),
        map((response) => {
          const text = decoder.decode(response.data);
          return text.split('\n');
        }),
        mergeMap((text) => from(text)),
        filter((line) => line.length > 0),
        map((line) => {
          try {
            return JSON.parse(line);
          } catch (e) {
            console.warn('Invalid JSON in watch stream:', e, line);
            return null;
          }
        }),

View on GitHub (pinned to ae3104e369)

Solutions

Let the existing retry({count:3,delay:1000}) handle transient failures; for persistent ones, re-authenticate / reload.
Check backend health and the watched resource's GVR still exists on this Grafana version.
Verify auth/session is still valid (re-login if expired).
If behind a proxy, ensure it permits long-lived chunked responses and does not buffer/timeout them.

Example fix

// before
client.watch(params).subscribe((e) => handle(e));
// after
client.watch(params).subscribe({
  next: (e) => handle(e),
  error: (err) => {
    console.error('watch failed', err);
    scheduleReconnect();
  },
});

Defensive patterns

Strategy: retry

Validate before calling

const healthy = await getBackendSrv().get('/api/health').catch(() => null);
if (!healthy) {
  scheduleReconnect();
  return;
}

Type guard

const isRecoverableStatus = (status: number): boolean =>
  status === 408 || status === 425 || status === 429 || status >= 500;

Try / catch

client.watch(params).subscribe({
  next: (e) => handle(e),
  error: (err) => {
    if (/Watch request failed/.test(String(err))) {
      scheduleReconnect();
    } else { throw err; }
  },
});

Prevention

Rely on the built-in retry({count:3,delay:1000}) for transient failures.
Re-authenticate when sessions expire mid-watch.
Ensure proxies permit long-lived chunked responses without buffering.

When it happens

Trigger: Watching a k8s/apiserver resource (e.g. via /api/apiserver/... watch) when the backend returns a non-2xx chunk: server error, auth failure mid-stream, gateway timeout, or the apiserver is unreachable.

Common situations: Grafana backend restarted mid-watch; session expired during a long watch; proxy/load-balancer cut the chunked connection with a 502/504; the watched resource type was removed in a backend upgrade; rate limiting returns 429.

Related errors

AI-assisted analysis of grafana/grafana@ae3104e369 (2026-08-12). Data as JSON: /api/errors/90da089ddf42708e. Report an issue: GitHub.