openfaas/faas · error

string(upstreamBody)

Error message

string(upstreamBody)

What it means

AddMetricsHandler wraps the internal list-functions handler (it serves GET /system/functions when a prometheus_query is configured) and runs it in-process with an httptest.Recorder before enriching the result with invocation metrics. If the wrapped handler returns any status other than 200, the gateway forwards the upstream body and status code verbatim — the message you see is the provider's own error payload, not a fixed gateway string.

Source

Thrown at gateway/metrics/add_metrics.go:37

	return func(w http.ResponseWriter, r *http.Request) {

		recorder := httptest.NewRecorder()
		handler.ServeHTTP(recorder, r)
		upstreamCall := recorder.Result()

		if upstreamCall.Body == nil {
			log.Println("Upstream call had empty body.")
			return
		}

		defer upstreamCall.Body.Close()
		upstreamBody, _ := io.ReadAll(upstreamCall.Body)

		if recorder.Code != http.StatusOK {
			log.Printf("List functions responded with code %d, body: %s",
				recorder.Code,
				string(upstreamBody))
			http.Error(w, string(upstreamBody), recorder.Code)
			return
		}

		var functions []types.FunctionStatus

		err := json.Unmarshal(upstreamBody, &functions)
		if err != nil {
			log.Printf("Metrics upstream error: %s, value: %s", err, string(upstreamBody))

			http.Error(w, "Unable to parse list of functions from provider", http.StatusInternalServerError)
			return
		}

		// Ensure values are empty first.
		for i := range functions {
			functions[i].InvocationCount = 0
		}

View on GitHub (pinned to 8d803bf9e2)

Solutions

1. Read the forwarded body — it contains the provider's error message, often the Kubernetes error text
2. Check the provider's own logs (kubectl logs on the faas-netes deployment, or the faasd journal)
3. If the body mentions RBAC/forbidden, fix the gateway service account roles
4. Curl the provider's /system/functions directly to confirm the failure exists outside the metrics wrapper
5. If the Kubernetes API was flapping, retry once it recovers

resp, err := client.Get(gateway + "/system/functions")
if err != nil || resp.StatusCode >= 500 {
    // provider-side listing failure: read the forwarded provider error body,
    // back off and retry; fix RBAC/config if the body says forbidden
    time.Sleep(backoff)
    continue
}

When it happens

Trigger: GET /system/functions while the provider fails to list functions: faas-netes returning 500 on a Kubernetes API error, 403 from missing RBAC permissions on the gateway's service account, or faasd failing to read its container backend.

Common situations: Gateway service account losing list-deployments permissions after a chart upgrade; Kubernetes API unreachable or rate-limiting; provider crash-looping; stale faasd credentials.

Related errors

• log request failed
• unknown log request error (%v)
• A body is required for this endpoint
• err.Error()
• Error queuing request: %s

AI-assisted analysis of openfaas/faas@8d803bf9e2 (2026-08-16). Data as JSON: /api/errors/7353b90d1d4e72c6. Report an issue: GitHub.