hashicorp/nomad · error

unable to unmarshal response with status %d: %v

Error message

unable to unmarshal response with status %d: %v

What it means

In Nomad's Go API client (api/agent.go:285), Agent.Health decodes the HTTP response body into a health struct; when json decoding fails, it returns this error embedding the HTTP status code and the underlying decode error. It means the server responded, but the body was not the expected JSON (or the stream was empty/truncated). It is a client-side wrap of a JSON decode failure, not a transport error.

Source

Thrown at api/agent.go:285

	if err != nil {
		return nil, err
	}

	var health AgentHealthResponse
	_, resp, err := a.client.doRequest(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()

	// Always try to decode the response as JSON
	err = json.NewDecoder(resp.Body).Decode(&health)
	if err == nil {
		return &health, nil
	}

	// Return custom error when response is not expected JSON format
	return nil, fmt.Errorf("unable to unmarshal response with status %d: %v", resp.StatusCode, err)
}

// Host returns debugging context about the agent's host operating system
func (a *Agent) Host(serverID, nodeID string, q *QueryOptions) (*HostDataResponse, error) {
	if q == nil {
		q = &QueryOptions{}
	}
	if q.Params == nil {
		q.Params = make(map[string]string)
	}

	if serverID != "" {
		q.Params["server_id"] = serverID
	}

	if nodeID != "" {
		q.Params["node_id"] = nodeID
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check resp.StatusCode embedded in the message; if it is not 200, fix the proxy/routing so requests actually reach the Nomad agent
2. curl the Health endpoint directly (e.g. curl -v $NOMAD_ADDR/v1/agent/health) and inspect the raw body to see what is actually returned
3. Verify NOMAD_ADDR points at a Nomad agent HTTP port and not a UI or unrelated service
4. If behind a load balancer, configure it to pass through Nomad endpoints rather than serving HTML error pages
5. Retry on transient truncation; investigate the wrapped %v decode error for the exact JSON offset/problem

Example fix

// before: transparent proxy returns HTML 502, decode fails
health, err := agent.Health("", "")
// after: check status via a raw query first and fail fast with a clearer message
resp, err := a.client.rawQuery("/v1/agent/health", nil, nil)
if err != nil { return nil, err }
if resp.StatusCode != http.StatusOK {
    return nil, fmt.Errorf("agent health endpoint returned status %d; check proxy/routing", resp.StatusCode)
}

u, err := url.Parse(cfg.Address)
if err != nil || u.Host == "" { return fmt.Errorf("NOMAD_ADDR %q does not look like a Nomad server", cfg.Address) }

func isJSONContentType(h http.Header) bool {
	return strings.Contains(h.Get("Content-Type"), "application/json")
}

health, err := agent.Health("", "")
if err != nil {
	var serr *json.SyntaxError
	if errors.As(err, &serr) || strings.Contains(err.Error(), "unable to unmarshal") {
		return fmt.Errorf("non-JSON response from agent; check proxy/routing: %w", err)
	}
	return err
}

When it happens

Trigger: Calling (a *Agent).Health(serverID, region string) when the Nomad agent returns a non-JSON body: an HTML error page from a reverse proxy/load balancer, an empty body with 200, a proxy 502/503 page, or a truncated response.

Common situations: Hitting Nomad through an nginx/ALB that intercepts the request and serves its own HTML error page; pointing NOMAD_ADDR at a non-Nomad HTTP service; a proxy closing the connection mid-body; TLS-terminating proxy returning a plain-text error with 200.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• failed to marshal command: %W
• unexpected HTTP transport: %T
• invalid redirect location %q: %w
• Failed to parse X-Nomad-Index: %v
• failed to decode log endpoint response as JSON: %q

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f0199217ce2a520f. Report an issue: GitHub.