hashicorp/nomad · warning

failed to decode Docker response: %w

Error message

failed to decode Docker response: %w

What it means

After obtaining the stats stream, collectDockerStats JSON-decodes the response body into a containerapi.StatsResponse. A decode error other than io.EOF is wrapped as 'failed to decode Docker response', meaning the daemon returned truncated, non-JSON, or corrupted stats data.

Source

Thrown at drivers/docker/stats.go:152

	var stats *containerapi.StatsResponse

	statsReader, err := h.dockerClient.ContainerStats(ctx, h.containerID, mclient.ContainerStatsOptions{
		IncludePreviousSample: true,
	})
	if err != nil && err != io.EOF {
		return nil, fmt.Errorf("failed to collect stats: %w", err)
	}

	// Ensure the body is not nil to avoid potential panics. The statsReader
	// itself cannot be nil, so there is no need to check this.
	if statsReader.Body == nil {
		return nil, errors.New("error decoding stats data: no reader body")
	}

	err = json.NewDecoder(statsReader.Body).Decode(&stats)
	_ = statsReader.Body.Close()
	if err != nil && err != io.EOF {
		return nil, fmt.Errorf("failed to decode Docker response: %w", err)
	}

	if stats == nil {
		return nil, errors.New("error decoding stats data: stats were nil")
	}

	return stats, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check for proxies/middlewares between Nomad and the Docker daemon that could alter the stats response
2. Verify Docker daemon version compatibility with the Nomad Docker API client
3. Retry — collectStats retries transient decode failures; check daemon logs for stream resets
4. Confirm docker stats works directly on the host to isolate the daemon vs client

// sanity-check daemon stats endpoint before trusting the pipeline
body, err := cli.ContainerStats(ctx, containerID, false)
if err == nil {
    var probe map[string]json.RawMessage
    if json.NewDecoder(body).Decode(&probe) != nil {
        log.Warn("docker daemon returning non-JSON stats payload; check proxies")
    }
}

func isStatsDecodeErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "failed to decode Docker response")
}

usage, err := handle.Stats(ctx, interval, compute)
if isStatsDecodeErr(err) {
    log.Warn("corrupt stats stream; will rely on collector retry", "err", err)
}

When it happens

Trigger: json.NewDecoder(statsReader.Body).Decode(&stats) fails with non-EOF: the daemon closed the stream mid-frame, returned an error/HTML page instead of JSON (proxy in front of Docker), or the stats response is truncated.

Common situations: Reverse proxies or load balancers in front of a remote Docker daemon rewriting responses; Docker daemon version mismatch producing unexpected payload; connection dropped mid-stream under heavy load.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• error decoding stats data: stats were nil
• error decoding stats data: no reader body
• failed to collect stats: %w
• container stopped
• DriverStatsNotImplemented

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e3a2dca174c92adf. Report an issue: GitHub.