{"record":{"id":"e3a2dca174c92adf","repo":"hashicorp/nomad","slug":"failed-to-decode-docker-response-w","errorCode":null,"errorMessage":"failed to decode Docker response: %w","messagePattern":"failed to decode Docker response: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"drivers/docker/stats.go","lineNumber":152,"sourceCode":"\tvar stats *containerapi.StatsResponse\n\n\tstatsReader, err := h.dockerClient.ContainerStats(ctx, h.containerID, mclient.ContainerStatsOptions{\n\t\tIncludePreviousSample: true,\n\t})\n\tif err != nil && err != io.EOF {\n\t\treturn nil, fmt.Errorf(\"failed to collect stats: %w\", err)\n\t}\n\n\t// Ensure the body is not nil to avoid potential panics. The statsReader\n\t// itself cannot be nil, so there is no need to check this.\n\tif statsReader.Body == nil {\n\t\treturn nil, errors.New(\"error decoding stats data: no reader body\")\n\t}\n\n\terr = json.NewDecoder(statsReader.Body).Decode(&stats)\n\t_ = statsReader.Body.Close()\n\tif err != nil && err != io.EOF {\n\t\treturn nil, fmt.Errorf(\"failed to decode Docker response: %w\", err)\n\t}\n\n\tif stats == nil {\n\t\treturn nil, errors.New(\"error decoding stats data: stats were nil\")\n\t}\n\n\treturn stats, nil\n}\n","sourceCodeStart":134,"sourceCodeEnd":161,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/stats.go#L134-L161","documentation":"After obtaining the stats stream, collectDockerStats JSON-decodes the response body into a containerapi.StatsResponse. A decode error other than io.EOF is wrapped as 'failed to decode Docker response', meaning the daemon returned truncated, non-JSON, or corrupted stats data.","triggerScenarios":"json.NewDecoder(statsReader.Body).Decode(&stats) fails with non-EOF: the daemon closed the stream mid-frame, returned an error/HTML page instead of JSON (proxy in front of Docker), or the stats response is truncated.","commonSituations":"Reverse proxies or load balancers in front of a remote Docker daemon rewriting responses; Docker daemon version mismatch producing unexpected payload; connection dropped mid-stream under heavy load.","solutions":["Check for proxies/middlewares between Nomad and the Docker daemon that could alter the stats response","Verify Docker daemon version compatibility with the Nomad Docker API client","Retry — collectStats retries transient decode failures; check daemon logs for stream resets","Confirm docker stats works directly on the host to isolate the daemon vs client"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// sanity-check daemon stats endpoint before trusting the pipeline\nbody, err := cli.ContainerStats(ctx, containerID, false)\nif err == nil {\n    var probe map[string]json.RawMessage\n    if json.NewDecoder(body).Decode(&probe) != nil {\n        log.Warn(\"docker daemon returning non-JSON stats payload; check proxies\")\n    }\n}","typeGuard":"func isStatsDecodeErr(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"failed to decode Docker response\")\n}","tryCatchPattern":"usage, err := handle.Stats(ctx, interval, compute)\nif isStatsDecodeErr(err) {\n    log.Warn(\"corrupt stats stream; will rely on collector retry\", \"err\", err)\n}","preventionTips":["Remove or configure proxies between Nomad and the Docker daemon","Keep Docker engine versions within Nomad's supported range","Watch for daemon OOM/network resets during load spikes","Verify docker stats output directly on the host when in doubt"],"tags":["docker","stats","json-decode"],"backgroundTag":"docker-stats-stream-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}