{"record":{"id":"d38192a32021830c","repo":"hashicorp/nomad","slug":"failed-to-collect-stats-w","errorCode":null,"errorMessage":"failed to collect stats: %w","messagePattern":"failed to collect stats: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"drivers/docker/stats.go","lineNumber":140,"sourceCode":"\t\t\t\th.logger.Error(\"error collecting stats from container\", \"error\", err)\n\t\t\t\tticker.Reset(helper.Backoff(statsCollectorBackoffBaseline, statsCollectorBackoffLimit, retry))\n\t\t\t\tretry++\n\t\t\t}\n\t\t}\n\t}\n}\n\n// collectDockerStats performs the stats collection from the Docker API. It is\n// split into its own function for the purpose of aiding testing.\nfunc (h *taskHandle) collectDockerStats(ctx context.Context) (*containerapi.StatsResponse, error) {\n\n\tvar stats *containerapi.StatsResponse\n\n\tstatsReader, err := h.dockerClient.ContainerStats(ctx, h.containerID, mclient.ContainerStatsOptions{\n\t\tIncludePreviousSample: true,\n\t})\n\tif err != nil && err != io.EOF {\n\t\treturn nil, fmt.Errorf(\"failed to collect stats: %w\", err)\n\t}\n\n\t// Ensure the body is not nil to avoid potential panics. The statsReader\n\t// itself cannot be nil, so there is no need to check this.\n\tif statsReader.Body == nil {\n\t\treturn nil, errors.New(\"error decoding stats data: no reader body\")\n\t}\n\n\terr = json.NewDecoder(statsReader.Body).Decode(&stats)\n\t_ = statsReader.Body.Close()\n\tif err != nil && err != io.EOF {\n\t\treturn nil, fmt.Errorf(\"failed to decode Docker response: %w\", err)\n\t}\n\n\tif stats == nil {\n\t\treturn nil, errors.New(\"error decoding stats data: stats were nil\")\n\t}\n","sourceCodeStart":122,"sourceCodeEnd":158,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/stats.go#L122-L158","documentation":"collectDockerStats opens the Docker stats stream with ContainerStats(IncludePreviousSample:true). Any error from that call other than io.EOF (which signals the container stopped) is wrapped as 'failed to collect stats'. This means the Docker API refused or failed the stats request, not that the data was malformed.","triggerScenarios":"dockerClient.ContainerStats returns a non-EOF error: container no longer exists (404), daemon connection broken, stats stream connection reset, or context cancelled by shutdown while the call is in flight.","commonSituations":"Container exited or was removed while stats collection continued; Docker daemon restart; network partition to a remote daemon; task finishing during heavy telemetry sampling intervals.","solutions":["Verify the container is still running (docker ps) at the time of failure","Retry once the container is confirmed healthy — collectStats already applies a retry/backoff loop around this","Check Docker daemon health and connectivity (docker info, daemon logs)","Reduce stats collection frequency if daemon overload causes stream failures"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// verify container is live before requesting stats\nstate, err := cli.ContainerInspect(ctx, containerID)\nif err != nil || !state.State.Running {\n    return errors.New(\"container not running; stats unavailable\")\n}","typeGuard":"func isStatsCollectErr(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"failed to collect stats\")\n}","tryCatchPattern":"usage, err := handle.Stats(ctx, interval, compute)\nif isStatsCollectErr(err) {\n    log.Warn(\"stats collection failed transiently; relying on retry loop\", \"err\", err)\n}","preventionTips":["Skip stats polling for terminal/exited tasks","Ensure the collector's retry/backoff loop is active","Monitor Docker daemon health to catch restarts","Use reasonable stats intervals (>=1s) to avoid daemon overload"],"tags":["docker","stats","telemetry"],"backgroundTag":"docker-stats-stream-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}