hashicorp/nomad · warning

failed to collect stats: %w

Error message

failed to collect stats: %w

What it means

collectDockerStats opens the Docker stats stream with ContainerStats(IncludePreviousSample:true). Any error from that call other than io.EOF (which signals the container stopped) is wrapped as 'failed to collect stats'. This means the Docker API refused or failed the stats request, not that the data was malformed.

Source

Thrown at drivers/docker/stats.go:140

				h.logger.Error("error collecting stats from container", "error", err)
				ticker.Reset(helper.Backoff(statsCollectorBackoffBaseline, statsCollectorBackoffLimit, retry))
				retry++
			}
		}
	}
}

// collectDockerStats performs the stats collection from the Docker API. It is
// split into its own function for the purpose of aiding testing.
func (h *taskHandle) collectDockerStats(ctx context.Context) (*containerapi.StatsResponse, error) {

	var stats *containerapi.StatsResponse

	statsReader, err := h.dockerClient.ContainerStats(ctx, h.containerID, mclient.ContainerStatsOptions{
		IncludePreviousSample: true,
	})
	if err != nil && err != io.EOF {
		return nil, fmt.Errorf("failed to collect stats: %w", err)
	}

	// Ensure the body is not nil to avoid potential panics. The statsReader
	// itself cannot be nil, so there is no need to check this.
	if statsReader.Body == nil {
		return nil, errors.New("error decoding stats data: no reader body")
	}

	err = json.NewDecoder(statsReader.Body).Decode(&stats)
	_ = statsReader.Body.Close()
	if err != nil && err != io.EOF {
		return nil, fmt.Errorf("failed to decode Docker response: %w", err)
	}

	if stats == nil {
		return nil, errors.New("error decoding stats data: stats were nil")
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the container is still running (docker ps) at the time of failure
2. Retry once the container is confirmed healthy — collectStats already applies a retry/backoff loop around this
3. Check Docker daemon health and connectivity (docker info, daemon logs)
4. Reduce stats collection frequency if daemon overload causes stream failures

// verify container is live before requesting stats
state, err := cli.ContainerInspect(ctx, containerID)
if err != nil || !state.State.Running {
    return errors.New("container not running; stats unavailable")
}

func isStatsCollectErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "failed to collect stats")
}

usage, err := handle.Stats(ctx, interval, compute)
if isStatsCollectErr(err) {
    log.Warn("stats collection failed transiently; relying on retry loop", "err", err)
}

When it happens

Trigger: dockerClient.ContainerStats returns a non-EOF error: container no longer exists (404), daemon connection broken, stats stream connection reset, or context cancelled by shutdown while the call is in flight.

Common situations: Container exited or was removed while stats collection continued; Docker daemon restart; network partition to a remote daemon; task finishing during heavy telemetry sampling intervals.

Related errors

• error decoding stats data: no reader body
• error decoding stats data: stats were nil
• failed to decode Docker response: %w
• container stopped
• DriverStatsNotImplemented

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d38192a32021830c. Report an issue: GitHub.