hashicorp/nomad · info · RecoverableError

container stopped

Error message

container stopped

What it means

Stats() is the entry point the task runner uses to begin streaming resource usage. It first checks the task handle's doneCh; if the container run loop already finished (doneCh closed), it returns a non-recoverable structured error 'container stopped' instead of starting a pointless collector.

Source

Thrown at drivers/docker/stats.go:85

// close resource usage. Any further sends will be dropped.
func (u *usageSender) close() {
	u.mu.Lock()
	defer u.mu.Unlock()
	if u.closed {
		// already closed
		return
	}

	u.closed = true
	close(u.destCh)
}

// Stats starts collecting stats from the docker daemon and sends them on the
// returned channel.
func (h *taskHandle) Stats(ctx context.Context, interval time.Duration, compute cpustats.Compute) (<-chan *cstructs.TaskResourceUsage, error) {
	select {
	case <-h.doneCh:
		return nil, nstructs.NewRecoverableError(fmt.Errorf("container stopped"), false)
	default:
	}

	destCh, recvCh := newStatsChanPipe()
	go h.collectStats(ctx, destCh, interval, compute)
	return recvCh, nil
}

// collectStats starts collecting resource usage stats of a Docker container
// and does this until the context or the tasks handler done channel is closed.
func (h *taskHandle) collectStats(ctx context.Context, destCh *usageSender, interval time.Duration, compute cpustats.Compute) {
	defer destCh.close()

	// retry tracks the number of retries the collection has been through since
	// the last successful Docker API call. This is used to calculate the
	// backoff time for the collection ticker.
	var retry uint64

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Treat this as expected lifecycle behavior: the structured error's Recoverable flag is false, so stop polling and deregister stats for the finished task
2. Check task state before requesting stats; skip stats for terminal tasks
3. Adjust monitoring intervals or tolerate this error as a benign end-of-life signal
4. If frequent, investigate why tasks exit unexpectedly (exit codes, OOM)

Example fix

// before
ru, err := handle.Stats(ctx, interval, compute)
if err != nil { return err }
// after
ru, err := handle.Stats(ctx, interval, compute)
if err != nil {
    if strings.Contains(err.Error(), "container stopped") { return nil }
    return err
}

// check task state before polling stats
if taskState.State != structs.TaskStateRunning {
    return nil // don't call Stats for non-running tasks
}

func isContainerStoppedErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "container stopped")
}

ch, err := handle.Stats(ctx, interval, compute)
if err != nil {
    if isContainerStoppedErr(err) {
        return nil // benign: task finished between polls
    }
    return err
}

When it happens

Trigger: Calling taskHandle.Stats after the container has exited: metrics polling raced with task completion, or the Docker daemon's wait returned and run() closed doneCh via close(h.doneCh).

Common situations: Monitoring system polling stats at the exact moment the task exits; short-lived batch jobs finishing between metric intervals; task killed by OOM or user signal just before a stats request.

Related errors

• error decoding stats data: no reader body
• error decoding stats data: stats were nil
• must call StopTask for the given task before Destroy or set
• failed to collect stats: %w
• failed to decode Docker response: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/53768d03ff0b5966. Report an issue: GitHub.