{"record":{"id":"53768d03ff0b5966","repo":"hashicorp/nomad","slug":"container-stopped","errorCode":null,"errorMessage":"container stopped","messagePattern":"container stopped","errorType":"exception","errorClass":"RecoverableError","httpStatus":null,"severity":"info","filePath":"drivers/docker/stats.go","lineNumber":85,"sourceCode":"// close resource usage. Any further sends will be dropped.\nfunc (u *usageSender) close() {\n\tu.mu.Lock()\n\tdefer u.mu.Unlock()\n\tif u.closed {\n\t\t// already closed\n\t\treturn\n\t}\n\n\tu.closed = true\n\tclose(u.destCh)\n}\n\n// Stats starts collecting stats from the docker daemon and sends them on the\n// returned channel.\nfunc (h *taskHandle) Stats(ctx context.Context, interval time.Duration, compute cpustats.Compute) (<-chan *cstructs.TaskResourceUsage, error) {\n\tselect {\n\tcase <-h.doneCh:\n\t\treturn nil, nstructs.NewRecoverableError(fmt.Errorf(\"container stopped\"), false)\n\tdefault:\n\t}\n\n\tdestCh, recvCh := newStatsChanPipe()\n\tgo h.collectStats(ctx, destCh, interval, compute)\n\treturn recvCh, nil\n}\n\n// collectStats starts collecting resource usage stats of a Docker container\n// and does this until the context or the tasks handler done channel is closed.\nfunc (h *taskHandle) collectStats(ctx context.Context, destCh *usageSender, interval time.Duration, compute cpustats.Compute) {\n\tdefer destCh.close()\n\n\t// retry tracks the number of retries the collection has been through since\n\t// the last successful Docker API call. This is used to calculate the\n\t// backoff time for the collection ticker.\n\tvar retry uint64\n","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/stats.go#L67-L103","documentation":"Stats() is the entry point the task runner uses to begin streaming resource usage. It first checks the task handle's doneCh; if the container run loop already finished (doneCh closed), it returns a non-recoverable structured error 'container stopped' instead of starting a pointless collector.","triggerScenarios":"Calling taskHandle.Stats after the container has exited: metrics polling raced with task completion, or the Docker daemon's wait returned and run() closed doneCh via close(h.doneCh).","commonSituations":"Monitoring system polling stats at the exact moment the task exits; short-lived batch jobs finishing between metric intervals; task killed by OOM or user signal just before a stats request.","solutions":["Treat this as expected lifecycle behavior: the structured error's Recoverable flag is false, so stop polling and deregister stats for the finished task","Check task state before requesting stats; skip stats for terminal tasks","Adjust monitoring intervals or tolerate this error as a benign end-of-life signal","If frequent, investigate why tasks exit unexpectedly (exit codes, OOM)"],"exampleFix":"// before\nru, err := handle.Stats(ctx, interval, compute)\nif err != nil { return err }\n// after\nru, err := handle.Stats(ctx, interval, compute)\nif err != nil {\n    if strings.Contains(err.Error(), \"container stopped\") { return nil }\n    return err\n}","handlingStrategy":"try-catch","validationCode":"// check task state before polling stats\nif taskState.State != structs.TaskStateRunning {\n    return nil // don't call Stats for non-running tasks\n}","typeGuard":"func isContainerStoppedErr(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"container stopped\")\n}","tryCatchPattern":"ch, err := handle.Stats(ctx, interval, compute)\nif err != nil {\n    if isContainerStoppedErr(err) {\n        return nil // benign: task finished between polls\n    }\n    return err\n}","preventionTips":["Only request stats for tasks in running state","Treat this error as expected lifecycle noise in metrics pipelines","Back off polling on terminal task events","Correlate with exit events to confirm benign races"],"tags":["docker","stats","lifecycle","container-stopped"],"backgroundTag":"container-stopped","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}