{"record":{"id":"e5f093d81e8d3b71","repo":"hashicorp/nomad","slug":"failed-to-inspect-exec-v","errorCode":null,"errorMessage":"failed to inspect exec: %v","messagePattern":"failed to inspect exec: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":2011,"sourceCode":"\n\tgo func() {\n\t\tif !opts.Tty {\n\t\t\t_, _ = stdcopy.StdCopy(opts.Stdout, opts.Stderr, resp.Reader)\n\t\t} else {\n\t\t\t_, _ = io.Copy(opts.Stdout, resp.Reader)\n\t\t}\n\t}()\n\n\tgo func() {\n\t\t_, _ = io.Copy(resp.Conn, opts.Stdin)\n\t\t_ = resp.CloseWrite()\n\t}()\n\n\texitCode := 999\n\tfor {\n\t\tinspect, err := client.ExecInspect(ctx, exec.ID, mclient.ExecInspectOptions{})\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to inspect exec: %v\", err)\n\t\t}\n\n\t\trunning := inspect.Running\n\t\tif running {\n\t\t\ttime.Sleep(100 * time.Millisecond)\n\t\t\tcontinue\n\t\t}\n\n\t\texitCode = inspect.ExitCode\n\t\tbreak\n\t}\n\n\treturn &drivers.ExitResult{\n\t\tExitCode: exitCode,\n\t}, nil\n}\n\nfunc (d *Driver) getOrCreateClient(timeout time.Duration) (*mclient.Client, error) {","sourceCodeStart":1993,"sourceCodeEnd":2029,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L1993-L2029","documentation":"The driver polls `ExecInspect` in a loop until the exec is no longer Running to learn its exit code (initialized to 999). If any inspect call errors — rather than reporting Running=false — the driver returns this wrapped error immediately. It indicates loss of the exec session metadata, usually because the exec finished and was reaped or the daemon connection broke.","triggerScenarios":"client.ExecInspect(ctx, exec.ID, ...) returns an error during the polling loop — exec ID no longer exists (already finished and garbage-collected), ctx cancelled/timed out mid-poll, dockerd unreachable.","commonSituations":"Long-running commands whose exec completes between polls and is inspected too late on some Docker versions; ctx timeout expiring during polling; daemon restart while exec runs; remote daemon connection dropped.","solutions":["Handle a not-found inspect result as a completed exec (treat as finished) instead of a hard failure.","Increase the timeout so inspect polling doesn't race ctx cancellation.","Retry with a fresh exec if the session was lost mid-run.","Check Docker daemon health/logs; upgrade if the daemon reaps execs aggressively."],"exampleFix":"// before\ninspect, err := client.ExecInspect(ctx, exec.ID, mclient.ExecInspectOptions{})\nif err != nil {\n    return nil, fmt.Errorf(\"failed to inspect exec: %v\", err)\n}\n// after\ninspect, err := client.ExecInspect(ctx, exec.ID, mclient.ExecInspectOptions{})\nif err != nil {\n    if errdefs.IsNotFound(err) || errors.Is(ctx.Err(), context.DeadlineExceeded) {\n        break // exec finished or session ended; use last known state\n    }\n    return nil, fmt.Errorf(\"failed to inspect exec: %v\", err)\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"code, err := driver.ExecTask(taskID, cmd, timeout)\nif err != nil && strings.Contains(err.Error(), \"failed to inspect exec\") {\n    // retry exec once with fresh session and longer timeout\n    return driver.ExecTask(taskID, cmd, timeout*2)\n}","preventionTips":["Size the exec timeout larger than the expected command duration.","Treat inspect 'not found' as completion rather than failure where possible.","Keep exec sessions short; poll with backoff."],"tags":["docker","exec","polling"],"backgroundTag":"exec-inspect-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}