hashicorp/nomad · error

failed to inspect exec

Error message

failed to inspect exec: %v

What it means

The driver polls `ExecInspect` in a loop until the exec is no longer Running to learn its exit code (initialized to 999). If any inspect call errors — rather than reporting Running=false — the driver returns this wrapped error immediately. It indicates loss of the exec session metadata, usually because the exec finished and was reaped or the daemon connection broke.

Solutions

1. Handle a not-found inspect result as a completed exec (treat as finished) instead of a hard failure.
2. Increase the timeout so inspect polling doesn't race ctx cancellation.
3. Retry with a fresh exec if the session was lost mid-run.
4. Check Docker daemon health/logs; upgrade if the daemon reaps execs aggressively.

Example fix

// before
inspect, err := client.ExecInspect(ctx, exec.ID, mclient.ExecInspectOptions{})
if err != nil {
    return nil, fmt.Errorf("failed to inspect exec: %v", err)
}
// after
inspect, err := client.ExecInspect(ctx, exec.ID, mclient.ExecInspectOptions{})
if err != nil {
    if errdefs.IsNotFound(err) || errors.Is(ctx.Err(), context.DeadlineExceeded) {
        break // exec finished or session ended; use last known state
    }
    return nil, fmt.Errorf("failed to inspect exec: %v", err)
}

code, err := driver.ExecTask(taskID, cmd, timeout)
if err != nil && strings.Contains(err.Error(), "failed to inspect exec") {
    // retry exec once with fresh session and longer timeout
    return driver.ExecTask(taskID, cmd, timeout*2)
}

When it happens

Trigger: client.ExecInspect(ctx, exec.ID, ...) returns an error during the polling loop — exec ID no longer exists (already finished and garbage-collected), ctx cancelled/timed out mid-poll, dockerd unreachable.

Common situations: Long-running commands whose exec completes between polls and is inspected too late on some Docker versions; ctx timeout expiring during polling; daemon restart while exec runs; remote daemon connection dropped.

Related errors

• cmd is required, but was empty
• command is required but was empty
• failed to attach to exec
• failed to create exec object
• binary could not be found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e5f093d81e8d3b71. Report an issue: GitHub.

Appendix: source

Thrown at drivers/docker/driver.go:2011

	go func() {
		if !opts.Tty {
			_, _ = stdcopy.StdCopy(opts.Stdout, opts.Stderr, resp.Reader)
		} else {
			_, _ = io.Copy(opts.Stdout, resp.Reader)
		}
	}()

	go func() {
		_, _ = io.Copy(resp.Conn, opts.Stdin)
		_ = resp.CloseWrite()
	}()

	exitCode := 999
	for {
		inspect, err := client.ExecInspect(ctx, exec.ID, mclient.ExecInspectOptions{})
		if err != nil {
			return nil, fmt.Errorf("failed to inspect exec: %v", err)
		}

		running := inspect.Running
		if running {
			time.Sleep(100 * time.Millisecond)
			continue
		}

		exitCode = inspect.ExitCode
		break
	}

	return &drivers.ExitResult{
		ExitCode: exitCode,
	}, nil
}

func (d *Driver) getOrCreateClient(timeout time.Duration) (*mclient.Client, error) {

View on GitHub (pinned to 482b49bf1a)