hashicorp/nomad · error

no driver handle

Error message

no driver handle

What it means

LazyHandle wraps driver handle acquisition; refreshHandleLocked retries creating a driver handle with backoff until the shutdown context is canceled, and if it exhausts (shutdown) or otherwise never obtains a handle it returns the sentinel 'no driver handle'. Any LazyHandle operation (Exec, stats, etc.) surfaces this when the underlying driver plugin cannot be (re)started, e.g. after the task has exited or the driver is unavailable.

Source

Thrown at client/allocrunner/taskrunner/lazy_handle.go:105

	for i := range retrieveFailureLimit {
		l.h = l.retrieveHandle()
		if l.h != nil {
			return l.h, nil
		}

		// Calculate the new backoff
		backoff := min((1<<(2*uint64(i)))*retrieveBackoffBaseline, retrieveBackoffLimit)

		l.logger.Debug("failed to retrieve handle", "backoff", backoff)

		select {
		case <-l.shutdownCtx.Done():
			return nil, l.shutdownCtx.Err()
		case <-time.After(backoff):
		}
	}

	return nil, fmt.Errorf("no driver handle")
}

func (l *LazyHandle) Exec(timeout time.Duration, cmd string, args []string) ([]byte, int, error) {
	h, err := l.getHandle()
	if err != nil {
		return nil, 0, err
	}

	// Only retry once
	first := true

TRY:
	out, c, err := h.Exec(timeout, cmd, args)
	if err == bstructs.ErrPluginShutdown && first {
		first = false

		h, err = l.refreshHandle()
		if err == nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the allocation state — if the alloc/task is dead, re-run against a live allocation or resubmit the job
2. Verify the driver (e.g. docker) is healthy on the client (nomad node status -verbose, driver health checks)
3. Restart the nomad client agent or the specific driver plugin so handles can be recreated
4. If seen transiently during client shutdown, simply retry after the client is back up

Example fix

// before (CLI against dead alloc)
nomad alloc exec <dead-alloc-id> ls /
// after
nomad alloc status <alloc-id>   # confirm alloc is running
nomad alloc exec <running-alloc-id> ls /

// check alloc/task liveness and driver health first
nomad alloc status <alloc-id>      # must be running
nomad node status -verbose <node>  # driver must be healthy

// retry handle acquisition with backoff, bail on shutdown
for {
    h, err := lh.getHandle()
    if err == nil { return h }
    select {
    case <-ctx.Done(): return ctx.Err()
    case <-time.After(backoff):
    }
}

When it happens

Trigger: getHandle→refreshHandleLocked cannot create a driver handle: driver plugin crashed/was reattached and the task no longer exists, driver not running on the client, or shutdownCtx already canceled during client shutdown — then Exec/Stats/etc. return 'no driver handle'

Common situations: Calling nomad alloc exec against a task whose driver handle was lost after a client agent restart, docker driver not running or failing to reattach, querying an already-completed/dead allocation

Related errors

• DriverStatsNotImplemented
• Missing task driver
• plugin is shut down
• ErrTaskNotFound
• failed creating runner for task %q: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/fbbd1b88cafc97c4. Report an issue: GitHub.