hashicorp/nomad · error

failed to reattach to docker logger process: %v

Error message

failed to reattach to docker logger process: %v

What it means

The docker driver attempts to reattach to an existing docker-logger plugin subprocess when a task is recovered after Nomad/client restart. `docklog.ReattachDockerLogger(reattach)` uses the plugin reattach config (socket address, PID) stored in the driver task state; when that handshake fails, the error is wrapped with this message and RecoverTask fails.

Source

Thrown at drivers/docker/driver.go:210

		eventer:         eventer.NewEventer(ctx, logger),
		tasks:           newTaskStore(),
		config:          new(DriverConfig),
		pauseContainers: newPauseContainerStore(),
		ctx:             ctx,
		logger:          logger,
	}
	return driver
}

func (d *Driver) reattachToDockerLogger(reattachConfig *pstructs.ReattachConfig) (docklog.DockerLogger, *plugin.Client, error) {
	reattach, err := pstructs.ReattachConfigToGoPlugin(reattachConfig)
	if err != nil {
		return nil, nil, err
	}

	dlogger, dloggerPluginClient, err := docklog.ReattachDockerLogger(reattach)
	if err != nil {
		return nil, nil, fmt.Errorf("failed to reattach to docker logger process: %v", err)
	}

	return dlogger, dloggerPluginClient, nil
}

func (d *Driver) setupNewDockerLogger(container mclient.ContainerInspectResult, cfg *drivers.TaskConfig, startTime time.Time) (docklog.DockerLogger, *plugin.Client, error) {
	dlogger, pluginClient, err := docklog.LaunchDockerLogger(d.logger)
	if err != nil {
		if pluginClient != nil {
			pluginClient.Kill()
		}
		return nil, nil, fmt.Errorf("failed to launch docker logger plugin: %v", err)
	}

	if err := dlogger.Start(&docklog.StartOpts{
		Endpoint:    d.config.Endpoint,
		ContainerID: container.Container.ID,
		TTY:         container.Container.Config.Tty,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Stop and reschedule the affected allocation so StartTask runs and launches a fresh docker logger plugin instead of reattaching.
2. Verify the docker-logger plugin binary exists and is compatible with the running Nomad client version.
3. Check that the reattach socket/PID from the task state still points to a live process; if not, the state is stale and recovery cannot proceed.
4. Inspect client logs for the underlying error (e.g. 'connection refused' or 'no such process') to distinguish a dead process from a bad socket path.

Example fix

// before: relying on reattach after client restart
err := driver.RecoverTask(handle)
// after: detect un-recoverable handle and fall back to a fresh StartTask
if err := driver.RecoverTask(handle); err != nil {
    logger.Warn("reattach failed, rescheduling task", "err", err)
    // stop/destroy handle, then driver.StartTask(cfg)
}

if cfg, err := handle.GetDriverState(&st); err == nil && st.ReattachConfig != nil {
    if _, err := os.Stat(st.ReattachConfig.SocketPath); err != nil {
        return fmt.Errorf("reattach socket %s missing, reschedule needed", st.ReattachConfig.SocketPath)
    }
}

func isReattachErr(err error) bool { return err != nil && strings.Contains(err.Error(), "failed to reattach to docker logger process") }

if err := driver.RecoverTask(handle); err != nil {
    if isReattachErr(err) {
        logger.Warn("docker logger reattach failed; falling back to reschedule", "err", err)
        // stop/destroy handle, reschedule allocation
    } else {
        return err
    }
}

When it happens

Trigger: RecoverTask is called for a task handle whose saved state references a docker logger plugin process that is gone or unreachable: the plugin binary crashed, the client machine rebooted and the plugin process no longer exists, the reattach socket path was cleaned up, or the stored ReattachConfig is stale/corrupt.

Common situations: Nomad client restart where the plugin subprocess was killed but the task allocation state was preserved; log rotation or tmp-cleanup removing the plugin socket; version skew after a Nomad upgrade where the plugin protocol changed and reattachment can no longer succeed.

Related errors

• failed to launch docker logger plugin: %v
• failed to launch docker logger process %s: %v
• failed to get docker client: %w
• task with ID %q already started
• failed to decode driver config: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e151facb80cb0516. Report an issue: GitHub.