{"record":{"id":"e151facb80cb0516","repo":"hashicorp/nomad","slug":"failed-to-reattach-to-docker-logger-process-v","errorCode":null,"errorMessage":"failed to reattach to docker logger process: %v","messagePattern":"failed to reattach to docker logger process: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":210,"sourceCode":"\t\teventer:         eventer.NewEventer(ctx, logger),\n\t\ttasks:           newTaskStore(),\n\t\tconfig:          new(DriverConfig),\n\t\tpauseContainers: newPauseContainerStore(),\n\t\tctx:             ctx,\n\t\tlogger:          logger,\n\t}\n\treturn driver\n}\n\nfunc (d *Driver) reattachToDockerLogger(reattachConfig *pstructs.ReattachConfig) (docklog.DockerLogger, *plugin.Client, error) {\n\treattach, err := pstructs.ReattachConfigToGoPlugin(reattachConfig)\n\tif err != nil {\n\t\treturn nil, nil, err\n\t}\n\n\tdlogger, dloggerPluginClient, err := docklog.ReattachDockerLogger(reattach)\n\tif err != nil {\n\t\treturn nil, nil, fmt.Errorf(\"failed to reattach to docker logger process: %v\", err)\n\t}\n\n\treturn dlogger, dloggerPluginClient, nil\n}\n\nfunc (d *Driver) setupNewDockerLogger(container mclient.ContainerInspectResult, cfg *drivers.TaskConfig, startTime time.Time) (docklog.DockerLogger, *plugin.Client, error) {\n\tdlogger, pluginClient, err := docklog.LaunchDockerLogger(d.logger)\n\tif err != nil {\n\t\tif pluginClient != nil {\n\t\t\tpluginClient.Kill()\n\t\t}\n\t\treturn nil, nil, fmt.Errorf(\"failed to launch docker logger plugin: %v\", err)\n\t}\n\n\tif err := dlogger.Start(&docklog.StartOpts{\n\t\tEndpoint:    d.config.Endpoint,\n\t\tContainerID: container.Container.ID,\n\t\tTTY:         container.Container.Config.Tty,","sourceCodeStart":192,"sourceCodeEnd":228,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L192-L228","documentation":"The docker driver attempts to reattach to an existing docker-logger plugin subprocess when a task is recovered after Nomad/client restart. `docklog.ReattachDockerLogger(reattach)` uses the plugin reattach config (socket address, PID) stored in the driver task state; when that handshake fails, the error is wrapped with this message and RecoverTask fails.","triggerScenarios":"RecoverTask is called for a task handle whose saved state references a docker logger plugin process that is gone or unreachable: the plugin binary crashed, the client machine rebooted and the plugin process no longer exists, the reattach socket path was cleaned up, or the stored ReattachConfig is stale/corrupt.","commonSituations":"Nomad client restart where the plugin subprocess was killed but the task allocation state was preserved; log rotation or tmp-cleanup removing the plugin socket; version skew after a Nomad upgrade where the plugin protocol changed and reattachment can no longer succeed.","solutions":["Stop and reschedule the affected allocation so StartTask runs and launches a fresh docker logger plugin instead of reattaching.","Verify the docker-logger plugin binary exists and is compatible with the running Nomad client version.","Check that the reattach socket/PID from the task state still points to a live process; if not, the state is stale and recovery cannot proceed.","Inspect client logs for the underlying error (e.g. 'connection refused' or 'no such process') to distinguish a dead process from a bad socket path."],"exampleFix":"// before: relying on reattach after client restart\nerr := driver.RecoverTask(handle)\n// after: detect un-recoverable handle and fall back to a fresh StartTask\nif err := driver.RecoverTask(handle); err != nil {\n    logger.Warn(\"reattach failed, rescheduling task\", \"err\", err)\n    // stop/destroy handle, then driver.StartTask(cfg)\n}","handlingStrategy":"try-catch","validationCode":"if cfg, err := handle.GetDriverState(&st); err == nil && st.ReattachConfig != nil {\n    if _, err := os.Stat(st.ReattachConfig.SocketPath); err != nil {\n        return fmt.Errorf(\"reattach socket %s missing, reschedule needed\", st.ReattachConfig.SocketPath)\n    }\n}","typeGuard":"func isReattachErr(err error) bool { return err != nil && strings.Contains(err.Error(), \"failed to reattach to docker logger process\") }","tryCatchPattern":"if err := driver.RecoverTask(handle); err != nil {\n    if isReattachErr(err) {\n        logger.Warn(\"docker logger reattach failed; falling back to reschedule\", \"err\", err)\n        // stop/destroy handle, reschedule allocation\n    } else {\n        return err\n    }\n}","preventionTips":["Keep the plugin binary and Nomad client versions in lockstep; reattach fails across protocol changes.","Avoid cleaning tmp/plugin socket directories while allocations are running.","Prefer full allocation reschedule over in-place recovery after client restarts when reattach errors recur.","Monitor for this error after upgrades and stop/reschedule affected allocations proactively."],"tags":["docker","plugin-reattach","nomad-driver","process-recovery"],"backgroundTag":"plugin-reattach-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}