{"record":{"id":"e93a1df5e3e99b91","repo":"hashicorp/nomad","slug":"failed-to-setup-replacement-docker-logger-v","errorCode":null,"errorMessage":"failed to setup replacement docker logger: %v","messagePattern":"failed to setup replacement docker logger: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":299,"sourceCode":"\t\tcontainerImage:          container.Container.Image,\n\t\tdoneCh:                  make(chan bool),\n\t\twaitCh:                  make(chan struct{}),\n\t\tremoveContainerOnExit:   d.config.GC.Container,\n\t\tnet:                     handleState.DriverNetwork,\n\t\tdisableCpusetManagement: d.config.disableCpusetManagement,\n\t}\n\n\tif loggingIsEnabled(d.config, handle.Config) {\n\t\th.dlogger, h.dloggerPluginClient, err = d.reattachToDockerLogger(handleState.ReattachConfig)\n\t\tif err != nil {\n\t\t\td.logger.Warn(\"failed to reattach to docker logger process\", \"error\", err)\n\n\t\t\th.dlogger, h.dloggerPluginClient, err = d.setupNewDockerLogger(container, handle.Config, time.Now())\n\t\t\tif err != nil {\n\t\t\t\tif _, err := dockerClient.ContainerStop(d.ctx, handleState.ContainerID, stopWithZeroTimeout()); err != nil {\n\t\t\t\t\td.logger.Warn(\"failed to stop container during cleanup\", \"container_id\", handleState.ContainerID, \"error\", err)\n\t\t\t\t}\n\t\t\t\treturn fmt.Errorf(\"failed to setup replacement docker logger: %v\", err)\n\t\t\t}\n\n\t\t\tif err := handle.SetDriverState(h.buildState()); err != nil {\n\t\t\t\tif _, err := dockerClient.ContainerStop(d.ctx, handleState.ContainerID, stopWithZeroTimeout()); err != nil {\n\t\t\t\t\td.logger.Warn(\"failed to stop container during cleanup\", \"container_id\", handleState.ContainerID, \"error\", err)\n\t\t\t\t}\n\t\t\t\treturn fmt.Errorf(\"failed to store driver state: %v\", err)\n\t\t\t}\n\t\t}\n\t}\n\n\td.tasks.Set(handle.Config.ID, h)\n\n\t// find a pause container?\n\n\tgo h.run()\n\n\treturn nil","sourceCodeStart":281,"sourceCodeEnd":317,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L281-L317","documentation":"When reattaching to the original docker logger plugin process fails, RecoverTask attempts to start a fresh logger (setupNewDockerLogger) that streams the container's logs. If that setup also fails — plugin launch error, Start() error, bad log paths, TLS options — the container is stopped with a zero timeout as cleanup and this error is returned.","triggerScenarios":"Docker logger plugin binary/plugin not present or fails to launch; docklog plugin Start() fails due to bad endpoint, TTY mismatch, unwritable StdoutPath/StderrPath, or TLS cert/key/CA options invalid; plugin client handshake timeout.","commonSituations":"nomad/docker-logger plugin removed from the plugin directory after upgrade; log allocation directory permissions changed so the plugin cannot open the log files; TLS secrets referenced by the task no longer exist; plugin version incompatible with daemon output format.","solutions":["Check the wrapped cause in logs: 'failed to launch docker logger plugin' vs 'failed to launch docker logger process' — fix the plugin installation or its Start() inputs accordingly.","Verify the docker_logger plugin binary exists, is executable, and matches the driver version.","Ensure the task's stdout/stderr log paths (allocation log dir) are writable by the plugin process and TLS cert/key/CA files still exist.","If log collection is not needed, set DisableLogCollection or send logs to /dev/null so recovery skips logger setup entirely."],"exampleFix":"// before (task config sending logs to a read-only path)\n// StdoutPath: /var/log/immutable/app.log\n// after (let the driver manage the log dir, or disable collection)\n// StdoutPath: <alloc_dir>/logs/app.stdout.0\n// or driver config: { \"disable_log_collection\": true }","handlingStrategy":"validation","validationCode":"// pre-check logger plugin and log paths before recovery\nfunc loggerReady(pluginPath string, stdoutPath, stderrPath string) error {\n\tinfo, err := os.Stat(pluginPath)\n\tif err != nil || info.IsDir() {\n\t\treturn fmt.Errorf(\"docker logger plugin missing: %s\", pluginPath)\n\t}\n\tif err := os.MkdirAll(filepath.Dir(stdoutPath), 0o755); err != nil {\n\t\treturn fmt.Errorf(\"log dir not creatable: %w\", err)\n\t}\n\treturn nil\n}","typeGuard":null,"tryCatchPattern":"// Go: check the cause chain for plugin-launch vs start failures\nif err := driver.RecoverTask(handle); err != nil {\n\tif strings.Contains(err.Error(), \"failed to setup replacement docker logger\") {\n\t\tcause := fmt.Sprintf(\"%v\", errors.Unwrap(errors.Unwrap(err)))\n\t\tswitch {\n\t\tcase strings.Contains(cause, \"launch docker logger plugin\"):\n\t\t\tlog.Print(\"reinstall/fix the docker_logger plugin\")\n\t\tcase strings.Contains(cause, \"launch docker logger process\"):\n\t\t\tlog.Print(\"check Start() opts: endpoint, log paths, TLS\")\n\t\t}\n\t}\n}","preventionTips":["Ship and pin the docker_logger plugin with every agent upgrade.","Keep allocation log directories writable by the agent/plugin user.","Test recovery after changing TLS config or rotating secrets.","Use DisableLogCollection when log streaming is not required."],"tags":["docker","logging-plugin","task-recovery","cleanup"],"backgroundTag":"docker-logger-setup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}