hashicorp/nomad · error

failed to setup replacement docker logger: %v

Error message

failed to setup replacement docker logger: %v

What it means

When reattaching to the original docker logger plugin process fails, RecoverTask attempts to start a fresh logger (setupNewDockerLogger) that streams the container's logs. If that setup also fails — plugin launch error, Start() error, bad log paths, TLS options — the container is stopped with a zero timeout as cleanup and this error is returned.

Source

Thrown at drivers/docker/driver.go:299

		containerImage:          container.Container.Image,
		doneCh:                  make(chan bool),
		waitCh:                  make(chan struct{}),
		removeContainerOnExit:   d.config.GC.Container,
		net:                     handleState.DriverNetwork,
		disableCpusetManagement: d.config.disableCpusetManagement,
	}

	if loggingIsEnabled(d.config, handle.Config) {
		h.dlogger, h.dloggerPluginClient, err = d.reattachToDockerLogger(handleState.ReattachConfig)
		if err != nil {
			d.logger.Warn("failed to reattach to docker logger process", "error", err)

			h.dlogger, h.dloggerPluginClient, err = d.setupNewDockerLogger(container, handle.Config, time.Now())
			if err != nil {
				if _, err := dockerClient.ContainerStop(d.ctx, handleState.ContainerID, stopWithZeroTimeout()); err != nil {
					d.logger.Warn("failed to stop container during cleanup", "container_id", handleState.ContainerID, "error", err)
				}
				return fmt.Errorf("failed to setup replacement docker logger: %v", err)
			}

			if err := handle.SetDriverState(h.buildState()); err != nil {
				if _, err := dockerClient.ContainerStop(d.ctx, handleState.ContainerID, stopWithZeroTimeout()); err != nil {
					d.logger.Warn("failed to stop container during cleanup", "container_id", handleState.ContainerID, "error", err)
				}
				return fmt.Errorf("failed to store driver state: %v", err)
			}
		}
	}

	d.tasks.Set(handle.Config.ID, h)

	// find a pause container?

	go h.run()

	return nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped cause in logs: 'failed to launch docker logger plugin' vs 'failed to launch docker logger process' — fix the plugin installation or its Start() inputs accordingly.
2. Verify the docker_logger plugin binary exists, is executable, and matches the driver version.
3. Ensure the task's stdout/stderr log paths (allocation log dir) are writable by the plugin process and TLS cert/key/CA files still exist.
4. If log collection is not needed, set DisableLogCollection or send logs to /dev/null so recovery skips logger setup entirely.

Example fix

// before (task config sending logs to a read-only path)
// StdoutPath: /var/log/immutable/app.log
// after (let the driver manage the log dir, or disable collection)
// StdoutPath: <alloc_dir>/logs/app.stdout.0
// or driver config: { "disable_log_collection": true }

// pre-check logger plugin and log paths before recovery
func loggerReady(pluginPath string, stdoutPath, stderrPath string) error {
	info, err := os.Stat(pluginPath)
	if err != nil || info.IsDir() {
		return fmt.Errorf("docker logger plugin missing: %s", pluginPath)
	}
	if err := os.MkdirAll(filepath.Dir(stdoutPath), 0o755); err != nil {
		return fmt.Errorf("log dir not creatable: %w", err)
	}
	return nil
}

// Go: check the cause chain for plugin-launch vs start failures
if err := driver.RecoverTask(handle); err != nil {
	if strings.Contains(err.Error(), "failed to setup replacement docker logger") {
		cause := fmt.Sprintf("%v", errors.Unwrap(errors.Unwrap(err)))
		switch {
		case strings.Contains(cause, "launch docker logger plugin"):
			log.Print("reinstall/fix the docker_logger plugin")
		case strings.Contains(cause, "launch docker logger process"):
			log.Print("check Start() opts: endpoint, log paths, TLS")
		}
	}
}

When it happens

Trigger: Docker logger plugin binary/plugin not present or fails to launch; docklog plugin Start() fails due to bad endpoint, TTY mismatch, unwritable StdoutPath/StderrPath, or TLS cert/key/CA options invalid; plugin client handshake timeout.

Common situations: nomad/docker-logger plugin removed from the plugin directory after upgrade; log allocation directory permissions changed so the plugin cannot open the log files; TLS secrets referenced by the task no longer exist; plugin version incompatible with daemon output format.

Related errors

• failed to fetch docker daemon info: %v
• failed to get docker long operations client: %w
• failed to inspect container for id %q: %v
• failed to store driver state: %v
• failed to remove container %s: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e93a1df5e3e99b91. Report an issue: GitHub.