hashicorp/nomad · warning

process failed to exit after 15 seconds

Error message

process failed to exit after 15 seconds

What it means

During graceful Shutdown, the executor sends the kill signal to the container's initial processes and then waits on the l.userProcExited channel. If the task's user process does not exit within 15 seconds of the signal, Shutdown gives up and returns this timeout error, after which Nomad proceeds with forceful cleanup.

Source

Thrown at drivers/shared/executor/executor_linux_cgo.go:406

			return nil
		case <-time.After(grace):
			if err := l.container.Signal(os.Kill); err != nil {
				return err
			}
		}
	} else {
		err := l.container.Signal(os.Kill)
		if err != nil {
			l.logger.Info("no grace fail", "error", err)
			return err
		}
	}

	select {
	case <-l.userProcExited:
		return nil
	case <-time.After(time.Second * 15):
		return fmt.Errorf("process failed to exit after 15 seconds")
	}
}

// UpdateResources updates the resource isolation with new values to be enforced
func (l *LibcontainerExecutor) UpdateResources(resources *drivers.Resources) error {
	return nil
}

// Version returns the api version of the executor
func (l *LibcontainerExecutor) Version() (*ExecutorVersion, error) {
	return &ExecutorVersion{Version: ExecutorVersionLatest}, nil
}

// Stats returns the resource statistics for processes managed by the executor
func (l *LibcontainerExecutor) Stats(ctx context.Context, interval time.Duration) (<-chan *cstructs.TaskResourceUsage, error) {
	ch := make(chan *cstructs.TaskResourceUsage)
	go l.handleStats(ch, ctx, interval)
	return ch, nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Configure the app to exit promptly on the configured kill signal (install a SIGTERM handler that terminates) or change task kill_signal to one the app honors
2. Reduce app shutdown work or make it async so it finishes under 15 seconds
3. Use Nomad's kill_timeout plus driver force-kill behavior to ensure the task is eventually SIGKILLed
4. Investigate with ps/strace why the process remains after the signal (deadlock, D-state, ignored signal)

// verify the app honors the configured signal before relying on graceful shutdown
// e.g. ensure the process installs a SIGTERM handler:
//   signal.Notify/sh, or test: kill -TERM <pid> and confirm exit within timeout

if err := executor.Shutdown(); err != nil && strings.Contains(err.Error(), "failed to exit after 15 seconds") {
    // rely on driver force-kill, and alert that the app ignores/handles signals slowly
    log.Printf("graceful shutdown timed out: %v", err)
}

When it happens

Trigger: The container's initial process (and children) remain alive for >15s after receiving the configured shutdown signal — app ignores SIGTERM/SIGINT, or shutdown handler takes too long.

Common situations: Application traps SIGTERM but its cleanup exceeds 15s; app never installs a signal handler and ignores the default; blocked in I/O or a deadlock preventing exit; kill signal sent but the process actually listens on a different mechanism (e.g. child process holding the tty).

Related errors

• process did not exit after 15 seconds
• executor failed to shutdown error: no process found
• error unknown signal given for shutdown: %s
• executor shutdown error: %v
• rcp.connection_write_timeout must be greater than zero

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/2b00900d45027ef6. Report an issue: GitHub.