hashicorp/nomad · warning

websocket closed before receiving exit code: %w

Error message

websocket closed before receiving exit code: %w

What it means

In api/allocations_exec.go:228, the output-reading goroutine treats a normal websocket close (CloseNormalClosure, 1000) as an error if it happens before the final frame carrying the exit code was received. It means the node closed the exec websocket cleanly but the session terminated before reporting an exit status, so the client cannot tell whether the command succeeded.

Source

Thrown at api/allocations_exec.go:228

			}
		}
	}()

	return errCh
}

func (s *execSession) startReceiving(ctx context.Context, conn *websocket.Conn) (<-chan int, <-chan error) {
	exitCodeCh := make(chan int, 1)
	errCh := make(chan error, 1)

	go func() {
		for ctx.Err() == nil {

			// Decode the next frame
			var frame ExecStreamingOutput
			err := conn.ReadJSON(&frame)
			if websocket.IsCloseError(err, websocket.CloseNormalClosure) {
				errCh <- fmt.Errorf("websocket closed before receiving exit code: %w", err)
				return
			} else if err != nil {
				errCh <- err
				return
			}

			switch {
			case frame.Stdout != nil:
				if len(frame.Stdout.Data) != 0 {
					s.stdout.Write(frame.Stdout.Data)
				}
				// don't really do anything if stdout is closing
			case frame.Stderr != nil:
				if len(frame.Stderr.Data) != 0 {
					s.stderr.Write(frame.Stderr.Data)
				}
				// don't really do anything if stderr is closing
			case frame.Exited && frame.Result != nil:

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check Nomad client/server version compatibility; upgrade the agent if it predates reliable exit-code streaming
2. Read the wrapped close error and treat the command's outcome as unknown — rerun or check task logs via the AllocFS API
3. Inspect allocation/task events (nomad alloc status) to see how the task actually exited
4. Retry the exec command if it is idempotent
5. Check for intermediary proxies/load balancers that terminate websockets early

Example fix

// before: error surfaces as opaque failure
exit, err := execStream(conn)
// after: treat as unknown exit and fall back to logs
if err != nil && strings.Contains(err.Error(), "closed before receiving exit code") {
    logs, lerr := allocs.Logs(allocID, "main", "stderr", false, "", nil, nil)
    if lerr == nil { fmt.Fprint(os.Stderr, logs) }
    return -1, nil // unknown status
}

// verify server capability before streaming
if !nodeSupportsExecStreaming(client, nodeID) { return errors.New("node agent too old for exit-code streaming") }

if err != nil && strings.Contains(err.Error(), "websocket closed before receiving exit code") {
	// exit status unknown: consult allocation logs/events instead
	exitCode = -1
}

When it happens

Trigger: Streaming exec output when the server closes the websocket with a normal closure before sending the ExecStreamingExit frame: task/agent killed the session early, server-side timeout, or a protocol mismatch between client and node versions.

Common situations: Nomad client/server version skew where the node's exec handler does not send the exit frame; the allocation's task exits in a way that tears down the socket before the exit code frame; proxies that buffer and close the stream.

Related errors

• failed to send input: %w
• failed to marshal command: %W
• unsupported scheme: %v
• invalid redirect location %q: %w
• failed to inspect exit code of exec object: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8e2a9fc53909acd7. Report an issue: GitHub.