{"record":{"id":"8e2a9fc53909acd7","repo":"hashicorp/nomad","slug":"websocket-closed-before-receiving-exit-code-w","errorCode":null,"errorMessage":"websocket closed before receiving exit code: %w","messagePattern":"websocket closed before receiving exit code: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"api/allocations_exec.go","lineNumber":228,"sourceCode":"\t\t\t}\n\t\t}\n\t}()\n\n\treturn errCh\n}\n\nfunc (s *execSession) startReceiving(ctx context.Context, conn *websocket.Conn) (<-chan int, <-chan error) {\n\texitCodeCh := make(chan int, 1)\n\terrCh := make(chan error, 1)\n\n\tgo func() {\n\t\tfor ctx.Err() == nil {\n\n\t\t\t// Decode the next frame\n\t\t\tvar frame ExecStreamingOutput\n\t\t\terr := conn.ReadJSON(&frame)\n\t\t\tif websocket.IsCloseError(err, websocket.CloseNormalClosure) {\n\t\t\t\terrCh <- fmt.Errorf(\"websocket closed before receiving exit code: %w\", err)\n\t\t\t\treturn\n\t\t\t} else if err != nil {\n\t\t\t\terrCh <- err\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tswitch {\n\t\t\tcase frame.Stdout != nil:\n\t\t\t\tif len(frame.Stdout.Data) != 0 {\n\t\t\t\t\ts.stdout.Write(frame.Stdout.Data)\n\t\t\t\t}\n\t\t\t\t// don't really do anything if stdout is closing\n\t\t\tcase frame.Stderr != nil:\n\t\t\t\tif len(frame.Stderr.Data) != 0 {\n\t\t\t\t\ts.stderr.Write(frame.Stderr.Data)\n\t\t\t\t}\n\t\t\t\t// don't really do anything if stderr is closing\n\t\t\tcase frame.Exited && frame.Result != nil:","sourceCodeStart":210,"sourceCodeEnd":246,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/api/allocations_exec.go#L210-L246","documentation":"In api/allocations_exec.go:228, the output-reading goroutine treats a normal websocket close (CloseNormalClosure, 1000) as an error if it happens before the final frame carrying the exit code was received. It means the node closed the exec websocket cleanly but the session terminated before reporting an exit status, so the client cannot tell whether the command succeeded.","triggerScenarios":"Streaming exec output when the server closes the websocket with a normal closure before sending the ExecStreamingExit frame: task/agent killed the session early, server-side timeout, or a protocol mismatch between client and node versions.","commonSituations":"Nomad client/server version skew where the node's exec handler does not send the exit frame; the allocation's task exits in a way that tears down the socket before the exit code frame; proxies that buffer and close the stream.","solutions":["Check Nomad client/server version compatibility; upgrade the agent if it predates reliable exit-code streaming","Read the wrapped close error and treat the command's outcome as unknown — rerun or check task logs via the AllocFS API","Inspect allocation/task events (nomad alloc status) to see how the task actually exited","Retry the exec command if it is idempotent","Check for intermediary proxies/load balancers that terminate websockets early"],"exampleFix":"// before: error surfaces as opaque failure\nexit, err := execStream(conn)\n// after: treat as unknown exit and fall back to logs\nif err != nil && strings.Contains(err.Error(), \"closed before receiving exit code\") {\n    logs, lerr := allocs.Logs(allocID, \"main\", \"stderr\", false, \"\", nil, nil)\n    if lerr == nil { fmt.Fprint(os.Stderr, logs) }\n    return -1, nil // unknown status\n}","handlingStrategy":"fallback","validationCode":"// verify server capability before streaming\nif !nodeSupportsExecStreaming(client, nodeID) { return errors.New(\"node agent too old for exit-code streaming\") }","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"websocket closed before receiving exit code\") {\n\t// exit status unknown: consult allocation logs/events instead\n\texitCode = -1\n}","preventionTips":["Keep Nomad client and server agents on compatible versions","Avoid proxies that terminate websockets early","Check nomad alloc status/task events when the exit code is unknown","Only rely on exit codes for tasks on up-to-date agents"],"tags":["websocket","exec","exit-code","nomad-api"],"backgroundTag":"websocket-closed-before-exit-code","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}