{"record":{"id":"63fa7a5cdd779e4d","repo":"hashicorp/nomad","slug":"exec-timed-out-v","errorCode":null,"errorMessage":"exec timed out: %v","messagePattern":"exec timed out: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"plugins/drivers/server.go","lineNumber":355,"sourceCode":"\texecOpts, errCh := StreamToExecOptions(server.Context(),\n\t\tmsg.Setup.Command, msg.Setup.Tty,\n\t\tserver)\n\n\tresult, err := d.ExecTaskStreaming(server.Context(),\n\t\tmsg.Setup.TaskId, execOpts)\n\n\texecOpts.Stdout.Close()\n\texecOpts.Stderr.Close()\n\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// wait for copy to be done\n\tselect {\n\tcase err = <-errCh:\n\tcase <-server.Context().Done():\n\t\terr = fmt.Errorf(\"exec timed out: %v\", server.Context().Err())\n\t}\n\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tserver.Send(&ExecTaskStreamingResponseMsg{\n\t\tExited: true,\n\t\tResult: exitResultToProto(result),\n\t})\n\n\treturn err\n}\n\nfunc (b *driverPluginServer) SignalTask(ctx context.Context, req *proto.SignalTaskRequest) (*proto.SignalTaskResponse, error) {\n\terr := b.impl.SignalTask(req.TaskId, req.Signal)\n\tif err != nil {\n\t\treturn nil, err","sourceCodeStart":337,"sourceCodeEnd":373,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/plugins/drivers/server.go#L337-L373","documentation":"After wiring up exec I/O copying, the handler waits on either the exec error channel or the stream context being done. If the gRPC server context finishes first (client disconnect, deadline, cancellation), the exec is reported as timed out with the context's error embedded.","triggerScenarios":"The ExecTaskStreaming RPC context is canceled or its deadline expires while the exec is still running; the client drops the connection mid-exec; caller-supplied context timeout is shorter than the command duration.","commonSituations":"Long-running commands exceeding the RPC deadline; user closes an exec session (alloc exec) abruptly; network partition between Nomad client and plugin; overly tight exec_timeout in tooling wrapping alloc exec.","solutions":["Inspect the wrapped context error: DeadlineExceeded means increase the exec/RPC deadline; Canceled means the client or server canceled the stream.","Increase the timeout configured for the exec operation (e.g.Nomad's exec command/API timeout).","Ensure the client keeps the stream open until the command finishes instead of disconnecting early.","Verify plugin process health and network stability between Nomad client and driver plugin."],"exampleFix":"// before\nctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\n// after\nctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) // allow long execs","handlingStrategy":"try-catch","validationCode":"// Estimate command duration and compare with the RPC deadline before exec\ndeadline, _ := ctx.Deadline()\nif time.Until(deadline) < expectedExecDuration {\n\tctx, _ = context.WithDeadline(context.Background(), time.Now().Add(expectedExecDuration*2))\n}","typeGuard":null,"tryCatchPattern":"err := srv.ExecTaskStreaming(ctx)\nif err != nil && strings.HasPrefix(err.Error(), \"exec timed out:\") {\n\tcause := ctx.Err() // DeadlineExceeded -> raise deadline; Canceled -> client disconnect\n\tif errors.Is(cause, context.DeadlineExceeded) { /* retry with larger timeout */ }\n}","preventionTips":["Set exec timeouts generously above worst-case command duration.","Keep clients connected for the whole exec session; handle user disconnects explicitly.","Harden network links between Nomad client and plugin (timeouts, keepalives).","Log context.Err() alongside the timeout to distinguish deadline vs cancellation."],"tags":["grpc","timeout","exec","context-canceled"],"backgroundTag":"rpc-deadline-exceeded","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}