{"record":{"id":"47b4fc6f4a4fe350","repo":"hashicorp/nomad","slug":"failed-to-stream-q-v","errorCode":null,"errorMessage":"failed to stream %q: %v","messagePattern":"failed to stream %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/fs_endpoint.go","lineNumber":632,"sourceCode":"\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn nil\n\t\tdefault:\n\t\t}\n\n\t\tif err != nil {\n\t\t\t// Check if there was an error where the file does not exist. That means\n\t\t\t// it got rotated out from under us.\n\t\t\tif os.IsNotExist(err) {\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\t// Check if the connection was closed\n\t\t\tif err == syscall.EPIPE {\n\t\t\t\treturn nil\n\t\t\t}\n\n\t\t\treturn fmt.Errorf(\"failed to stream %q: %v\", p, err)\n\t\t}\n\n\t\tif exitAfter {\n\t\t\treturn nil\n\t\t}\n\n\t\t// defensively check to make sure StreamFramer hasn't stopped\n\t\t// running to avoid tight loops with goroutine leaks as in\n\t\t// #3342\n\t\tselect {\n\t\tcase <-framer.ExitCh():\n\t\t\treturn nil\n\t\tdefault:\n\t\t}\n\n\t\t// Since we successfully streamed, update the overall offset/idx.\n\t\toffset = int64(0)\n\t\tnextIdx = idx + 1","sourceCodeStart":614,"sourceCodeEnd":650,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/fs_endpoint.go#L614-L650","documentation":"During log streaming (logsImpl), a read from the open log file failed for a reason other than a closed connection. EPIPE (client disconnect) is treated as normal termination; anything else is wrapped as 'failed to stream %q: %v' with the file path and underlying error.","triggerScenarios":"Reading a log file that was rotated/truncated or deleted mid-stream; underlying file read I/O errors; the log file handle becoming invalid because the task or alloc was cleaned up while streaming with follow=true.","commonSituations":"Long-running follow sessions across log rotation; task restarts truncating stdout/stderr files mid-read; disk errors or files removed by GC during streaming; container runtime restarting and replacing log files.","solutions":["Retry the log stream from the latest index; rotation-deletion is handled by re-listing (goto logic) but other errors need a fresh request.","If streaming a completed task, ensure the file still exists (Stat before following).","Check the wrapped OS error for hardware/permission problems on the client node.","For long follow sessions, add client-side reconnect logic on stream errors."],"exampleFix":"// before\nr, err := client.Allocs().Logs(alloc, true, \"web\", \"stdout\", nil, nil)\n// after\nvar r *bufio.Scanner\nfor attempt := 0; attempt < 3; attempt++ {\n    r, err = client.Allocs().Logs(alloc, true, \"web\", \"stdout\", nil, nil)\n    if err == nil {\n        break\n    }\n    time.Sleep(time.Duration(attempt) * time.Second)\n}","handlingStrategy":"retry","validationCode":"info, _, err := client.AllocFS().Stat(alloc, \"/alloc/logs/\"+task+\".stdout.0\", nil)\nif err != nil {\n    return fmt.Errorf(\"log file missing: %w\", err)\n}","typeGuard":"func isStreamFailure(err error) bool {\n    return strings.Contains(err.Error(), \"failed to stream\")\n}","tryCatchPattern":"err := retry(3, backoff, func() error {\n    return streamLogsWithReconnect(alloc, task)\n})\nif err != nil && isStreamFailure(err) {\n    log.Printf(\"stream ended with %v; restarting from latest index\", err)\n}","preventionTips":["Implement reconnect logic for long follow sessions.","Expect rotation/deletion when tasks restart and re-open from the newest index.","Monitor client disk health; I/O failures surface here."],"tags":["nomad","client","logs","streaming","io"],"backgroundTag":"log-file-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}