hashicorp/nomad · error

failed to stream %q: %v

Error message

failed to stream %q: %v

What it means

During log streaming (logsImpl), a read from the open log file failed for a reason other than a closed connection. EPIPE (client disconnect) is treated as normal termination; anything else is wrapped as 'failed to stream %q: %v' with the file path and underlying error.

Source

Thrown at client/fs_endpoint.go:632

		select {
		case <-ctx.Done():
			return nil
		default:
		}

		if err != nil {
			// Check if there was an error where the file does not exist. That means
			// it got rotated out from under us.
			if os.IsNotExist(err) {
				continue
			}

			// Check if the connection was closed
			if err == syscall.EPIPE {
				return nil
			}

			return fmt.Errorf("failed to stream %q: %v", p, err)
		}

		if exitAfter {
			return nil
		}

		// defensively check to make sure StreamFramer hasn't stopped
		// running to avoid tight loops with goroutine leaks as in
		// #3342
		select {
		case <-framer.ExitCh():
			return nil
		default:
		}

		// Since we successfully streamed, update the overall offset/idx.
		offset = int64(0)
		nextIdx = idx + 1

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the log stream from the latest index; rotation-deletion is handled by re-listing (goto logic) but other errors need a fresh request.
2. If streaming a completed task, ensure the file still exists (Stat before following).
3. Check the wrapped OS error for hardware/permission problems on the client node.
4. For long follow sessions, add client-side reconnect logic on stream errors.

Example fix

// before
r, err := client.Allocs().Logs(alloc, true, "web", "stdout", nil, nil)
// after
var r *bufio.Scanner
for attempt := 0; attempt < 3; attempt++ {
    r, err = client.Allocs().Logs(alloc, true, "web", "stdout", nil, nil)
    if err == nil {
        break
    }
    time.Sleep(time.Duration(attempt) * time.Second)
}

info, _, err := client.AllocFS().Stat(alloc, "/alloc/logs/"+task+".stdout.0", nil)
if err != nil {
    return fmt.Errorf("log file missing: %w", err)
}

func isStreamFailure(err error) bool {
    return strings.Contains(err.Error(), "failed to stream")
}

err := retry(3, backoff, func() error {
    return streamLogsWithReconnect(alloc, task)
})
if err != nil && isStreamFailure(err) {
    log.Printf("stream ended with %v; restarting from latest index", err)
}

When it happens

Trigger: Reading a log file that was rotated/truncated or deleted mid-stream; underlying file read I/O errors; the log file handle becoming invalid because the task or alloc was cleaned up while streaming with follow=true.

Common situations: Long-running follow sessions across log rotation; task restarts truncating stdout/stderr files mid-read; disk errors or files removed by GC during streaming; container runtime restarting and replacing log files.

Related errors

• failed to list entries: %v
• unable to determine remaining read limit
• state for allocation %s not found on client
• unknown task name %q
• task %q not started yet. No logs available

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/47b4fc6f4a4fe350. Report an issue: GitHub.