hashicorp/nomad · error

failed to decode and failed to read buffered data: %w

Error message

failed to decode and failed to read buffered data: %w

What it means

When streaming task logs/frames from the FS endpoint, the client JSON-decodes each frame. If decoding fails AND the remaining buffered data in the decoder also cannot be read, both errors are joined with errors.Join and surfaced with this message. It signals the stream delivered data that is not valid JSON and the buffer read also failed.

Source

Thrown at api/fs.go:305

		for {
			// Check if we have been cancelled
			select {
			case <-cancel:
				close(frames)
				return
			default:
			}

			// Decode the next frame
			var frame StreamFrame
			if err := dec.Decode(&frame); err != nil {
				if err == io.EOF || err == io.ErrClosedPipe {
					close(frames)
				} else {
					buf, err2 := io.ReadAll(dec.Buffered())
					if err2 != nil {
						errCh <- fmt.Errorf("failed to decode and failed to read buffered data: %w", errors.Join(err, err2))
					} else {
						errCh <- fmt.Errorf("failed to decode log endpoint response as JSON: %q", buf)
					}
				}
				return
			}

			// Discard heartbeat frames
			if frame.IsHeartbeat() {
				continue
			}

			frames <- &frame
		}
	}()

	return frames, errCh
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the log stream; transient disconnects are common with short-lived allocations
2. Check the target allocation is still running (nomad alloc status)
3. Capture errors.Join()'d wrapped error to see both the decode and buffer-read causes
4. Verify network path to the Nomad client node isn't dropping connections

for attempt := 0; attempt < 3; attempt++ {
    err := streamLogs(ctx, allocID)
    if err != nil && strings.Contains(err.Error(), "failed to decode") {
        time.Sleep(backoff(attempt))
        continue
    }
    return err
}

When it happens

Trigger: Calling api.Query().Logs()/Stream() where the HTTP stream returns non-JSON or truncated content while dec.Buffered() is unreadable (e.g. connection cut mid-frame).

Common situations: Alloc just terminated mid-stream; proxy terminated the connection; server returned an HTML/plain-text error page instead of the JSON frame stream; TLS/agent crash mid-stream.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to decode log endpoint response as JSON: %q
• received an error from stdout log stream: %v
• received an error from stderr log stream: %v
• failed to stream events: %w
• failed to marshal json from Topic: %s, Type: %s, Err: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/25074598d7008c47. Report an issue: GitHub.