hashicorp/nomad · error

failed to decode log endpoint response as JSON: %q

Error message

failed to decode log endpoint response as JSON: %q

What it means

While streaming logs, a JSON decode error occurred but the buffered leftover data WAS readable — the client reports the raw buffered bytes with this message so you can see exactly what non-JSON payload broke the stream. It is the diagnostic counterpart to the 'failed to decode and failed to read buffered data' error.

Source

Thrown at api/fs.go:307

			// Check if we have been cancelled
			select {
			case <-cancel:
				close(frames)
				return
			default:
			}

			// Decode the next frame
			var frame StreamFrame
			if err := dec.Decode(&frame); err != nil {
				if err == io.EOF || err == io.ErrClosedPipe {
					close(frames)
				} else {
					buf, err2 := io.ReadAll(dec.Buffered())
					if err2 != nil {
						errCh <- fmt.Errorf("failed to decode and failed to read buffered data: %w", errors.Join(err, err2))
					} else {
						errCh <- fmt.Errorf("failed to decode log endpoint response as JSON: %q", buf)
					}
				}
				return
			}

			// Discard heartbeat frames
			if frame.IsHeartbeat() {
				continue
			}

			frames <- &frame
		}
	}()

	return frames, errCh
}

// FrameReader is used to convert a stream of frames into a read closer.

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the %q payload in the error to identify what the server actually sent
2. Check the allocation/client node health (nomad alloc status)
3. Confirm the request goes to a Nomad agent HTTP endpoint, not a proxy or UI
4. Retry the stream after the transient condition clears

if err := streamLogs(ctx, allocID); err != nil {
    var payloadErr *nomadStreamError
    if errors.As(err, &payloadErr) {
        log.Printf("non-JSON payload received: %q", payloadErr.buf)
    }
    return fallbackToDirectFetch(ctx, allocID)
}

When it happens

Trigger: The log/FS stream endpoint returns non-JSON content (plain text error, HTML error page, empty/garbage body) where decode fails but io.ReadAll(dec.Buffered()) succeeds.

Common situations: Nomad agent returning 5xx body mid-stream; hitting the wrong port; auth proxy returning an HTML login page; allocator crashed and stream closed with junk tail.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• unable to unmarshal response with status %d: %v
• failed to marshal command: %W
• failed to decode and failed to read buffered data: %w
• failed to marshal json from Topic: %s, Type: %s, Err: %w
• error marshaling json for stream: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d45105ec58e9623c. Report an issue: GitHub.