grafana/k6 · error

marking time series as stale failed: %w

Error message

marking time series as stale failed: %w

What it means

On test completion, the remote-write output tried to send stale markers for all seen time series (K6_PROMETHEUS_RW_STALE_MARKERS=true) and the Store POST to Prometheus failed; Stop wraps the transport error.

Source

Thrown at internal/output/prometheusrw/remotewrite/remotewrite.go:117

// Stop stops the output.
func (o *Output) Stop() error {
	o.logger.Debug("Stopping the output")
	defer o.logger.Debug("Output stopped")
	o.periodicFlusher.Stop()

	if !o.config.StaleMarkers.Bool {
		return nil
	}
	staleMarkers := o.staleMarkers()
	if len(staleMarkers) < 1 {
		o.logger.Debug("No time series to mark as stale")
		return nil
	}
	o.logger.WithField("staleMarkers", len(staleMarkers)).Debug("Marking time series as stale")

	err := o.client.Store(context.Background(), staleMarkers)
	if err != nil {
		return fmt.Errorf("marking time series as stale failed: %w", err)
	}
	return nil
}

// staleMarkers maps all the seen time series with a stale marker.
func (o *Output) staleMarkers() []*prompb.TimeSeries {
	// Add 1ms so in the extreme case that the time frame
	// between the last and the next flush operation is under-millisecond,
	// we can avoid the sample being seen as a duplicate,
	// if we force it in the future.
	// It is essential because if it overlaps, the remote write discards the last sample,
	// so the stale marker and the metric will remain active for the next 5 min
	// as the default logic without stale markers.
	timestamp := o.now().
		Truncate(time.Millisecond).Add(1 * time.Millisecond).UnixMilli()

	staleMarkers := make([]*prompb.TimeSeries, 0, len(o.tsdb))
	for _, swm := range o.tsdb {

View on GitHub (pinned to 01ffac6f24)

Solutions

1. Check the wrapped error: connectivity, TLS, or server rejection of the stale-marker write
2. Ensure the remote write endpoint is still reachable when the test ends
3. Disable stale markers if not needed (K6_PROMETHEUS_RW_STALE_MARKERS=false)

When it happens

Trigger: Thrown at internal/output/prometheusrw/remotewrite/remotewrite.go:117 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of grafana/k6@01ffac6f24 (2026-08-18). Data as JSON: /api/errors/81ba8677ced2af3c. Report an issue: GitHub.