What causes "at modifier after end" in thanos-io/thanos?

Adjust the @ modifier timestamp to be <= the query end time

at modifier after end

Error message

at modifier after end

What it means

resultsCache.isAtModifierCachable uses this sentinel error internally; the @ modifier in a PromQL expression points to a time after the query's end (and relevant cacheability checks fail), so the response is not safe to cache. It signals shouldCacheResponse to bypass the cache rather than store a potentially wrong result.

Solutions

Adjust the @ modifier timestamp to be <= the query end time
Avoid caching-sensitive @ modifier usage; query with the modifier within the range
Remove the @ modifier or compute the value in a separate query

Example fix

// before: end=2026-01-01, query uses later timestamp
sum(rate(errors[5m] @ 1767300000))
// after: modifier within [start,end]
sum(rate(errors[5m] @ end()))

Defensive patterns

Strategy: validation

Validate before calling

atMod := parseAtModifier(query)
if atMod != nil && atMod.After(end) {
	return errors.New("@ modifier is after query end; response will not be cached")
}

Try / catch

resp, err := frontend.Query(ctx, q)
if err != nil && strings.Contains(err.Error(), "at modifier after end") {
	// treat as non-cacheable; still proceed or rewrite query
}

Prevention

Clamp @ modifier timestamps to the query range client-side
Prefer @ start() / @ end() over absolute epoch timestamps
Avoid mixing @ modifier pinned times with short cacheable ranges

When it happens

Trigger: Querying the frontend with PromQL like `metric @ 1700000000` (or `@ start()/@ end()` variants) where the modifier timestamp is later than the query end time while iterating requests during shouldCacheResponse.

Common situations: Dashboard queries using @ modifier to pin evaluation against a fixed timestamp beyond the selected range; users comparing historical data with future-pointing @ modifiers.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e6780aeaaf9a945c. Report an issue: GitHub.

Appendix: source

Thrown at internal/cortex/querier/queryrange/results_cache.go:371

	if len(genNumbersFromResp) == 0 && genNumberFromCtx != "" {
		level.Debug(s.logger).Log("msg", fmt.Sprintf("we found results cache gen number %s set in store but none in headers", genNumberFromCtx))
		return false
	}

	for _, gen := range genNumbersFromResp {
		if gen != genNumberFromCtx {
			level.Debug(s.logger).Log("msg", fmt.Sprintf("inconsistency in results cache gen numbers %s (GEN-FROM-RESPONSE) != %s (GEN-FROM-STORE), not caching the response", gen, genNumberFromCtx))
			return false
		}
	}

	return true
}

// isAtModifierCachable returns true if the @ modifier result
// is safe to cache.
func (s resultsCache) isAtModifierCachable(r Request, maxCacheTime int64) bool {
	var errAtModifierAfterEnd = errors.New("at modifier after end")
	// There are 2 cases when @ modifier is not safe to cache:
	//   1. When @ modifier points to time beyond the maxCacheTime.
	//   2. If the @ modifier time is > the query range end while being
	//      below maxCacheTime. In such cases if any tenant is intentionally
	//      playing with old data, we could cache empty result if we look
	//      beyond query end.
	query := r.GetQuery()
	if !strings.Contains(query, "@") {
		return true
	}
	expr, err := extpromql.ParseExpr(query)
	if err != nil {
		// We are being pessimistic in such cases.
		level.Warn(s.logger).Log("msg", "failed to parse query, considering @ modifier as not cachable", "query", query, "err", err)
		return false
	}

	// This resolves the start() and end() used with the @ modifier.

View on GitHub (pinned to 35b8b99117)