{"record":{"id":"e6780aeaaf9a945c","repo":"thanos-io/thanos","slug":"at-modifier-after-end","errorCode":null,"errorMessage":"at modifier after end","messagePattern":"at modifier after end","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"info","filePath":"internal/cortex/querier/queryrange/results_cache.go","lineNumber":371,"sourceCode":"\tif len(genNumbersFromResp) == 0 && genNumberFromCtx != \"\" {\n\t\tlevel.Debug(s.logger).Log(\"msg\", fmt.Sprintf(\"we found results cache gen number %s set in store but none in headers\", genNumberFromCtx))\n\t\treturn false\n\t}\n\n\tfor _, gen := range genNumbersFromResp {\n\t\tif gen != genNumberFromCtx {\n\t\t\tlevel.Debug(s.logger).Log(\"msg\", fmt.Sprintf(\"inconsistency in results cache gen numbers %s (GEN-FROM-RESPONSE) != %s (GEN-FROM-STORE), not caching the response\", gen, genNumberFromCtx))\n\t\t\treturn false\n\t\t}\n\t}\n\n\treturn true\n}\n\n// isAtModifierCachable returns true if the @ modifier result\n// is safe to cache.\nfunc (s resultsCache) isAtModifierCachable(r Request, maxCacheTime int64) bool {\n\tvar errAtModifierAfterEnd = errors.New(\"at modifier after end\")\n\t// There are 2 cases when @ modifier is not safe to cache:\n\t//   1. When @ modifier points to time beyond the maxCacheTime.\n\t//   2. If the @ modifier time is > the query range end while being\n\t//      below maxCacheTime. In such cases if any tenant is intentionally\n\t//      playing with old data, we could cache empty result if we look\n\t//      beyond query end.\n\tquery := r.GetQuery()\n\tif !strings.Contains(query, \"@\") {\n\t\treturn true\n\t}\n\texpr, err := extpromql.ParseExpr(query)\n\tif err != nil {\n\t\t// We are being pessimistic in such cases.\n\t\tlevel.Warn(s.logger).Log(\"msg\", \"failed to parse query, considering @ modifier as not cachable\", \"query\", query, \"err\", err)\n\t\treturn false\n\t}\n\n\t// This resolves the start() and end() used with the @ modifier.","sourceCodeStart":353,"sourceCodeEnd":389,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/internal/cortex/querier/queryrange/results_cache.go#L353-L389","documentation":"resultsCache.isAtModifierCachable uses this sentinel error internally; the @ modifier in a PromQL expression points to a time after the query's end (and relevant cacheability checks fail), so the response is not safe to cache. It signals shouldCacheResponse to bypass the cache rather than store a potentially wrong result.","triggerScenarios":"Querying the frontend with PromQL like `metric @ 1700000000` (or `@ start()/@ end()` variants) where the modifier timestamp is later than the query end time while iterating requests during shouldCacheResponse.","commonSituations":"Dashboard queries using @ modifier to pin evaluation against a fixed timestamp beyond the selected range; users comparing historical data with future-pointing @ modifiers.","solutions":["Adjust the @ modifier timestamp to be <= the query end time","Avoid caching-sensitive @ modifier usage; query with the modifier within the range","Remove the @ modifier or compute the value in a separate query"],"exampleFix":"// before: end=2026-01-01, query uses later timestamp\nsum(rate(errors[5m] @ 1767300000))\n// after: modifier within [start,end]\nsum(rate(errors[5m] @ end()))","handlingStrategy":"validation","validationCode":"atMod := parseAtModifier(query)\nif atMod != nil && atMod.After(end) {\n\treturn errors.New(\"@ modifier is after query end; response will not be cached\")\n}","typeGuard":null,"tryCatchPattern":"resp, err := frontend.Query(ctx, q)\nif err != nil && strings.Contains(err.Error(), \"at modifier after end\") {\n\t// treat as non-cacheable; still proceed or rewrite query\n}","preventionTips":["Clamp @ modifier timestamps to the query range client-side","Prefer @ start() / @ end() over absolute epoch timestamps","Avoid mixing @ modifier pinned times with short cacheable ranges"],"tags":["promql","cache","at-modifier","cortex"],"backgroundTag":"unsupported-cache-condition","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}