{"record":{"id":"64b744dd790ed1c6","repo":"multica-ai/multica","slug":"rebuild-hourly-rollup-for-update-window-s-s-w","errorCode":null,"errorMessage":"rebuild hourly rollup for update window %s..%s: %w","messagePattern":"rebuild hourly rollup for update window (.+?)\\.\\.(.+?): %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/cmd/backfill_codex_usage_cache/main.go","lineNumber":149,"sourceCode":"\tif err != nil {\n\t\treturn err\n\t}\n\tslog.Info(\"task_usage rows updated\", \"rows\", updatedRows, \"input_tokens_removed\", removedTokens)\n\tif updatedRows == 0 || !cfg.rebuildRollup {\n\t\treturn nil\n\t}\n\n\tupdateFinishedAt, err := databaseClock(ctx, pool)\n\tif err != nil {\n\t\treturn err\n\t}\n\trollupFrom, rollupTo := rollupWindow(updateStartedAt, updateFinishedAt)\n\tvar rollupRows int64\n\tif err := pool.QueryRow(ctx,\n\t\t`SELECT rollup_task_usage_hourly_window($1::timestamptz, $2::timestamptz)`,\n\t\trollupFrom, rollupTo,\n\t).Scan(&rollupRows); err != nil {\n\t\treturn fmt.Errorf(\"rebuild hourly rollup for update window %s..%s: %w\",\n\t\t\trollupFrom.Format(time.RFC3339), rollupTo.Format(time.RFC3339), err)\n\t}\n\tslog.Info(\"hourly rollup rebuilt\",\n\t\t\"from\", rollupFrom.Format(time.RFC3339),\n\t\t\"to\", rollupTo.Format(time.RFC3339),\n\t\t\"rows_touched\", rollupRows)\n\treturn nil\n}\n\nfunc (cfg *config) parseAndValidate(now time.Time) error {\n\tif cfg.cutoffRaw == \"\" {\n\t\treturn fmt.Errorf(\"--cutoff is required; use the hosted deployment time of the Codex usage normalization fix\")\n\t}\n\tcutoff, err := time.Parse(time.RFC3339, cfg.cutoffRaw)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"parse --cutoff as RFC3339: %w\", err)\n\t}\n\tcutoff = cutoff.UTC()","sourceCodeStart":131,"sourceCodeEnd":167,"githubUrl":"https://github.com/multica-ai/multica/blob/2c0912b6ec764b373d44eeea1e80f0d9f11ab417/server/cmd/backfill_codex_usage_cache/main.go#L131-L167","documentation":"Returned after the batch updates complete, when the final `SELECT rollup_task_usage_hourly_window($1, $2)` call to rebuild the hourly usage rollup over the update window (updateStartedAt/updateFinishedAt padded by ±1s) fails. At this point the task_usage rows are already corrected; only the derived rollup is stale. The window bounds are included in the message so the exact range can be re-rolled manually.","triggerScenarios":"The rollup function erroring due to a schema drift (function missing/renamed after a partial migration), a dead connection at the end of a long run, or an error raised inside rollup_task_usage_hourly_window itself.","commonSituations":"Running the backfill against a database whose migrations are behind (rollup function or hourly table absent); postgres restart between the update batches and the rollup; timeout on a very large window.","solutions":["Manually re-run just the rollup for the window printed in the error: SELECT rollup_task_usage_hourly_window('<from>'::timestamptz, '<to>'::timestamptz);","Confirm the database schema is up to date (rollup_task_usage_hourly_window and its table exist) via migrations before re-running the tool.","If the connection died, re-run the whole tool — already-corrected rows no longer match the eligibility predicate, so it self-skip them."],"exampleFix":"-- before: backfill aborted with\n-- rebuild hourly rollup for update window 2026-01-01T00:00:00Z..2026-01-01T01:00:00Z: ...\n\n-- after: finish by hand with the exact window from the message\nSELECT rollup_task_usage_hourly_window('2026-01-01T00:00:00Z'::timestamptz, '2026-01-01T01:00:00Z'::timestamptz);","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":["Apply all migrations before running the backfill so rollup_task_usage_hourly_window exists.","Keep the printed window from the error message — it is exactly what to re-roll manually if needed.","After any partial failure, verify hourly rollup rows match the dry-run totals before closing the incident."],"tags":["database","backfill","rollup","postgres","partial-state"],"backgroundTag":null,"analyzedSha":"2c0912b6ec764b373d44eeea1e80f0d9f11ab417","analyzedAt":"2026-08-15T13:25:18.241Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}