multica-ai/multica · error
rebuild hourly rollup for update window %s..%s: %w
Error message
rebuild hourly rollup for update window %s..%s: %w
What it means
Returned after the batch updates complete, when the final `SELECT rollup_task_usage_hourly_window($1, $2)` call to rebuild the hourly usage rollup over the update window (updateStartedAt/updateFinishedAt padded by ±1s) fails. At this point the task_usage rows are already corrected; only the derived rollup is stale. The window bounds are included in the message so the exact range can be re-rolled manually.
Source
Thrown at server/cmd/backfill_codex_usage_cache/main.go:149
if err != nil {
return err
}
slog.Info("task_usage rows updated", "rows", updatedRows, "input_tokens_removed", removedTokens)
if updatedRows == 0 || !cfg.rebuildRollup {
return nil
}
updateFinishedAt, err := databaseClock(ctx, pool)
if err != nil {
return err
}
rollupFrom, rollupTo := rollupWindow(updateStartedAt, updateFinishedAt)
var rollupRows int64
if err := pool.QueryRow(ctx,
`SELECT rollup_task_usage_hourly_window($1::timestamptz, $2::timestamptz)`,
rollupFrom, rollupTo,
).Scan(&rollupRows); err != nil {
return fmt.Errorf("rebuild hourly rollup for update window %s..%s: %w",
rollupFrom.Format(time.RFC3339), rollupTo.Format(time.RFC3339), err)
}
slog.Info("hourly rollup rebuilt",
"from", rollupFrom.Format(time.RFC3339),
"to", rollupTo.Format(time.RFC3339),
"rows_touched", rollupRows)
return nil
}
func (cfg *config) parseAndValidate(now time.Time) error {
if cfg.cutoffRaw == "" {
return fmt.Errorf("--cutoff is required; use the hosted deployment time of the Codex usage normalization fix")
}
cutoff, err := time.Parse(time.RFC3339, cfg.cutoffRaw)
if err != nil {
return fmt.Errorf("parse --cutoff as RFC3339: %w", err)
}
cutoff = cutoff.UTC()View on GitHub (pinned to 2c0912b6ec)
Solutions
- Manually re-run just the rollup for the window printed in the error: SELECT rollup_task_usage_hourly_window('<from>'::timestamptz, '<to>'::timestamptz);
- Confirm the database schema is up to date (rollup_task_usage_hourly_window and its table exist) via migrations before re-running the tool.
- If the connection died, re-run the whole tool — already-corrected rows no longer match the eligibility predicate, so it self-skip them.
Example fix
-- before: backfill aborted with
-- rebuild hourly rollup for update window 2026-01-01T00:00:00Z..2026-01-01T01:00:00Z: ...
-- after: finish by hand with the exact window from the message
SELECT rollup_task_usage_hourly_window('2026-01-01T00:00:00Z'::timestamptz, '2026-01-01T01:00:00Z'::timestamptz); Defensive patterns
Strategy: retry
Prevention
- Apply all migrations before running the backfill so rollup_task_usage_hourly_window exists.
- Keep the printed window from the error message — it is exactly what to re-roll manually if needed.
- After any partial failure, verify hourly rollup rows match the dry-run totals before closing the incident.
When it happens
Trigger: The rollup function erroring due to a schema drift (function missing/renamed after a partial migration), a dead connection at the end of a long run, or an error raised inside rollup_task_usage_hourly_window itself.
Common situations: Running the backfill against a database whose migrations are behind (rollup function or hourly table absent); postgres restart between the update batches and the rollup; timeout on a very large window.
Related errors
- connect to database: %w
- ping database: %w
- acquire advisory lock %d: %w
- acquire advisory-lock connection: %w
- load dry-run summary: %w
AI-assisted analysis of multica-ai/multica@2c0912b6ec (2026-08-15).
Data as JSON: /api/errors/64b744dd790ed1c6.
Report an issue: GitHub.