multica-ai/multica · error
read database clock: %w
Error message
read database clock: %w
What it means
Returned when the `SELECT clock_timestamp()` probe fails. The tool reads the database's own clock (not the client's) before and after the update phase to derive the rollup window, so the hourly rollup covers exactly the rows the backfill touched. Failure here is a plain connection/statement failure on a trivial query — the interesting part is when it happens: the second call occurs after updates, so data may already be corrected.
Source
Thrown at server/cmd/backfill_codex_usage_cache/main.go:314
}
totalRows += rows
totalRemoved += removed
slog.Info("updated Codex task_usage batch", "rows", rows, "input_tokens_removed", removed, "total_rows", totalRows)
if cfg.sleepBetweenBatches > 0 {
select {
case <-time.After(cfg.sleepBetweenBatches):
case <-ctx.Done():
return totalRows, totalRemoved, ctx.Err()
}
}
}
return totalRows, totalRemoved, nil
}
func databaseClock(ctx context.Context, pool *pgxpool.Pool) (time.Time, error) {
var ts time.Time
if err := pool.QueryRow(ctx, `SELECT clock_timestamp()`).Scan(&ts); err != nil {
return time.Time{}, fmt.Errorf("read database clock: %w", err)
}
return ts.UTC(), nil
}
func rollupWindow(startedAt, finishedAt time.Time) (time.Time, time.Time) {
return startedAt.UTC().Add(-time.Second), finishedAt.UTC().Add(time.Second)
}
func correctedInputTokens(inputTokens, cacheReadTokens int64) int64 {
corrected := inputTokens - cacheReadTokens
if corrected < 0 {
return 0
}
return corrected
}
View on GitHub (pinned to 2c0912b6ec)
Solutions
- Determine which clock read failed: if before updates (right after taking the advisory lock), simply re-run; if after updates, re-run anyway — corrected rows self-exclude and the rollup is rebuilt at the end.
- Stabilize connectivity (network, pgbouncer idle limits) for the duration of the run.
- After a successful re-run, verify the hourly rollup matches the dry-run totals in the logs.
Example fix
# before ./backfill_codex_usage_cache --cutoff ... --execute # read database clock: ... connection refused (db failover) # after # wait for postgres to stabilize, then re-run; the tool resumes safely ./backfill_codex_usage_cache --cutoff 2026-06-01T12:00:00Z --execute
Defensive patterns
Strategy: retry
Prevention
- Stabilize connectivity (or reconnecting pooler settings) before the --execute phase.
- After re-running to completion, verify the rollup log totals match the dry-run summary.
- A failure at the second clock read means updates already applied — always re-run rather than starting from scratch.
When it happens
Trigger: Connection dying between the update batches and the clock read; postgres restart; ctx cancelled right at phase end. If it is the first clock read (before updates), nothing was modified.
Common situations: Flaky connections dropping at phase boundaries; failover during the run; the same environmental issues as other connection errors but surfacing at the cheapest query.
Related errors
- connect to database: %w
- update Codex task_usage batch: %w
- ping database: %w
- acquire advisory-lock connection: %w
- acquire advisory lock %d: %w
AI-assisted analysis of multica-ai/multica@2c0912b6ec (2026-08-15).
Data as JSON: /api/errors/c147906ef9d17ac8.
Report an issue: GitHub.