JuliusBrussee/caveman · error · Error

cave_mastra_usage_missing

cave_mastra_usage_missing

Error message

cave_mastra_usage_missing

What it means

Returned by ValidateTenantSchema when pool.BeginTx with RepeatableRead + ReadOnly fails before the schema inspection can run. This is a connection/session-level failure (pool exhausted, connection dropped, server refusing read-only transactions), not a schema problem. The validation is designed to run at startup to prove tenant-isolation invariants; failing to even open its transaction means the database side is unhealthy or the pool is misconfigured.

Source

Thrown at packages/agent/src/adapters.ts:407

    throw new Error("cave_vercel_usage_missing");
  }
  const normalized = normalizeUsage({
    inputTokens: noCache,
    outputTokens: usage.outputTokens,
    cacheReadTokens: cacheRead,
    cacheWriteTokens: cacheWrite,
    reasoningTokens: output.reasoningTokens,
    totalTokens: usage.totalTokens,
  }, reasoningRequired, "cave_vercel_usage_missing");
  return normalized;
}

function usageFromMastra(value: unknown, reasoningRequired: boolean): NormalizedUsage {
  const usage = record(value, "cave_mastra_usage_missing");
  const totalInput = strictInteger(usage.inputTokens, "cave_mastra_usage_missing");
  const cacheRead = optionalInteger(usage.cachedInputTokens, "cave_mastra_usage_missing");
  const cacheWrite = optionalInteger(usage.cacheCreationInputTokens, "cave_mastra_usage_missing");
  if (cacheRead + cacheWrite > totalInput) throw new Error("cave_mastra_usage_missing");
  return normalizeUsage({
    inputTokens: totalInput - cacheRead - cacheWrite,
    outputTokens: usage.outputTokens,
    cacheReadTokens: cacheRead,
    cacheWriteTokens: cacheWrite,
    reasoningTokens: usage.reasoningTokens,
    totalTokens: usage.totalTokens,
  }, reasoningRequired, "cave_mastra_usage_missing");
}

function usageFromEveEvents(events: unknown[], reasoningRequired: boolean): NormalizedUsage {
  let totalInputTokens = 0;
  let outputTokens = 0;
  let cacheReadTokens = 0;
  let cacheWriteTokens = 0;
  let steps = 0;
  for (const event of events) {
    if (!isRecord(event) || event.type !== "step.completed" || !isRecord(event.data)) continue;

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Check the wrapped error: 'context deadline exceeded' -> enlarge the startup/health timeout; 'pool exhausted' -> raise pool_max_conns or serialize startup probes
2. Ensure the database accepts READ ONLY REPEATABLE READ transactions where validation runs (primary, not a restricted standby)
3. Run validation once at startup and retry with backoff on transient connection errors rather than failing the deploy immediately
4. If a migration is in flight, order startup so validation runs after migrations complete

Example fix

// before
ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
err := postgresconfig.ValidateTenantSchema(ctx, pool) // may hit deadline on cold start

// after
err := retry.Do(func() error {
    vctx, vcancel := context.WithTimeout(ctx, 30*time.Second)
    defer vcancel()
    return postgresconfig.ValidateTenantSchema(vctx, pool)
}, retry.WithDelay(2*time.Second), retry.WithAttempts(5))

func poolCanBegin(ctx context.Context, pool *pgxpool.Pool) error {
    tx, err := pool.BeginTx(ctx, pgx.TxOptions{IsoLevel: pgx.RepeatableRead, AccessMode: pgx.ReadOnly})
    if err != nil {
        return fmt.Errorf("cannot open read-only repeatable-read transaction (pool/server state): %w", err)
    }
    return tx.Rollback(ctx)
}

func isBeginInspectionErr(err error) bool {
    return err != nil && strings.HasPrefix(err.Error(), "postgres: begin tenant schema inspection")
}

func isPoolExhausted(err error) bool {
    return err != nil && (strings.Contains(err.Error(), "pool") && strings.Contains(err.Error(), "exhausted"))
}

err := postgresconfig.ValidateTenantSchema(ctx, pool)
if isBeginInspectionErr(err) {
    if isPoolExhausted(errors.Unwrap(err)) {
        // raise pool_max_conns or serialize startup probes, then retry
    }
    // transient connection failure: retry with backoff
    err = retryCall(ctx, 3, 2*time.Second, func() error {
        return postgresconfig.ValidateTenantSchema(ctx, pool)
    })
}
if err != nil { return err }

When it happens

Trigger: Calling ValidateTenantSchema when the pool cannot hand out a connection (max conns reached), the server rejects READ ONLY REPEATABLE READ (rare, e.g. hot-standby restrictions), or the context is already cancelled/expired at call time.

Common situations: Running schema validation concurrently with migrations that hold locks; calling it with a short startup timeout against a cold database; pool_max_conns too small for the number of startup validation goroutines; standby replicas in recovery that reject some transaction modes.

Related errors

• cave_mastra_terminal_failure
• cave_harness_incomplete_evidence
• cave_eve_usage_missing
• cave_vercel_usage_missing
• cave_eve_runtime_identity_missing

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/b98fff6bf0388ec2. Report an issue: GitHub.