JuliusBrussee/caveman · critical · Error

cave_mastra_terminal_failure

cave_mastra_terminal_failure

Error message

cave_mastra_terminal_failure

What it means

Returned by the runtime-identity probe when the catalog query that reads current_user, session_user, superuser/bypassrls flags, role membership, and tenant-table ownership fails to execute or scan. This query runs on every runtime pool as a defense-in-depth check; this particular error means the introspection itself broke (query error, connection problem, scan/type mismatch), not that the identity is wrong.

Source

Thrown at packages/agent/src/adapters.ts:320

    throw new Error("cave_mastra_max_steps_invalid");
  }
  assertSupportedUpstream(identity, MASTRA_VERSION, "mastra", "@mastra/core");
  return createHarnessAdapter("mastra", identity, {
    package: "@mastra/core/agent",
    class: "Agent",
    method: "generate",
    usage: "FullOutput.totalUsage",
    ...(maxSteps === undefined ? {} : { preExecutionControls: { maxSteps } }),
  }, async (request) => {
    const startedAt = performance.now();
    const response = await agent.generate(request.prompt, {
      maxProcessorRetries: 0,
      runId: request.runID,
      ...(maxSteps === undefined ? {} : { maxSteps }),
      ...(request.signal === undefined ? {} : { abortSignal: request.signal }),
    });
    if (response.error !== undefined || terminalFinishReason(response.finishReason) === false) {
      throw new Error("cave_mastra_terminal_failure");
    }
    const expected = expectedProviderModel(request.plan);
    const actualModel = normalizedResponseModel(response.response?.modelId, expected.provider);
    return harnessExecution({
      request,
      text: response.text,
      provider: expected.provider,
      model: actualModel,
      usage: usageFromMastra(response.totalUsage ?? response.usage, request.plan.reasoning !== "none"),
      latencyMs: Math.round(performance.now() - startedAt),
    });
  });
}

interface NormalizedUsage {
  inputTokens: number;
  outputTokens: number;
  cacheReadTokens: number;

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Read the wrapped error first: if it is a network/canceled error, fix connectivity or the pool's connect timeout; the probe is retried on the next pool creation
2. Grant the runtime role SELECT on the catalog views used (pg_roles, pg_catalog.pg_class, information_schema.columns) or membership in a role that can read them
3. Sanitize expectedRole: validate it matches ^[A-Za-z_][A-Za-z0-9_]*$ before it reaches the query
4. Run the query manually as the same database user (psql) to see which catalog access fails

Example fix

-- before: runtime role cannot read catalogs
CREATE ROLE app_runtime LOGIN PASSWORD '...';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO app_runtime;
-- identity probe fails: permission denied for table pg_roles

-- after
GRANT pg_read_all_data TO app_runtime;  -- or targeted catalog grants

// Verify the probe can run as the configured role before app start:
func canInspectIdentity(ctx context.Context, conn *pgx.Conn, expectedRole string) error {
    var ok bool
    err := conn.QueryRow(ctx, `SELECT pg_has_role(current_user, $1, 'member')`, expectedRole).Scan(&ok)
    if err != nil {
        return fmt.Errorf("catalog read failed for identity probe; grant pg_catalog read access: %w", err)
    }
    return nil
}

func isIdentityProbeErr(err error) bool {
    return err != nil && strings.HasPrefix(err.Error(), "postgres: inspect runtime identity:")
}

func isTransientDBErr(err error) bool {
    var netErr net.Error
    return errors.As(err, &netErr) || errors.Is(err, context.DeadlineExceeded)
}

pool, err := newRuntimePool(ctx, dbURL, expectedRole)
if err != nil {
    if isIdentityProbeErr(err) && isTransientDBErr(errors.Unwrap(err)) {
        // transient: retry pool construction with backoff
        pool, err = retryNewRuntimePool(ctx, dbURL, expectedRole, 3)
    }
    if err != nil { return err }
}

When it happens

Trigger: Calling the runtime-pool constructor with a connection that drops mid-query, a database user that lacks read access to pg_roles/pg_class/information_schema, an expectedRole containing characters that break the embedded query, or a server version whose catalog columns differ from what the query selects.

Common situations: The database user was granted only DML and cannot read pg_catalog views; network interruption between dial and the identity probe; migrating to a managed Postgres (RDS/Cloud SQL) that restricts some catalog visibility; role names with quotes or unusual characters injected from config.

Related errors

• cave_harness_incomplete_evidence
• cave_vercel_usage_missing
• cave_eve_runtime_identity_missing
• cave_mastra_usage_missing
• cave_eve_usage_missing

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/4359d6b23c566c1c. Report an issue: GitHub.