t8y2/dbx · warning

agent operation capacity is temporarily exhausted

Error message

agent operation capacity is temporarily exhausted

What it means

errOperationCapacity signals that the agent's connection runtime permit pool (operationPermitTimeout, e.g. 30s) is temporarily exhausted — all concurrent operation permits are in use. Downstream consumers (e.g. cassandra-go protocol_error.go) classify it as category "resource" and Retryable=true, because it is transient back-pressure, not a fault.

Source

Thrown at agents/drivers/vastbase-go/runtime_pool.go:24

	"database/sql"
	"errors"
	"fmt"
	"os"
	"strconv"
	"strings"
	"sync"
	"time"
)

const (
	defaultRuntimePoolSize       = 32
	defaultRuntimeMetadataLimit  = 8
	defaultValidatorPoolSize     = 8
	connectionRuntimeGracePeriod = 30 * time.Second
	operationPermitTimeout       = 30 * time.Second
)

var errOperationCapacity = errors.New("agent operation capacity is temporarily exhausted")

type connectionRuntime struct {
	mu                  sync.Mutex
	validator           *sql.DB
	listTablesStatement *sql.Stmt
	permits             chan struct{}
	metadataPermits     chan struct{}
	references          int
	lastReleased        time.Time
}

func newConnectionRuntime() *connectionRuntime {
	poolSize := runtimePoolSize()
	return &connectionRuntime{
		permits:         make(chan struct{}, poolSize),
		metadataPermits: make(chan struct{}, runtimeMetadataLimit(poolSize)),
	}
}

View on GitHub (pinned to c0390bff16)

Solutions

1. Retry the operation with backoff — the error is explicitly marked retryable
2. Reduce client-side concurrency (fewer parallel RPCs per session)
3. Increase defaultValidatorPoolSize / permit pool sizing in the agent configuration
4. Investigate queries that hold permits longer than operationPermitTimeout

Example fix

// before
runAll(tables) // 50 parallel RPCs -> capacity exhausted
// after
sem := make(chan struct{}, 8) // match agent pool size
for _, t := range tables { sem <- struct{}{}; go func(){ defer func(){ <-sem }(); run(t) }() }

// bound client concurrency to the agent's permit pool
sem := make(chan struct{}, 8)
if !tryAcquire(sem) { return errors.New("local concurrency too high") }

func isRetryableCapacity(err error) bool { return errors.Is(err, errOperationCapacity) || strings.Contains(err.Error(), "temporarily exhausted") }

err := agent.Call(ctx, p)
if errors.Is(err, errOperationCapacity) {
    time.Sleep(backoff)
    err = agent.Call(ctx, p) // category=resource, retryable=true
}

When it happens

Trigger: More concurrent RPCs than the validator pool size (defaultValidatorPoolSize=8) hit acquire(); permits not released within operationPermitTimeout; long-running queries holding permits block new ones.

Common situations: Bursty parallel metadata scans from multiple sessions; a stuck/slow query monopolizing permits; undersized pool for the client's concurrency level.

Related errors

• agent request capacity is temporarily exhausted
• agent operation capacity is temporarily exhausted
• %w: %d
• agent session limit reached: {MAX_AGENT_SESSIONS}
• JDBC Session was quarantined while waiting for a connection

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/73b024c10f7a07da. Report an issue: GitHub.