{"record":{"id":"73b024c10f7a07da","repo":"t8y2/dbx","slug":"agent-operation-capacity-is-temporarily-exhausted-73b024","errorCode":null,"errorMessage":"agent operation capacity is temporarily exhausted","messagePattern":"agent operation capacity is temporarily exhausted","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"agents/drivers/vastbase-go/runtime_pool.go","lineNumber":24,"sourceCode":"\t\"database/sql\"\n\t\"errors\"\n\t\"fmt\"\n\t\"os\"\n\t\"strconv\"\n\t\"strings\"\n\t\"sync\"\n\t\"time\"\n)\n\nconst (\n\tdefaultRuntimePoolSize       = 32\n\tdefaultRuntimeMetadataLimit  = 8\n\tdefaultValidatorPoolSize     = 8\n\tconnectionRuntimeGracePeriod = 30 * time.Second\n\toperationPermitTimeout       = 30 * time.Second\n)\n\nvar errOperationCapacity = errors.New(\"agent operation capacity is temporarily exhausted\")\n\ntype connectionRuntime struct {\n\tmu                  sync.Mutex\n\tvalidator           *sql.DB\n\tlistTablesStatement *sql.Stmt\n\tpermits             chan struct{}\n\tmetadataPermits     chan struct{}\n\treferences          int\n\tlastReleased        time.Time\n}\n\nfunc newConnectionRuntime() *connectionRuntime {\n\tpoolSize := runtimePoolSize()\n\treturn &connectionRuntime{\n\t\tpermits:         make(chan struct{}, poolSize),\n\t\tmetadataPermits: make(chan struct{}, runtimeMetadataLimit(poolSize)),\n\t}\n}","sourceCodeStart":6,"sourceCodeEnd":42,"githubUrl":"https://github.com/t8y2/dbx/blob/c0390bff16418b651f4728520d99adf8ce48829a/agents/drivers/vastbase-go/runtime_pool.go#L6-L42","documentation":"errOperationCapacity signals that the agent's connection runtime permit pool (operationPermitTimeout, e.g. 30s) is temporarily exhausted — all concurrent operation permits are in use. Downstream consumers (e.g. cassandra-go protocol_error.go) classify it as category \"resource\" and Retryable=true, because it is transient back-pressure, not a fault.","triggerScenarios":"More concurrent RPCs than the validator pool size (defaultValidatorPoolSize=8) hit acquire(); permits not released within operationPermitTimeout; long-running queries holding permits block new ones.","commonSituations":"Bursty parallel metadata scans from multiple sessions; a stuck/slow query monopolizing permits; undersized pool for the client's concurrency level.","solutions":["Retry the operation with backoff — the error is explicitly marked retryable","Reduce client-side concurrency (fewer parallel RPCs per session)","Increase defaultValidatorPoolSize / permit pool sizing in the agent configuration","Investigate queries that hold permits longer than operationPermitTimeout"],"exampleFix":"// before\nrunAll(tables) // 50 parallel RPCs -> capacity exhausted\n// after\nsem := make(chan struct{}, 8) // match agent pool size\nfor _, t := range tables { sem <- struct{}{}; go func(){ defer func(){ <-sem }(); run(t) }() }","handlingStrategy":"retry","validationCode":"// bound client concurrency to the agent's permit pool\nsem := make(chan struct{}, 8)\nif !tryAcquire(sem) { return errors.New(\"local concurrency too high\") }","typeGuard":"func isRetryableCapacity(err error) bool { return errors.Is(err, errOperationCapacity) || strings.Contains(err.Error(), \"temporarily exhausted\") }","tryCatchPattern":"err := agent.Call(ctx, p)\nif errors.Is(err, errOperationCapacity) {\n    time.Sleep(backoff)\n    err = agent.Call(ctx, p) // category=resource, retryable=true\n}","preventionTips":["Cap parallel RPCs to the agent pool size (default 8)","Retry with exponential backoff on this specific error","Watch for operations holding permits near operationPermitTimeout (30s)"],"tags":["capacity","retryable","concurrency","backpressure"],"backgroundTag":"operation-capacity-exhausted","analyzedSha":"c0390bff16418b651f4728520d99adf8ce48829a","analyzedAt":"2026-09-05T23:05:10.900Z","contentChangedAt":"2026-09-05T23:05:10.900Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}