{"record":{"id":"c5b2c87d466f9773","repo":"t8y2/dbx","slug":"agent-operation-capacity-is-temporarily-exhausted","errorCode":null,"errorMessage":"agent operation capacity is temporarily exhausted","messagePattern":"agent operation capacity is temporarily exhausted","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"agents/drivers/cassandra-go/runtime.go","lineNumber":24,"sourceCode":"\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"os\"\n\t\"strconv\"\n\t\"strings\"\n\t\"sync\"\n\t\"time\"\n\n\tgocql \"github.com/apache/cassandra-gocql-driver/v2\"\n)\n\nconst (\n\tdefaultRuntimePoolSize      = 32\n\tdefaultRuntimeMetadataLimit = 8\n\toperationPermitTimeout      = 30 * time.Second\n)\n\nvar errOperationCapacity = errors.New(\"agent operation capacity is temporarily exhausted\")\n\ntype connectionRuntime struct {\n\tmu               sync.Mutex\n\tconfig           cassandraConfig\n\tsessions         map[string]*gocql.Session\n\tretiredSessions  []*gocql.Session\n\tpermits          chan struct{}\n\tmetadataPermits  chan struct{}\n\tactiveOperations int\n\treferences       int\n\tclosed           bool\n}\n\nfunc newConnectionRuntime(cp connectParams) (*connectionRuntime, error) {\n\tconfig, err := parseCassandraConfig(cp)\n\tif err != nil {\n\t\treturn nil, err\n\t}","sourceCodeStart":6,"sourceCodeEnd":42,"githubUrl":"https://github.com/t8y2/dbx/blob/c0390bff16418b651f4728520d99adf8ce48829a/agents/drivers/cassandra-go/runtime.go#L6-L42","documentation":"errOperationCapacity signals that the driver's bounded runtime is saturated: either all operation permits are in use (acquire timed out after operationPermitTimeout) or the metadata acquisition limit was hit and the context expired. It is classified by classifyRPCError as a retryable 'resource' category error.","triggerScenarios":"acquire waits operationPermitTimeout (30s) without a free permit; allKeyspaceMetadata/metadata acquisition returns errOperationCapacity when ctx.Done fires while metadataAcquired is false; concurrent load exceeds the 32-pool-size / 8-metadata-limit defaults.","commonSituations":"Sudden spike of concurrent queries against one agent connection; slow/overloaded Cassandra cluster making metadata fetches hang past the context deadline; clients issuing more parallel operations than the runtime pool supports.","solutions":["Retry the operation with backoff — classifyRPCError marks it Retryable=true","Reduce client-side concurrency or serialize burst operations","Increase defaultRuntimePoolSize / defaultRuntimeMetadataLimit if the workload legitimately needs more","Investigate why permits are held so long (slow queries, stuck metadata fetches) or raise the context deadline"],"exampleFix":"// before\n// result := rpc(\"execute_query\", opts)\n// after\n// result := withRetry(func() error { return rpc(\"execute_query\", opts) }, // retryable: resource category\n//   backoff.Exponential(100*time.Millisecond, 5))","handlingStrategy":"retry","validationCode":null,"typeGuard":"function isCapacityError(err) {\n  return err && err.message.includes(\"agent operation capacity is temporarily exhausted\");\n}","tryCatchPattern":"try { result = await rpc(\"execute_query\", opts) }\ncatch (e) {\n  if (isCapacityError(e)) {\n    await sleep(backoff(attempt++)); // retryable: resource category\n    result = await rpc(\"execute_query\", opts);\n  }\n}","preventionTips":["Cap client concurrency below the runtime pool size (default 32)","Implement exponential backoff with jitter on 'resource' category errors","Monitor permit wait times and raise limits before saturation","Prevent metadata fetch storms after topology changes"],"tags":["capacity","rate-limiting","retryable","cassandra","resource-exhaustion"],"backgroundTag":"capacity-exhausted","analyzedSha":"c0390bff16418b651f4728520d99adf8ce48829a","analyzedAt":"2026-09-05T23:05:10.900Z","contentChangedAt":"2026-09-05T23:05:10.900Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}