{"record":{"id":"e9acfb22e623ed04","repo":"t8y2/dbx","slug":"agent-runtime-jdbc-physical-connection-limit-reach","errorCode":null,"errorMessage":"Agent runtime JDBC physical connection limit reached: {maximum}","messagePattern":"Agent runtime JDBC physical connection limit reached: (.+?)","errorType":"exception","errorClass":"PhysicalConnectionLimitException","httpStatus":null,"severity":"error","filePath":"agents/common/src/main/java/com/dbx/agent/JdbcConnectionPoolRegistry.java","lineNumber":1685,"sourceCode":"    private static final class PhysicalConnectionBudget {\n        private final int maximum;\n        private final Semaphore permits;\n\n        private PhysicalConnectionBudget(int maximum) {\n            this.maximum = maximum;\n            this.permits = new Semaphore(maximum, true);\n        }\n\n        private void acquire(OperationDeadline deadline) throws SQLException {\n            try {\n                if (permits.tryAcquire(deadline.remainingNanos(), TimeUnit.NANOSECONDS)) {\n                    return;\n                }\n            } catch (InterruptedException error) {\n                Thread.currentThread().interrupt();\n                throw new SQLException(\"Interrupted while waiting for the JDBC physical connection budget\", error);\n            }\n            throw new PhysicalConnectionLimitException(maximum);\n        }\n\n        private Connection wrap(\n            Connection connection,\n            PhysicalConnectionCloser physicalConnectionCloser,\n            ConnectionFactoryDataSource factoryDataSource,\n            long closeTimeoutMillis,\n            ConnectionFactoryDataSource.HikariSetupAttempt setupAttempt\n        ) {\n            AtomicBoolean released = new AtomicBoolean();\n            return (Connection) Proxy.newProxyInstance(\n                JdbcConnectionPoolRegistry.class.getClassLoader(),\n                new Class<?>[] {Connection.class, HikariSetupTrackedConnection.class},\n                (proxy, method, arguments) -> {\n                    if (\"completeHikariSetup\".equals(method.getName()) && method.getParameterCount() == 0) {\n                        setupAttempt.completeSuccessfully();\n                        return null;\n                    }","sourceCodeStart":1667,"sourceCodeEnd":1703,"githubUrl":"https://github.com/t8y2/dbx/blob/c0390bff16418b651f4728520d99adf8ce48829a/agents/common/src/main/java/com/dbx/agent/JdbcConnectionPoolRegistry.java#L1667-L1703","documentation":"PhysicalConnectionLimitException thrown when the physical-connection budget semaphore cannot be acquired within the deadline: the pool already holds its maximum number of physical connections. The message interpolates the configured maximum so the operator can see the limit that was hit. This is a deliberate resource-limit signal (backpressure).","triggerScenarios":"Concurrent physical connects exceed the configured maximum physical connection limit and permits.tryAcquire times out at line 1685, i.e. all permits held longer than deadline.remainingNanos().","commonSituations":"maxPhysicalConnections configured too low for workload concurrency; slow/hung physical connections (long queries, network stalls) holding permits; leaky connections never released; cold-start storms where many requests need new physical connections at once.","solutions":["Increase the configured maximum physical connection limit to match peak concurrency","Check for connection leaks — connections not returned to the pool hold permits indefinitely","Investigate slow physical connections (DB latency, network) that keep permits occupied past deadlines","Add backpressure/retry with jitter on the client side and warm the pool before traffic spikes"],"exampleFix":"// before\nregistry = JdbcConnectionPoolRegistry.builder().maxPhysicalConnections(5).build();\n// after\nregistry = JdbcConnectionPoolRegistry.builder().maxPhysicalConnections(50).build();","handlingStrategy":"retry","validationCode":"// before scaling out, confirm the limit actually matches workload:\nif (expectedConcurrentPhysicalConnections >= configuredMaxPhysicalConnections) {\n    raiseLimit(configuredMaxPhysicalConnections * 2);\n}","typeGuard":"static boolean isPhysicalLimit(SQLException e) {\n    Throwable t = e;\n    while (t != null) {\n        if (t instanceof PhysicalConnectionLimitException) return true;\n        t = t.getCause();\n    }\n    return false;\n}","tryCatchPattern":"try {\n    lease = pool.checkout();\n} catch (SQLException e) {\n    if (isPhysicalLimit(e)) {\n        Thread.sleep(backoffWithJitter());\n        return checkoutWithRetry(attempt + 1);\n    }\n    throw e;\n}","preventionTips":["Set max physical connections to at least peak expected concurrency","Instrument and alarm on permit wait times; rising waits predict exhaustion","Fix connection leaks — leaked connections permanently consume permits","Warm the pool before traffic spikes and keep DB/network latency low"],"tags":["jdbc","connection-limit","backpressure","pool-exhaustion"],"backgroundTag":"connection-pool-exhausted","analyzedSha":"c0390bff16418b651f4728520d99adf8ce48829a","analyzedAt":"2026-09-05T23:05:10.900Z","contentChangedAt":"2026-09-05T23:05:10.900Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}