{"record":{"id":"d3fc48a071fc2a91","repo":"argoproj/argo-workflows","slug":"reconnection-failed-after-d-retries-last-error","errorCode":null,"errorMessage":"reconnection failed after %d retries, last error: %w","messagePattern":"reconnection failed after (.+?) retries, last error: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"util/sqldb/session.go","lineNumber":365,"sourceCode":"\n\t\t// If this is the last attempt, don't wait\n\t\tif attempt == sp.maxRetries || !sp.isNetworkError(err) {\n\t\t\tbreak\n\t\t}\n\n\t\t// Calculate delay for next retry with linear backoff\n\t\tdelay := time.Duration(float64(sp.baseDelay) * float64(attempt+1) * sp.retryMultiple)\n\t\tdelay = min(delay, sp.maxDelay)\n\n\t\t// Wait before retrying with context cancellation support\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn ctx.Err()\n\t\tcase <-time.After(delay):\n\t\t}\n\t}\n\n\treturn fmt.Errorf(\"reconnection failed after %d retries, last error: %w\", sp.maxRetries, err)\n}\n\n// Session returns the underlying session. Use With() for operations that need reconnection.\n// This method is provided for cases where you need direct access to the session,\n// but it won't provide automatic reconnection.\nfunc (sp *SessionProxy) Session() db.Session {\n\tsp.mu.RLock()\n\tdefer sp.mu.RUnlock()\n\treturn sp.sess\n}\n\n// Close closes the session proxy and underlying session\nfunc (sp *SessionProxy) Close() error {\n\tsp.mu.Lock()\n\tdefer sp.mu.Unlock()\n\n\tif sp.closed {\n\t\treturn nil","sourceCodeStart":347,"sourceCodeEnd":383,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/util/sqldb/session.go#L347-L383","documentation":"reconnectLocked exhausted all retry attempts (maxRetries with linear backoff, bounded by maxDelay) trying to re-establish the database session and every attempt failed; err wraps the last connection error. Callers (reconnectIfStale within With, or explicit Reconnect) surface this wrapped cause.","triggerScenarios":"connect(ctx) fails maxRetries+1 consecutive times — DB down, unreachable, or rejecting credentials — during automatic reconnection after a network error, or during an explicit Reconnect call.","commonSituations":"Extended database outage (pod crash-looping, node down); misrotated DB credentials in k8s secrets; NetworkPolicy/security-group blocking the controller; context deadline exceeded during backoff; non-network errors breaking out of the retry loop on the first attempt.","solutions":["Unwrap the last error (%w) to see why connect failed and fix that root cause (network, DNS, auth, DB health).","Confirm DB credentials/username+password secrets are current after rotations.","Increase maxRetries / baseDelay / maxDelay in SessionProxy configuration for longer outages.","Ensure the calling context is not cancelled/expiring before the backoff loop finishes.","Restore the database itself (check StatefulSet/pod status, DB logs, max_connections)."],"exampleFix":"// before\nproxy := NewSessionProxy(connect, 2, 500*time.Millisecond, 1.0, 2*time.Second) // fails on long outages\n// after\nproxy := NewSessionProxy(connect, 10, 500*time.Millisecond, 1.0, 30*time.Second) // ~ ride out restarts","handlingStrategy":"retry","validationCode":"// preflight connectivity check before issuing operations\nconn, err := net.DialTimeout(\"tcp\", dbHostPort, 3*time.Second)\nif err != nil {\n\treturn fmt.Errorf(\"database unreachable before operation: %w\", err)\n}\nconn.Close()","typeGuard":null,"tryCatchPattern":"if err := proxy.Reconnect(ctx); err != nil {\n\tif strings.Contains(err.Error(), \"reconnection failed after\") {\n\t\t// retry budget exhausted; escalate / circuit-break\n\t\tlogger.Error(ctx, \"db unavailable after retries\", err)\n\t}\n}","preventionTips":["Tune maxRetries and backoff to survive typical DB pod restart durations.","Alert on the database itself (readiness probes, max_connections, disk) — this error is usually a DB-side outage.","Keep credential secrets in sync with rotations; auth failures abort retries immediately (non-network error).","Use a circuit breaker upstream so a dead DB doesn't pile up retry storms."],"tags":["database","reconnection","retry-exhausted","network"],"backgroundTag":"db-reconnect-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}