{"record":{"id":"967b75c29d661e6b","repo":"argoproj/argo-workflows","slug":"operation-failed-after-reconnection-w","errorCode":null,"errorMessage":"operation failed after reconnection: %w","messagePattern":"operation failed after reconnection: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"util/sqldb/session.go","lineNumber":303,"sourceCode":"\t// If it's not a network error or inside a tx do not retry\n\tif !sp.isNetworkError(err) || sp.insideTransaction {\n\t\treturn err\n\t}\n\n\tif reconnectErr := sp.reconnectIfStale(ctx, sess); reconnectErr != nil {\n\t\treturn fmt.Errorf(\"operation failed and reconnection failed: %w\", reconnectErr)\n\t}\n\n\tsp.mu.RLock()\n\tsess = sp.sess\n\tsp.mu.RUnlock()\n\n\tif sess == nil {\n\t\treturn fmt.Errorf(\"no active session after reconnection\")\n\t}\n\n\tif retryErr := fn(sess); retryErr != nil {\n\t\treturn fmt.Errorf(\"operation failed after reconnection: %w\", retryErr)\n\t}\n\n\treturn nil\n}\n\n// reconnectIfStale reconnects only if the current session is still the\n// same one that produced the error. If another goroutine already\n// reconnected (sp.sess != staleSess), this is a no-op.\nfunc (sp *SessionProxy) reconnectIfStale(ctx context.Context, staleSess db.Session) error {\n\tsp.mu.Lock()\n\tdefer sp.mu.Unlock()\n\tif sp.sess != staleSess {\n\t\treturn nil\n\t}\n\treturn sp.reconnectLocked(ctx)\n}\n\n// Reconnect performs reconnection with retry logic and linear backoff","sourceCodeStart":285,"sourceCodeEnd":321,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/util/sqldb/session.go#L285-L321","documentation":"SessionProxy reconnected successfully after a network error and retried the caller's function on the new session, but the retry also failed. The retry error is wrapped with this prefix; the original error was discarded in favor of the more relevant post-reconnect failure.","triggerScenarios":"fn(sess) fails a second time on the fresh session after automatic reconnection — i.e. the failure is not purely transient connectivity (e.g. query error, constraint violation, permission error, or DB that reconnects at TCP level but rejects queries).","commonSituations":"DB restored but schema/permissions changed; the operation is non-idempotent and partially applied before the network blip; the new session lands on a replica or different DB with different state; context deadline expires during retry.","solutions":["Inspect the wrapped cause of the retry error — it reflects the real problem now that connectivity is restored.","Make operations idempotent (upserts) so a retry after reconnect is safe.","Check DB-side state: permissions, schema, and whether the retry hit a context cancellation/deadline.","If the retry always fails while the first attempt succeeded historically, compare connection config (DB name, user, TLS) between old and new sessions."],"exampleFix":"// before\nerr := proxy.With(ctx, save) // \"operation failed after reconnection: pq: duplicate key value violates unique constraint\"\n// after\n// make save idempotent\nsess.Insert(orUpdate(wf)) // INSERT ... ON CONFLICT (name) DO UPDATE","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := proxy.With(ctx, fn)\nif err != nil {\n\tvar postReconn bool\n\tif strings.Contains(err.Error(), \"operation failed after reconnection\") {\n\t\tpostReconn = true\n\t}\n\tif postReconn {\n\t\t// connectivity is fine; the failure is application/DB-level — do not blind-retry\n\t\treturn classifyAndHandle(err)\n\t}\n}","preventionTips":["Make write operations idempotent (upsert/ON CONFLICT) so reconnect-retries are safe.","Check DB grants/schema after maintenance windows; reconnect lands on the 'fixed' DB.","Watch for context deadlines expiring across the first attempt + reconnect + retry path.","Log both failures (first and retry) for diagnosis — the retry cause is the actionable one."],"tags":["database","reconnection","retry"],"backgroundTag":"db-reconnect-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}