{"record":{"id":"a193249bef17c65d","repo":"gastownhall/beads","slug":"transient-error-d-attempt-d-d","errorCode":null,"errorMessage":"transient error %d (attempt %d/%d)","messagePattern":"transient error (.+?) \\(attempt (.+?)/(.+?)\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"internal/ado/client.go","lineNumber":274,"sourceCode":"\t\tretriable := resp.StatusCode == http.StatusTooManyRequests ||\n\t\t\tresp.StatusCode >= 500\n\n\t\tif retriable && attempt < maxAttempts {\n\t\t\tdelay := RetryDelay * time.Duration(1<<uint(attempt))\n\t\t\tuseServerDelay := false\n\t\t\tif retryAfter := resp.Header.Get(\"Retry-After\"); retryAfter != \"\" {\n\t\t\t\tif seconds, parseErr := strconv.Atoi(retryAfter); parseErr == nil {\n\t\t\t\t\tdelay = time.Duration(seconds) * time.Second\n\t\t\t\t\tuseServerDelay = true\n\t\t\t\t}\n\t\t\t}\n\t\t\t// Only add jitter to our own exponential backoff, not server-mandated delays\n\t\t\tif !useServerDelay {\n\t\t\t\tif half := int64(delay / 2); half > 0 {\n\t\t\t\t\tdelay += time.Duration(rand.Int64N(half)) //nolint:gosec // G404: jitter for retry backoff does not need crypto rand\n\t\t\t\t}\n\t\t\t}\n\t\t\tlastErr = fmt.Errorf(\"transient error %d (attempt %d/%d)\", resp.StatusCode, attempt+1, maxAttempts+1)\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn nil, ctx.Err()\n\t\t\tcase <-time.After(delay):\n\t\t\t\tcontinue\n\t\t\t}\n\t\t}\n\n\t\treturn nil, &APIError{StatusCode: resp.StatusCode, Body: string(respBody)}\n\t}\n\n\treturn nil, fmt.Errorf(\"max retries (%d) exceeded: %w\", maxAttempts+1, lastErr)\n}\n\n// addAPIVersion appends the api-version query parameter to a URL string.\nfunc addAPIVersion(urlStr string) string {\n\tif strings.Contains(urlStr, \"?\") {\n\t\treturn urlStr + \"&api-version=\" + APIVersion","sourceCodeStart":256,"sourceCodeEnd":292,"githubUrl":"https://github.com/gastownhall/beads/blob/71377f276968b452ee607177637970a4ff888584/internal/ado/client.go#L256-L292","documentation":"The Azure DevOps client retry loop in doRequest hit a transient (retryable) HTTP status (e.g. 429, 5xx) and records this as lastErr on each failed attempt before backing off. The message includes the status code and attempt counter (attempt+1 of maxAttempts+1). It is usually surfaced wrapped by \"max retries exceeded\" (1408) rather than on its own; on its own it means the loop exited early via ctx cancellation. The client adds jittered exponential backoff, or honors a server-provided Retry-After delay when present.","triggerScenarios":"Any ADO REST call routed through doRequest (FetchWorkItems, fetchWorkItemsByWIQL, CreateWorkItem, UpdateWorkItem, AddWorkItemLink, RemoveWorkItemLink) receiving a retryable status on every attempt — typically 429 rate limiting or 500/502/503/504 from dev.azure.com.","commonSituations":"Bulk scripts hammering the WIQL/work-item API and tripping rate limits; shared PAT across many jobs exhausting org quotas; ADO service degradation/incidents; long batch operations where context deadline cancels mid-backoff.","solutions":["Retry the operation later with backoff; if you see it repeatedly, reduce request rate or batch IDs into fewer calls.","Check the HTTP status code embedded in the message: 429 means rate limiting — slow down or request a limit increase; 5xx suggests a server-side issue.","Verify your context/deadline isn't cutting retries short; increase the timeout so all retry attempts fit.","Check Azure DevOps service status for ongoing incidents.","Refresh/validate the PAT — 401/403-adjacent auth churn can masquerade as repeated failures."],"exampleFix":"// before\nfor _, id := range all3000IDs { c.FetchWorkItems(ctx, idsChunk300) } // hammers API, 429s\n// after\nfor i := 0; i < len(allIDs); i += 200 { c.FetchWorkItems(ctx, allIDs[i:i+200]); time.Sleep(500*time.Millisecond) }","handlingStrategy":"retry","validationCode":"// no pre-call check possible; optionally check ADO status endpoint\nif rateLimitProbeFail() { time.Sleep(30 * time.Second) }","typeGuard":null,"tryCatchPattern":"_, err := client.FetchWorkItems(ctx, ids)\nif err != nil {\n    var re *ado.RetryableError // or inspect wrapped status\n    if errors.As(err, &re) {\n        time.Sleep(backoffWithJitter(attempt))\n        // retry\n    }\n}","preventionTips":["Throttle client-side request rate; batch work-item ID fetches.","Honor Retry-After headers on 429 responses.","Spread bulk jobs across time windows instead of bursting.","Use a unique PAT per job to avoid shared quota exhaustion.","Monitor for transient-error messages in logs to detect chronic rate limiting."],"tags":["http","azure-devops","rate-limit","retry","transient"],"backgroundTag":"http-429-rate-limited","analyzedSha":"71377f276968b452ee607177637970a4ff888584","analyzedAt":"2026-08-30T18:55:39.744Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}