{"record":{"id":"4429b8be1e5acd4d","repo":"vitessio/vitess","slug":"failed-to-automatically-buffer-and-retry-failed-re","errorCode":null,"errorMessage":"failed to automatically buffer and retry failed request during failover. original err (type=%T): %v","messagePattern":"failed to automatically buffer and retry failed request during failover\\. original err \\(type=%T\\): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtgate/tabletgateway.go","lineNumber":373,"sourceCode":"\t\t// Check if we should buffer PRIMARY queries which failed due to an ongoing failover.\n\t\t// Note: We only buffer once and only \"!inTransaction\" queries i.e.\n\t\t// a) no transaction is necessary (e.g. critical reads) or\n\t\t// b) no transaction was created yet.\n\t\tif gw.buffer != nil && !bufferedOnce && !opts.InTransaction && target.TabletType == topodatapb.TabletType_PRIMARY {\n\t\t\t// The next call blocks if we should buffer during a failover.\n\t\t\tretryDone, bufferErr := gw.buffer.WaitForFailoverEnd(ctx, target.Keyspace, target.Shard, gw.kev, err)\n\n\t\t\t// Request may have been buffered.\n\t\t\tif retryDone != nil {\n\t\t\t\t// We're going to retry this request as part of a buffer drain.\n\t\t\t\t// Notify the buffer after we retried.\n\t\t\t\tdefer retryDone()\n\t\t\t\tbufferedOnce = true\n\t\t\t}\n\n\t\t\tif bufferErr != nil {\n\t\t\t\terr = vterrors.Wrapf(bufferErr,\n\t\t\t\t\t\"failed to automatically buffer and retry failed request during failover. original err (type=%T): %v\",\n\t\t\t\t\terr, err)\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\n\t\ttablets := gw.hc.GetHealthyTabletStats(target)\n\t\tif len(tablets) == 0 {\n\t\t\t// if we have a keyspace event watcher, check if the reason why our primary is not available is that it's currently being resharded\n\t\t\t// or if a reparent operation is in progress.\n\t\t\t// We only check for whether reshard is ongoing or primary is serving or not, only if the target is primary. We don't want to buffer\n\t\t\t// replica queries, so it doesn't make any sense to check for resharding or reparenting in that case.\n\t\t\tif kev := gw.kev; kev != nil && target.TabletType == topodatapb.TabletType_PRIMARY {\n\t\t\t\tif kev.TargetIsBeingResharded(ctx, target) {\n\t\t\t\t\tlog.V(2).Info(fmt.Sprintf(\"current keyspace is being resharded, retrying: %s: %s\", target.Keyspace, debug.Stack()))\n\t\t\t\t\terr = vterrors.Errorf(vtrpcpb.Code_CLUSTER_EVENT, buffer.ClusterEventReshardingInProgress)\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tprimary, shouldBuffer := kev.ShouldStartBufferingForTarget(ctx, target)","sourceCodeStart":355,"sourceCodeEnd":391,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtgate/tabletgateway.go#L355-L391","documentation":"This error is produced by TabletGateway.withRetry when buffering a failed PRIMARY query during a failover fails: WaitForFailoverEnd returns a bufferErr (e.g. the failover buffer is full, the buffer's redis/go-mem-manager is unavailable, or buffering was denied) and the gateway wraps it together with the original query error. The message preserves both the failover-buffer failure and the original error (with its Go type) that triggered the buffering attempt.","triggerScenarios":"A non-transactional query targeting a PRIMARY tablet fails (e.g. during reparent), the gateway's failover buffer is enabled (gw.buffer != nil), the request has not been buffered yet, and gw.buffer.WaitForFailoverEnd returns an error — e.g. buffer capacity exceeded, buffer shut down mid-wait, or context cancelled while waiting for failover end.","commonSituations":"Keyspace reparenting while vtgate buffering is enabled and the buffer overflows or is disabled/restarted; vtgate shutdown concurrent with a failover; misconfigured buffer settings (too small) during heavy failover traffic; clients seeing wrapped errors like 'failed to automatically buffer and retry ... original err (type=*vterrors.VtError): ...'.","solutions":["Inspect the wrapped 'original err' in the message — it is the root query failure (often a reparent-in-progress cluster event) — and resolve the underlying failover issue.","Check vtgate logs for buffer errors ('failed to buffer' / buffer full) and tune buffer settings (size/drain concurrency) if overflow is recurring.","Retry the request after the failover completes; the error is typically transient for non-transactional reads.","Verify vtgate and the buffer are healthy (vtgate restart during the event can abort buffering); pin to a stable vtgate instance or enable client-side retry with backoff for retryable codes (CLUSTER_EVENT)."],"exampleFix":"// before: single-shot query, fails during reparent\nerr := execute(ctx, target, query)\n// after: retry retryable cluster-event failures with backoff\nerr := executeWithRetry(ctx, target, query, func(err error) bool {\n    code := vterrors.Code(err)\n    return code == vtrpcpb.Code_CLUSTER_EVENT || code == vtrpcpb.Code_UNAVAILABLE\n})","handlingStrategy":"retry","validationCode":"// Check failover state before issuing PRIMARY queries\nif kev, _ := keyspaceEvents.Get(ctx, target.Keyspace); kev != nil && kev.IsReparenting() {\n    return ErrFailoverInProgress // delay or route elsewhere\n}","typeGuard":null,"tryCatchPattern":"err := exec.Query(ctx, target, query)\nif err != nil {\n    code := vterrors.Code(err)\n    if code == vtrpcpb.Code_CLUSTER_EVENT || code == vtrpcpb.Code_UNAVAILABLE {\n        // failover-related; retry with backoff after reparent completes\n        return retryWithBackoff(ctx, query, 3)\n    }\n    return err\n}","preventionTips":["Size the failover buffer for peak PRIMARY traffic so buffering requests are not rejected.","Avoid restarting vtgate during planned reparents; keep buffer configuration stable.","Enable client-side retries only for retryable codes (CLUSTER_EVENT, UNAVAILABLE) and never for transactions already in flight.","Monitor buffer usage metrics and alert on capacity or drain-latency issues before failures occur."],"tags":["vtgate","failover","buffering","retry"],"backgroundTag":"failover-buffer-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}