dgraph-io/dgraph · error

failed to initiate external snapshot stream: %v

Error message

failed to initiate external snapshot stream: %v

What it means

The server rejected the request to start the external snapshot streaming session (UpdateExtSnapshotStreamingState with Start=true). The client retries only 'overloaded' errors for up to 60s; any other error (or exhausted deadline) is wrapped and returned. The cluster stays out of drain mode and streaming never begins.

Source

Thrown at dgraph/cmd/dgraphimport/import_client.go:96

	glog.Info("[import] Initiating external snapshot stream")
	req := &api.UpdateExtSnapshotStreamingStateRequest{
		Start: true,
	}

	const maxRetryDuration = 60 * time.Second
	deadline := time.Now().Add(maxRetryDuration)
	retryDelay := time.Second

	for {
		resp, err := dc.UpdateExtSnapshotStreamingState(ctx, req)
		if err == nil {
			glog.Info("[import] External snapshot stream initiated successfully")
			return resp, nil
		}

		if !isRetryableError(err) || time.Now().After(deadline) {
			glog.Errorf("[import] failed to initiate external snapshot stream: %v", err)
			return nil, fmt.Errorf("failed to initiate external snapshot stream: %v", err)
		}

		glog.Warningf("[import] transient error initiating snapshot stream, retrying in %v: %v", retryDelay, err)
		select {
		case <-ctx.Done():
			return nil, ctx.Err()
		case <-time.After(retryDelay):
		}
		retryDelay = min(retryDelay*2, 10*time.Second)
	}
}

// streamSnapshot takes a p directory and a set of group IDs and streams the data from the
// p directory to the corresponding group IDs. It first scans the provided directory for
// subdirectories named with numeric group IDs.
func streamSnapshot(ctx context.Context, dc api.DgraphClient, baseDir string, groups []uint32) error {
	glog.Infof("[import] Starting to stream snapshot from directory: %s", baseDir)

View on GitHub (pinned to 759e242be6)

Solutions

1. Inspect the wrapped cause (%v) for the server-side message (no leader, permission denied, overloaded...).
2. If 'overloaded': reduce write load, wait for Raft proposals to drain, then retry; the client already retries for 60s.
3. Verify cluster health (Zero leaders, Alpha group health) before initiating streaming.
4. Check ACL/auth setup so the gRPC call is authorized; confirm endpoint points at a live Alpha.

Example fix

// before
err := dgraphimport.Import(ctx, "alpha1:9080", outDir) // cluster has no Zero quorum
// after
// ensure zero quorum & leaders first:
//   kubectl exec zero-0 -- curl localhost:6080/state
err := dgraphimport.Import(ctx, "alpha1:9080", outDir)

// health-check before initiating
resp, err := http.Get("http://alpha1:8080/health")
if err != nil || resp.StatusCode != http.StatusOK {
    return fmt.Errorf("alpha not healthy; aborting snapshot import")
}

err := dgraphimport.Import(ctx, addr, outDir)
if err != nil && strings.Contains(err.Error(), "failed to initiate external snapshot stream") {
    if strings.Contains(err.Error(), "overloaded") {
        time.Sleep(30 * time.Second) // let Raft drain, then retry once
        err = dgraphimport.Import(ctx, addr, outDir)
    }
}
return err

When it happens

Trigger: Alpha/Zero returns an error on UpdateExtSnapshotStreamingState(Start=true): connectivity loss, non-retryable server error, or 'overloaded with pending proposals' persisting beyond the 60s deadline; also raised when a non-retryable error occurs on the first attempt.

Common situations: Cluster under heavy Raft load (many writes during import), quorum loss / no leader elected, wrong endpoint, ACL interceptors rejecting the call, context canceled mid-retry (surfaced separately as ctx.Err).

Related errors

• failed to turn off drain mode: %v
• unable to connect to the leader of group [%v] : %v
• connection string cannot be empty
• failed to connect to endpoint [%s]: %w
• Unhealthy connection to %v

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/e55fd56e8eb4ac0f. Report an issue: GitHub.