{"record":{"id":"0c6912aff83c73e0","repo":"dgraph-io/dgraph","slug":"badger-streaming-failed-for-group-v-v","errorCode":null,"errorMessage":"badger streaming failed for group [%v]: %v","messagePattern":"badger streaming failed for group \\[(.+?)\\]: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"dgraph/cmd/dgraphimport/import_client.go","lineNumber":207,"sourceCode":"\t\t}\n\t}()\n\n\t// Send group ID as the first message in the stream\n\tglog.Infof(\"[import] Sending request for streaming external snapshot for group ID [%v]\", groupId)\n\tgroupReq := &api.StreamExtSnapshotRequest{GroupId: groupId}\n\tif err := out.Send(groupReq); err != nil {\n\t\treturn fmt.Errorf(\"failed to send request for group ID [%v] to the server: %w\", groupId, err)\n\t}\n\tif _, err := out.Recv(); err != nil {\n\t\treturn fmt.Errorf(\"failed to receive response for group ID [%v] from the server: %w\", groupId, err)\n\t}\n\n\tglog.Infof(\"[import] Group [%v]: Received ACK for sending group request\", groupId)\n\n\t// Configure and start the BadgerDB stream\n\tglog.Infof(\"[import] Starting BadgerDB stream for group [%v]\", groupId)\n\tif err := streamBadger(ctx, ps, out, groupId); err != nil {\n\t\treturn fmt.Errorf(\"badger streaming failed for group [%v]: %v\", groupId, err)\n\t}\n\treturn nil\n}\n\n// streamBadger runs a BadgerDB stream to send key-value pairs to the specified group.\n// It creates a new stream at the maximum sequence number and sends the data to the specified group.\n// It also sends a final 'done' signal to mark completion.\nfunc streamBadger(ctx context.Context, ps *badger.DB, out api.Dgraph_StreamExtSnapshotClient, groupId uint32) error {\n\tstream := ps.NewStreamAt(math.MaxUint64)\n\tstream.LogPrefix = \"[import] Sending external snapshot to group [\" + fmt.Sprintf(\"%d\", groupId) + \"]\"\n\tstream.KeyToList = nil\n\tstream.Send = func(buf *z.Buffer) error {\n\t\tp := &api.StreamPacket{Data: buf.Bytes()}\n\t\tif err := out.Send(&api.StreamExtSnapshotRequest{Pkt: p}); err != nil && !errors.Is(err, io.EOF) {\n\t\t\treturn fmt.Errorf(\"failed to send data chunk: %w\", err)\n\t\t}\n\t\tif _, err := out.Recv(); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to receive response for group ID [%v] from the server: %w\", groupId, err)","sourceCodeStart":189,"sourceCodeEnd":225,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/dgraph/cmd/dgraphimport/import_client.go#L189-L225","documentation":"Wrapper returned by streamSnapshotForGroup when streamBadger fails: Badger's Stream.Orchestrate, per-chunk send/recv, or the final done-signal exchange errored. It records the group ID so operators know which group's pdir stream aborted.","triggerScenarios":"Any failure inside streamBadger: stream.Orchestrate error, Send/Recv failures per chunk, done-signal send failure, or context cancellation during the loop — all surfaced through this wrap.","commonSituations":"Long-running stream killed by network interruption mid-transfer; Alpha OOM/crash while receiving packets; context cancelled because import deadline too small for the dataset size; badger read errors from a corrupted export directory.","solutions":["Unwrap the message to find the inner stage (orchestration vs send/recv vs done signal) and its gRPC/badger cause.","Check Alpha logs and health around the failure time (OOM, restart, disk full on server).","Re-run the import for that group; snapshot streaming is all-or-nothing per group.","Increase the context deadline to cover the full transfer of the pdir.","Validate the pdir's badger DB integrity before streaming (open read-only and iterate a few keys)."],"exampleFix":"// before\nif err := streamBadger(ctx, ps, out, groupId); err != nil {\n\treturn fmt.Errorf(\"badger streaming failed for group [%v]: %v\", groupId, err)\n}\n// after\nif err := streamBadger(ctx, ps, out, groupId); err != nil {\n\tif ctx.Err() != nil {\n\t\treturn fmt.Errorf(\"badger streaming for group %d cancelled/timed out: %w\", groupId, ctx.Err())\n\t}\n\treturn fmt.Errorf(\"badger streaming failed for group [%v]: %v\", groupId, err)\n}","handlingStrategy":"fallback","validationCode":"db, err := badger.OpenManaged(badger.DefaultOptions(pdir).WithReadOnly(true))\nif err != nil {\n\treturn fmt.Errorf(\"precheck: pdir unusable: %w\", err)\n}\n_ = db.Close()","typeGuard":null,"tryCatchPattern":"if err := streamBadger(ctx, ps, out, groupId); err != nil {\n\tlog.Printf(\"group %d stream failed (%v); scheduling full retry after fixing cause\", groupId, err)\n\treturn scheduleGroupImportRetry(ctx, dc, pdir, groupId)\n}","preventionTips":["Pre-validate the pdir opens cleanly before streaming.","Monitor Alpha resources (memory, disk) during imports.","Use deadlines proportional to pdir size.","Treat per-group streams as atomic; retry the whole group on failure."],"tags":["badger","grpc","stream","snapshot-import"],"backgroundTag":"badger-stream-failed","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}