{"record":{"id":"f22121962719395e","repo":"dgraph-io/dgraph","slug":"downstream-d-closed-before-finish-true","errorCode":null,"errorMessage":"downstream(%d) closed before Finish=true","messagePattern":"downstream\\((.+?)\\) closed before Finish=true","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"worker/import.go","lineNumber":427,"sourceCode":"\t\tif req.Pkt == nil {\n\t\t\treturn fmt.Errorf(\"unexpected empty request\")\n\t\t}\n\n\t\tif req.Pkt.Done {\n\t\t\t// Forward Done, half-close downstream send.\n\t\t\tif err := out.Send(&api.StreamExtSnapshotRequest{Pkt: req.Pkt}); err != nil && !errors.Is(err, io.EOF) {\n\t\t\t\treturn fmt.Errorf(\"send done downstream(%d): %w\", groupId, err)\n\t\t\t}\n\t\t\t_ = out.CloseSend()\n\n\t\t\t// Drain downstream and relay upstream until Finish=true.\n\t\t\tfor {\n\t\t\t\tif err := ctx.Err(); err != nil {\n\t\t\t\t\treturn err\n\t\t\t\t}\n\t\t\t\tresp, err := out.Recv()\n\t\t\t\tif errors.Is(err, io.EOF) {\n\t\t\t\t\treturn fmt.Errorf(\"downstream(%d) closed before Finish=true\", groupId)\n\t\t\t\t}\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"recv final downstream(%d): %w\", groupId, err)\n\t\t\t\t}\n\t\t\t\tif err := in.Send(resp); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"relay final upstream: %w\", err)\n\t\t\t\t}\n\t\t\t\tif resp.Finish {\n\t\t\t\t\tglog.Infof(\"[import] [forward %d -> %d] finish\", currentGroup, groupId)\n\t\t\t\t\treturn nil\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\t// Normal data chunk: send -> wait ack -> send upstream ack.\n\t\tif err := out.Send(&api.StreamExtSnapshotRequest{Pkt: req.Pkt}); err != nil {\n\t\t\treturn fmt.Errorf(\"send data downstream(%d): %w\", groupId, err)\n\t\t}","sourceCodeStart":409,"sourceCodeEnd":445,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/worker/import.go#L409-L445","documentation":"After half-closing the downstream send, the relay drains the leader's responses waiting for the frame with Finish=true. If out.Recv() returns io.EOF instead, the leader closed its side without signalling completion, so the import result is unknown and this error is returned.","triggerScenarios":"The leader's StreamExtSnapshot server handler returned or closed the stream before sending Finish=true: it errored on its own snapshot pipeline, was killed, or its context was cancelled.","commonSituations":"Leader hit an internal error (disk full, memory) mid-finalization; leader restarted at the very end of the import; a proxy closed the connection after inactivity during the leader's final flush.","solutions":["Check the leader's logs for the error that ended its handler before Finish","Re-run the import; the snapshot state may be incomplete","Ensure the leader has adequate disk/memory for finalizing an external snapshot","Remove idle-timeout intermediaries between the two alphas"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// ensure leader resources before import so finalization can complete\nprecheckDiskAndMemory(leaderHost)","typeGuard":"func finishedCorrectly(resp *api.StreamExtSnapshotResponse) bool {\n    return resp != nil && resp.Finish // only Finish=true means success\n}","tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"closed before Finish=true\") {\n    // outcome unknown: check leader logs, then re-run the import\n}","preventionTips":["Never kill or upgrade the leader mid-import","Give the leader disk/memory headroom for finalization","Remove proxies that close quiet connections during final flush","Treat any Finish-less close as a failed import and retry"],"tags":["grpc","streaming","protocol","incomplete"],"backgroundTag":"stream-closed-before-completion","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}