{"record":{"id":"c78a449cbde4aa4a","repo":"xai-org/grok-build","slug":"the-probe-s-output-did-not-drain-within-drain-gra","errorCode":null,"errorMessage":"the probe's output did not drain within {DRAIN_GRACE:?}","messagePattern":"the probe's output did not drain within (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/codegen/xai-fast-worktree/src/git/probe.rs","lineNumber":236,"sourceCode":"    }\n    drop(writer);\n    let (mut stdout, mut stderr) = (Vec::new(), Vec::new());\n    let (mut read_stdout, drained_by) = (false, Instant::now() + DRAIN_GRACE);\n    for _ in 0..reading {\n        match drained.recv_timeout(drained_by.saturating_duration_since(Instant::now())) {\n            Ok((Stream::Stdout, read)) => (stdout, read_stdout) = (read, true),\n            Ok((Stream::Stderr, read)) => stderr = read,\n            Err(_) => {\n                // The child exited but a grandchild still holds the pipes.\n                // Signalling the pgid is safe: that live grandchild keeps the\n                // group non-empty, so the group id cannot have been recycled.\n                if let Some(group) = group\n                    && let Err(error) = kill_group(group)\n                {\n                    tracing::warn!(%error, \"failed to kill process group still holding the probe's pipes\");\n                }\n                if !read_stdout {\n                    return Err(std::io::Error::new(\n                        ErrorKind::TimedOut,\n                        format!(\"the probe's output did not drain within {DRAIN_GRACE:?}\"),\n                    ));\n                }\n                tracing::warn!(\n                    ?DRAIN_GRACE,\n                    \"the probe's stderr did not drain; reporting none\"\n                );\n                break;\n            }\n        }\n    }\n    Ok(Output {\n        status: child.wait()?,\n        stdout,\n        stderr,\n    })\n}","sourceCodeStart":218,"sourceCodeEnd":254,"githubUrl":"https://github.com/xai-org/grok-build/blob/bc7f02eddd3d84085849dc19ed216f11c23b0571/crates/codegen/xai-fast-worktree/src/git/probe.rs#L218-L254","documentation":"In `run_with_timeout`, after the git probe command times out, the child process group is killed, but the probe's pipes can still be held open by grandchildren. The function waits DRAIN_GRACE for stdout to reach EOF; if it doesn't drain and it wasn't reading stdout successfully, it returns TimedOut. It means the probe process was killed but its output pipes never closed.","triggerScenarios":"A git command spawned by run_probe (via run_with_timeout) exceeds its timeout, the process group kill succeeds but an orphaned grandchild keeps the stdout/stderr pipe open past DRAIN_GRACE, and read_stdout is false at that point.","commonSituations":"git hooks or credential helpers spawning long-lived children that inherit the pipes; a hung `git fetch` over a stalled network with an ssh subprocess surviving the group kill; extremely slow filesystems making git commands exceed the timeout.","solutions":["Investigate what git subprocess (helper/hook/ssh) is inheriting the probe's pipes and hangs; use GIT_SSH_COMMAND with timeouts or disable hooks for probes.","Run git with stdio redirected away from inherited descriptors for spawned helpers (e.g. set GIT_TERMINAL_PROMPT=0, avoid interactive credential prompts).","Increase the probe timeout or DRAIN_GRACE if legitimate operations are just slow (NFS/slow disks).","Treat the TimedOut error as a probe failure and fall back to a non-probe code path, as callers like run_probe already handle errors."],"exampleFix":"// before\nlet out = probe.run(args).map_err(|e| anyhow!(\"probe failed: {e}\"))?;\n// after\nlet out = match probe.run(args) {\n    Ok(out) => out,\n    Err(e) if e.kind() == std::io::ErrorKind::TimedOut => {\n        tracing::warn!(\"git probe timed out; using conservative defaults\");\n        ProbeOutput::default()\n    }\n    Err(e) => return Err(e.into()),\n};","handlingStrategy":"fallback","validationCode":null,"typeGuard":"fn is_probe_drain_timeout(err: &std::io::Error) -> bool {\n    err.kind() == std::io::ErrorKind::TimedOut\n        && err.to_string().contains(\"did not drain within\")\n}","tryCatchPattern":"match probe.run(args) {\n    Ok(out) => out,\n    Err(e) if is_probe_drain_timeout(&e) => {\n        tracing::warn!(%e, \"probe hung; using default repo assumptions\");\n        ProbeOutput::default()\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Disable interactive hooks/credential helpers for probe git commands (GIT_TERMINAL_PROMPT=0)","Use ssh settings with ConnectTimeout for remote git operations","Keep probe timeouts generous on slow (NFS) filesystems","Ensure spawned git subprocesses do not inherit probe stdio pipes"],"tags":["git","timeout","process","io"],"backgroundTag":"pipe-drain-timeout","analyzedSha":"bc7f02eddd3d84085849dc19ed216f11c23b0571","analyzedAt":"2026-08-31T04:59:42.031Z","schemaVersion":2},"datasetVersion":"2026-08-31T09:17:48.483Z"}