xai-org/grok-build · error

the probe's output did not drain within {DRAIN_GRACE:?}

Error message

the probe's output did not drain within {DRAIN_GRACE:?}

What it means

In `run_with_timeout`, after the git probe command times out, the child process group is killed, but the probe's pipes can still be held open by grandchildren. The function waits DRAIN_GRACE for stdout to reach EOF; if it doesn't drain and it wasn't reading stdout successfully, it returns TimedOut. It means the probe process was killed but its output pipes never closed.

Source

Thrown at crates/codegen/xai-fast-worktree/src/git/probe.rs:236

    }
    drop(writer);
    let (mut stdout, mut stderr) = (Vec::new(), Vec::new());
    let (mut read_stdout, drained_by) = (false, Instant::now() + DRAIN_GRACE);
    for _ in 0..reading {
        match drained.recv_timeout(drained_by.saturating_duration_since(Instant::now())) {
            Ok((Stream::Stdout, read)) => (stdout, read_stdout) = (read, true),
            Ok((Stream::Stderr, read)) => stderr = read,
            Err(_) => {
                // The child exited but a grandchild still holds the pipes.
                // Signalling the pgid is safe: that live grandchild keeps the
                // group non-empty, so the group id cannot have been recycled.
                if let Some(group) = group
                    && let Err(error) = kill_group(group)
                {
                    tracing::warn!(%error, "failed to kill process group still holding the probe's pipes");
                }
                if !read_stdout {
                    return Err(std::io::Error::new(
                        ErrorKind::TimedOut,
                        format!("the probe's output did not drain within {DRAIN_GRACE:?}"),
                    ));
                }
                tracing::warn!(
                    ?DRAIN_GRACE,
                    "the probe's stderr did not drain; reporting none"
                );
                break;
            }
        }
    }
    Ok(Output {
        status: child.wait()?,
        stdout,
        stderr,
    })
}

View on GitHub (pinned to bc7f02eddd)

Solutions

1. Investigate what git subprocess (helper/hook/ssh) is inheriting the probe's pipes and hangs; use GIT_SSH_COMMAND with timeouts or disable hooks for probes.
2. Run git with stdio redirected away from inherited descriptors for spawned helpers (e.g. set GIT_TERMINAL_PROMPT=0, avoid interactive credential prompts).
3. Increase the probe timeout or DRAIN_GRACE if legitimate operations are just slow (NFS/slow disks).
4. Treat the TimedOut error as a probe failure and fall back to a non-probe code path, as callers like run_probe already handle errors.

Example fix

// before
let out = probe.run(args).map_err(|e| anyhow!("probe failed: {e}"))?;
// after
let out = match probe.run(args) {
    Ok(out) => out,
    Err(e) if e.kind() == std::io::ErrorKind::TimedOut => {
        tracing::warn!("git probe timed out; using conservative defaults");
        ProbeOutput::default()
    }
    Err(e) => return Err(e.into()),
};

fn is_probe_drain_timeout(err: &std::io::Error) -> bool {
    err.kind() == std::io::ErrorKind::TimedOut
        && err.to_string().contains("did not drain within")
}

match probe.run(args) {
    Ok(out) => out,
    Err(e) if is_probe_drain_timeout(&e) => {
        tracing::warn!(%e, "probe hung; using default repo assumptions");
        ProbeOutput::default()
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: A git command spawned by run_probe (via run_with_timeout) exceeds its timeout, the process group kill succeeds but an orphaned grandchild keeps the stdout/stderr pipe open past DRAIN_GRACE, and read_stdout is false at that point.

Common situations: git hooks or credential helpers spawning long-lived children that inherit the pipes; a hung `git fetch` over a stalled network with an ssh subprocess surviving the group kill; extremely slow filesystems making git commands exceed the timeout.

Related errors

• git {shown} failed in {}: {}
• git fetch origin {oid} skipped: restore fetch budget exhaust
• git fetch --no-tags origin {spec} timed out after {}s{}{}
• reading command stdout: {e}
• external auth provider `{command}` timed out after 300s

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/c78a449cbde4aa4a. Report an issue: GitHub.