What causes "produced non-UTF-8 output on stdout" in xai-org/grok-build?

Ensure the provider writes only plain UTF-8 text (the token or JSON) to stdout and everything else to stderr.

produced non-UTF-8 output on stdout

Error message

produced non-UTF-8 output on stdout

What it means

parse_token_output requires the external auth provider's stdout to be valid UTF-8 before it can be interpreted as a bare token or JSON payload. If std::str::from_utf8 on output.stdout fails, this error is thrown (wrapped by the caller as 'external auth provider `<command>`: produced non-UTF-8 output on stdout').

Source

Thrown at crates/codegen/xai-grok-shell/src/auth/token_output.rs:54

pub(crate) struct ParsedTokenOutput {
    pub access_token: String,
    pub refresh_token: Option<String>,
    pub expires_at: Option<chrono::DateTime<chrono::Utc>>,
    pub issuer: Option<String>,
}

/// Accepts a bare token or JSON `{access_token, expires_in, issuer, ...}`. A
/// non-zero exit, non-UTF-8 or empty stdout, an empty `access_token`, or
/// JSON-object output that is not a valid token payload are all errors, so a
/// malformed mint fails closed rather than putting garbage on the wire.
pub(crate) fn parse_token_output(
    output: &std::process::Output,
) -> anyhow::Result<ParsedTokenOutput> {
    if !output.status.success() {
        anyhow::bail!("exited with {}", output.status);
    }
    let stdout = std::str::from_utf8(&output.stdout)
        .map_err(|_| anyhow::anyhow!("produced non-UTF-8 output on stdout"))?
        .trim();
    if stdout.is_empty() {
        anyhow::bail!("produced no output on stdout");
    }

    // Output that starts with `{` is meant to be a token payload: require it to
    // parse and carry a non-empty access_token. Anything else is a bare token
    // (JWTs and opaque tokens never start with `{`), so an error object like
    // `{"error":"expired"}` can never be mistaken for a bearer.
    if stdout.starts_with('{') {
        let parsed: ExternalAuthOutput = serde_json::from_str(stdout)
            .map_err(|e| anyhow::anyhow!("produced JSON that is not a token payload: {e}"))?;
        let access_token = parsed.access_token.trim().to_owned();
        if access_token.is_empty() {
            anyhow::bail!("produced JSON with an empty access_token");
        }
        reject_control_chars(&access_token)?;
        tracing::debug!(

View on GitHub (pinned to bc7f02eddd)

Solutions

Ensure the provider writes only plain UTF-8 text (the token or JSON) to stdout and everything else to stderr.
Disable progress/spinner output when stdout is not a TTY (e.g. respect NO_COLOR/CI, use --quiet).
Remove filters/transformers in the provider pipeline that alter encoding (e.g. ensure no binary-safe tools mangle output).
If tokens must be binary, base64-encode them so the payload remains ASCII.

Example fix

// before (provider)
printf '\x1b[1mFetching token...\x1b[0m' && cat token.bin
// after
echo 'Fetching token...' >&2 && base64 -w0 token.bin

Defensive patterns

Strategy: validation

Validate before calling

const buf = execFileSync(provider_cmd); // raw bytes
try { new TextDecoder('utf-8', { fatal: true }).decode(buf); }
catch { throw new Error('provider stdout is not UTF-8; fix provider output'); }

Type guard

function isUtf8(bytes) {
  try { new TextDecoder('utf-8', { fatal: true }).decode(bytes); return true; }
  catch { return false; }
}

Try / catch

match run_auth_flow(...).await {
    Err(e) if e.to_string().contains("non-UTF-8 output on stdout") => {
        eprintln!("Provider emitted binary/progress output on stdout — quiet mode or fix encoding, then retry");
    }
    other => other?,
}

Prevention

Silence progress bars/spinners when stdout is not a TTY
Emit only UTF-8 text on stdout; all diagnostics to stderr
Base64-encode if binary payloads are unavoidable
Test provider output with `provider | iconv -f utf-8 -t utf-8`

When it happens

Trigger: The provider writes binary data (progress bars with ANSI/binary escapes, gzipped output, credentials encrypted/binary-encoded) to stdout, so the raw bytes are not valid UTF-8.

Common situations: Provider script emits a binary logo or progress spinner; provider writes a compressed or binary-encoded token; locale/encoding issues corrupting multi-byte characters; provider piping through a tool that transforms encoding.

Related errors

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/28d5599200f1c83d. Report an issue: GitHub.