{"record":{"id":"28d5599200f1c83d","repo":"xai-org/grok-build","slug":"produced-non-utf-8-output-on-stdout","errorCode":null,"errorMessage":"produced non-UTF-8 output on stdout","messagePattern":"produced non-UTF-8 output on stdout","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/codegen/xai-grok-shell/src/auth/token_output.rs","lineNumber":54,"sourceCode":"pub(crate) struct ParsedTokenOutput {\n    pub access_token: String,\n    pub refresh_token: Option<String>,\n    pub expires_at: Option<chrono::DateTime<chrono::Utc>>,\n    pub issuer: Option<String>,\n}\n\n/// Accepts a bare token or JSON `{access_token, expires_in, issuer, ...}`. A\n/// non-zero exit, non-UTF-8 or empty stdout, an empty `access_token`, or\n/// JSON-object output that is not a valid token payload are all errors, so a\n/// malformed mint fails closed rather than putting garbage on the wire.\npub(crate) fn parse_token_output(\n    output: &std::process::Output,\n) -> anyhow::Result<ParsedTokenOutput> {\n    if !output.status.success() {\n        anyhow::bail!(\"exited with {}\", output.status);\n    }\n    let stdout = std::str::from_utf8(&output.stdout)\n        .map_err(|_| anyhow::anyhow!(\"produced non-UTF-8 output on stdout\"))?\n        .trim();\n    if stdout.is_empty() {\n        anyhow::bail!(\"produced no output on stdout\");\n    }\n\n    // Output that starts with `{` is meant to be a token payload: require it to\n    // parse and carry a non-empty access_token. Anything else is a bare token\n    // (JWTs and opaque tokens never start with `{`), so an error object like\n    // `{\"error\":\"expired\"}` can never be mistaken for a bearer.\n    if stdout.starts_with('{') {\n        let parsed: ExternalAuthOutput = serde_json::from_str(stdout)\n            .map_err(|e| anyhow::anyhow!(\"produced JSON that is not a token payload: {e}\"))?;\n        let access_token = parsed.access_token.trim().to_owned();\n        if access_token.is_empty() {\n            anyhow::bail!(\"produced JSON with an empty access_token\");\n        }\n        reject_control_chars(&access_token)?;\n        tracing::debug!(","sourceCodeStart":36,"sourceCodeEnd":72,"githubUrl":"https://github.com/xai-org/grok-build/blob/bc7f02eddd3d84085849dc19ed216f11c23b0571/crates/codegen/xai-grok-shell/src/auth/token_output.rs#L36-L72","documentation":"parse_token_output requires the external auth provider's stdout to be valid UTF-8 before it can be interpreted as a bare token or JSON payload. If std::str::from_utf8 on output.stdout fails, this error is thrown (wrapped by the caller as 'external auth provider `<command>`: produced non-UTF-8 output on stdout').","triggerScenarios":"The provider writes binary data (progress bars with ANSI/binary escapes, gzipped output, credentials encrypted/binary-encoded) to stdout, so the raw bytes are not valid UTF-8.","commonSituations":"Provider script emits a binary logo or progress spinner; provider writes a compressed or binary-encoded token; locale/encoding issues corrupting multi-byte characters; provider piping through a tool that transforms encoding.","solutions":["Ensure the provider writes only plain UTF-8 text (the token or JSON) to stdout and everything else to stderr.","Disable progress/spinner output when stdout is not a TTY (e.g. respect NO_COLOR/CI, use --quiet).","Remove filters/transformers in the provider pipeline that alter encoding (e.g. ensure no binary-safe tools mangle output).","If tokens must be binary, base64-encode them so the payload remains ASCII."],"exampleFix":"// before (provider)\nprintf '\\x1b[1mFetching token...\\x1b[0m' && cat token.bin\n// after\necho 'Fetching token...' >&2 && base64 -w0 token.bin","handlingStrategy":"validation","validationCode":"const buf = execFileSync(provider_cmd); // raw bytes\ntry { new TextDecoder('utf-8', { fatal: true }).decode(buf); }\ncatch { throw new Error('provider stdout is not UTF-8; fix provider output'); }","typeGuard":"function isUtf8(bytes) {\n  try { new TextDecoder('utf-8', { fatal: true }).decode(bytes); return true; }\n  catch { return false; }\n}","tryCatchPattern":"match run_auth_flow(...).await {\n    Err(e) if e.to_string().contains(\"non-UTF-8 output on stdout\") => {\n        eprintln!(\"Provider emitted binary/progress output on stdout — quiet mode or fix encoding, then retry\");\n    }\n    other => other?,\n}","preventionTips":["Silence progress bars/spinners when stdout is not a TTY","Emit only UTF-8 text on stdout; all diagnostics to stderr","Base64-encode if binary payloads are unavoidable","Test provider output with `provider | iconv -f utf-8 -t utf-8`"],"tags":["auth","encoding","utf-8","external-provider"],"backgroundTag":"non-utf8-output","analyzedSha":"bc7f02eddd3d84085849dc19ed216f11c23b0571","analyzedAt":"2026-08-31T04:59:42.031Z","schemaVersion":2},"datasetVersion":"2026-08-31T09:17:48.483Z"}