{"record":{"id":"807f69ff77576faa","repo":"Orange-OpenSource/hurl","slug":"stream-did-not-contain-valid-utf-8","errorCode":null,"errorMessage":"stream did not contain valid UTF-8","messagePattern":"stream did not contain valid UTF-8","errorType":"validation","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"packages/hurl_core/src/input.rs","lineNumber":116,"sourceCode":"    /// Reads the content of this input to a string, removing any BOM.\n    fn read_to_string(&self) -> Result<String, io::Error> {\n        match self {\n            InputKind::File(path) => {\n                let mut f = File::open(path)?;\n                let metadata = fs::metadata(path).unwrap();\n                let mut buffer = vec![0; metadata.len() as usize];\n                f.read_exact(&mut buffer)?;\n                string_from_utf8(buffer)\n            }\n            InputKind::Stdin(cached) => Ok(cached.clone()),\n        }\n    }\n}\n\nfn string_from_utf8(buffer: Vec<u8>) -> Result<String, io::Error> {\n    let mut buffer = buffer;\n    strip_bom(&mut buffer);\n    String::from_utf8(buffer).map_err(|e| io::Error::new(ErrorKind::InvalidData, e))\n}\n\n/// Remove BOM from the input bytes\nfn strip_bom(bytes: &mut Vec<u8>) {\n    if bytes.starts_with(&[0xefu8, 0xbb, 0xbf]) {\n        bytes.drain(0..3);\n    }\n}\n\n#[cfg(test)]\npub mod tests {\n    use super::*;\n\n    #[test]\n    fn test_strip_bom() {\n        let mut bytes = vec![];\n        strip_bom(&mut bytes);\n        assert!(bytes.is_empty());","sourceCodeStart":98,"sourceCodeEnd":134,"githubUrl":"https://github.com/Orange-OpenSource/hurl/blob/9572cc7c4363b5f8aa18136afc5df9a9ca02551e/packages/hurl_core/src/input.rs#L98-L134","documentation":"`Input::read_to_string` reads a hurl file (or stdin) fully into bytes, strips a UTF-8 BOM, then converts the bytes with `String::from_utf8`. If the bytes are not valid UTF-8, the error is wrapped into `io::Error` with `ErrorKind::InvalidData` and surfaced as 'stream did not contain valid UTF-8'. Hurl inputs must be UTF-8 text, so any binary or non-UTF-8 encoded file is rejected here.","triggerScenarios":"Calling `Input::read_to_string` (or `Input::from_stdin`, which reads stdin via `read_to_string`) when the file/stdin bytes contain invalid UTF-8: e.g. a .hurl file saved as UTF-16, Latin-1/ISO-8859-1, or a binary file passed as input. A UTF-16 BOM is not stripped (only the UTF-8 BOM EF BB BF is), so UTF-16 files always fail.","commonSituations":"Editor or PowerShell (5.x) saved the .hurl file as UTF-16; a download/output of a previous binary response was piped into `hurl` via stdin; a non-UTF-8 codepage (e.g. Windows-1252 with accented characters) was used to write test files.","solutions":["Re-save the input file as UTF-8 (no BOM needed; a UTF-8 BOM is stripped automatically), e.g. `iconv -f UTF-16 -t UTF-8 input.hurl > input.utf8.hurl` or in PowerShell 7 use `utf8NoBOM`.","Check the encoding first with `file -i input.hurl` — if it reports utf-16 or iso-8859-1, convert it.","If piping via stdin, ensure the producer emits UTF-8 bytes (e.g. `curl ... | hurl -` with text output, not binary).","If the input is intentionally binary, it cannot be a hurl input; point hurl at a real .hurl text file instead.","Validate the bytes before running: `iconv -f UTF-8 input.hurl > /dev/null && echo ok`.","On Windows, configure editors/redirects to write UTF-8 (PowerShell: `$PSDefaultParameterValues['Out-File:Encoding']='utf8NoBOM'`)."],"exampleFix":"# before\n$ hurl -f input.hurl\nerror: IO error: stream did not contain valid UTF-8\n\n# after (convert to UTF-8 first)\n$ file -i input.hurl            # charset=utf-16\n$ iconv -f UTF-16 -t UTF-8 input.hurl > input.utf8.hurl\n$ hurl -f input.utf8.hurl","handlingStrategy":"validation","validationCode":"fn ensure_utf8_file(path: &str) -> Result<(), String> {\n    let bytes = std::fs::read(path).map_err(|e| e.to_string())?;\n    match std::str::from_utf8(&bytes) {\n        Ok(_) => Ok(()),\n        Err(e) => Err(format!(\"{} is not valid UTF-8: {} (convert with iconv)\", path, e)),\n    }\n}","typeGuard":"fn is_utf8(bytes: &[u8]) -> bool {\n    std::str::from_utf8(bytes).is_ok()\n}","tryCatchPattern":"match input.read_to_string() {\n    Ok(content) => { /* use content */ }\n    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {\n        eprintln!(\"Input is not valid UTF-8: {} — re-save as UTF-8\", e);\n    }\n    Err(e) => eprintln!(\"IO error: {}\", e),\n}","preventionTips":["Save all .hurl files as UTF-8 (utf8NoBOM in editors/PowerShell).","Run `file -i` or `iconv -f UTF-8 -t UTF-8 <file> -o /dev/null` in CI to validate encodings before invoking hurl.","Never pipe binary data (images, zips) into hurl via stdin.","Detect UTF-16 BOMs (FF FE / FE FF) and convert before use.","Normalize encodings with `dos2unix`/`iconv` when files come from Windows users."],"tags":["io","encoding","utf-8","rust","invalid-data"],"backgroundTag":"invalid-utf8-input","analyzedSha":"9572cc7c4363b5f8aa18136afc5df9a9ca02551e","analyzedAt":"2026-09-02T17:44:19.043Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T21:17:11.164Z"}