Orange-OpenSource/hurl · error · io::Error

stream did not contain valid UTF-8

Error message

stream did not contain valid UTF-8

What it means

`Input::read_to_string` reads a hurl file (or stdin) fully into bytes, strips a UTF-8 BOM, then converts the bytes with `String::from_utf8`. If the bytes are not valid UTF-8, the error is wrapped into `io::Error` with `ErrorKind::InvalidData` and surfaced as 'stream did not contain valid UTF-8'. Hurl inputs must be UTF-8 text, so any binary or non-UTF-8 encoded file is rejected here.

Source

Thrown at packages/hurl_core/src/input.rs:116

    /// Reads the content of this input to a string, removing any BOM.
    fn read_to_string(&self) -> Result<String, io::Error> {
        match self {
            InputKind::File(path) => {
                let mut f = File::open(path)?;
                let metadata = fs::metadata(path).unwrap();
                let mut buffer = vec![0; metadata.len() as usize];
                f.read_exact(&mut buffer)?;
                string_from_utf8(buffer)
            }
            InputKind::Stdin(cached) => Ok(cached.clone()),
        }
    }
}

fn string_from_utf8(buffer: Vec<u8>) -> Result<String, io::Error> {
    let mut buffer = buffer;
    strip_bom(&mut buffer);
    String::from_utf8(buffer).map_err(|e| io::Error::new(ErrorKind::InvalidData, e))
}

/// Remove BOM from the input bytes
fn strip_bom(bytes: &mut Vec<u8>) {
    if bytes.starts_with(&[0xefu8, 0xbb, 0xbf]) {
        bytes.drain(0..3);
    }
}

#[cfg(test)]
pub mod tests {
    use super::*;

    #[test]
    fn test_strip_bom() {
        let mut bytes = vec![];
        strip_bom(&mut bytes);
        assert!(bytes.is_empty());

View on GitHub (pinned to 9572cc7c43)

Solutions

1. Re-save the input file as UTF-8 (no BOM needed; a UTF-8 BOM is stripped automatically), e.g. `iconv -f UTF-16 -t UTF-8 input.hurl > input.utf8.hurl` or in PowerShell 7 use `utf8NoBOM`.
2. Check the encoding first with `file -i input.hurl` — if it reports utf-16 or iso-8859-1, convert it.
3. If piping via stdin, ensure the producer emits UTF-8 bytes (e.g. `curl ... | hurl -` with text output, not binary).
4. If the input is intentionally binary, it cannot be a hurl input; point hurl at a real .hurl text file instead.
5. Validate the bytes before running: `iconv -f UTF-8 input.hurl > /dev/null && echo ok`.
6. On Windows, configure editors/redirects to write UTF-8 (PowerShell: `$PSDefaultParameterValues['Out-File:Encoding']='utf8NoBOM'`).

Example fix

# before
$ hurl -f input.hurl
error: IO error: stream did not contain valid UTF-8

# after (convert to UTF-8 first)
$ file -i input.hurl            # charset=utf-16
$ iconv -f UTF-16 -t UTF-8 input.hurl > input.utf8.hurl
$ hurl -f input.utf8.hurl

fn ensure_utf8_file(path: &str) -> Result<(), String> {
    let bytes = std::fs::read(path).map_err(|e| e.to_string())?;
    match std::str::from_utf8(&bytes) {
        Ok(_) => Ok(()),
        Err(e) => Err(format!("{} is not valid UTF-8: {} (convert with iconv)", path, e)),
    }
}

fn is_utf8(bytes: &[u8]) -> bool {
    std::str::from_utf8(bytes).is_ok()
}

match input.read_to_string() {
    Ok(content) => { /* use content */ }
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {
        eprintln!("Input is not valid UTF-8: {} — re-save as UTF-8", e);
    }
    Err(e) => eprintln!("IO error: {}", e),
}

When it happens

Trigger: Calling `Input::read_to_string` (or `Input::from_stdin`, which reads stdin via `read_to_string`) when the file/stdin bytes contain invalid UTF-8: e.g. a .hurl file saved as UTF-16, Latin-1/ISO-8859-1, or a binary file passed as input. A UTF-16 BOM is not stripped (only the UTF-8 BOM EF BB BF is), so UTF-16 files always fail.

Common situations: Editor or PowerShell (5.x) saved the .hurl file as UTF-16; a download/output of a previous binary response was piped into `hurl` via stdin; a non-UTF-8 codepage (e.g. Windows-1252 with accented characters) was used to write test files.

Related errors

• writing bytes to file

AI-assisted analysis of Orange-OpenSource/hurl@9572cc7c43 (2026-09-02). Data as JSON: /api/errors/807f69ff77576faa. Report an issue: GitHub.