pemistahl/grex · error

The specified file's encoding is not valid UTF-8

Error message

The specified file's encoding is not valid UTF-8

What it means

RegExpBuilder::from_file requires the input file to be valid UTF-8 because std::fs::read_to_string decodes it as such. When the underlying io::Error kind is InvalidData, the builder panics with 'The specified file's encoding is not valid UTF-8'. Non-UTF-8 encodings (e.g. Latin-1, UTF-16, or arbitrary binaries) cannot be read this way.

Solutions

  1. Re-save the file as UTF-8 (no BOM needed) and retry
  2. Convert the file's encoding explicitly (iconv -f UTF-16 -t UTF-8, or read with encoding_rs and re-encode)
  3. Read bytes yourself with std::fs::read, decode with String::from_utf8 (or a transcoder), then pass the lines to RegExpBuilder::from

Example fix

// before
let builder = RegExpBuilder::from_file("examples_latin1.txt");
// after
let bytes = std::fs::read("examples_latin1.txt")?;
let text = String::from_utf8_lossy(&bytes);
let cases: Vec<&str> = text.lines().collect();
let builder = RegExpBuilder::from(&cases);
Defensive patterns

Strategy: validation

Validate before calling

let bytes = std::fs::read(path)?;
if String::from_utf8(bytes.clone()).is_err() { return Err(anyhow!("file is not valid UTF-8: {}", path)); }

Try / catch

std::panic::catch_unwind(|| RegExpBuilder::from_file(path))
    .map_err(|_| format!("examples file is not UTF-8: {}", path.display()))

Prevention

When it happens

Trigger: Calling RegExpBuilder::from_file on a file saved in UTF-16, Windows-1252/Latin-1, or a binary file; files produced by tools that default to a legacy codepage.

Common situations: Windows-edited files exported in a legacy codepage; test case lists converted through Excel or PowerShell (UTF-16 by default); accidentally pointing the builder at a binary blob.

Related errors

AI-assisted analysis of pemistahl/grex@99cc347707 (2026-09-13). Data as JSON: /api/errors/65e21b03ea9e8bc3. Report an issue: GitHub.

Appendix: source

Thrown at src/builder.rs:78

    /// Each test case needs to be on a separate line.
    /// Lines may be ended with either a newline (`\n`) or
    /// a carriage return with a line feed (`\r\n`).
    /// The final line ending is optional.
    ///
    /// ⚠ Panics if:
    /// - the file cannot be found
    /// - the file's encoding is not valid UTF-8 data
    /// - the file cannot be opened because of conflicting permissions
    pub fn from_file<T: Into<PathBuf>>(file_path: T) -> Self {
        match std::fs::read_to_string(file_path.into()) {
            Ok(file_content) => Self {
                test_cases: file_content.lines().map(|it| it.to_string()).collect_vec(),
                config: RegExpConfig::new(),
            },
            Err(error) => match error.kind() {
                ErrorKind::NotFound => panic!("The specified file could not be found"),
                ErrorKind::InvalidData => {
                    panic!("The specified file's encoding is not valid UTF-8")
                }
                ErrorKind::PermissionDenied => {
                    panic!("Permission denied: The specified file could not be opened")
                }
                _ => panic!("{}", error),
            },
        }
    }

    /// Converts any Unicode decimal digit to character class `\d`.
    ///
    /// This method takes precedence over
    /// [`with_conversion_of_words`](Self::with_conversion_of_words) if both are set.
    /// Decimal digits are converted to `\d`, the remaining word characters to `\w`.
    ///
    /// This method takes precedence over
    /// [`with_conversion_of_non_whitespace`](Self::with_conversion_of_non_whitespace) if both are set.
    /// Decimal digits are converted to `\d`, the remaining non-whitespace characters to `\S`.

View on GitHub (pinned to 99cc347707)