tailwindlabs/tailwindcss · critical

Input must be valid UTF-8

Error message

Input must be valid UTF-8

What it means

Panics (via .expect) in the oxide extractor's pre_processor when std::str::from_utf8(input) fails on the raw bytes being annotated. The pre_processor computes source positions and splits input into lines; it assumes valid UTF-8. Any non-UTF-8 bytes (e.g. a Latin-1 or binary file) cause a panic.

Source

Thrown at crates/oxide/src/extractor/pre_processors/pre_processor.rs:126

            })
            .collect::<Vec<_>>();

        // Convert byte ranges to (line, start_col, end_col)
        let mut annotations = byte_ranges
            .into_iter()
            .map(|(start, end)| {
                let (line, start_col) = byte_offset_to_line_and_column(input, start);
                let (_, end_col) = byte_offset_to_line_and_column(input, end);
                (line, start_col, end_col)
            })
            .collect::<Vec<_>>();

        // Sort for safe insertion
        annotations.sort_by(|a, b| b.0.cmp(&a.0).then(b.1.cmp(&a.1)));

        // Convert input to lines
        let mut lines = std::str::from_utf8(input)
            .expect("Input must be valid UTF-8")
            .lines()
            .map(|line| line.to_string())
            .collect::<Vec<_>>();

        // Group annotations per line
        let mut grouped = BTreeMap::<usize, Vec<(usize, usize)>>::new();
        for (line, start_char, end_char) in annotations {
            grouped
                .entry(line)
                .or_default()
                .push((start_char, end_char));
        }

        // Inject annotation lines
        for (line_idx, spans) in grouped.into_iter().rev() {
            let display_line = &lines[line_idx];
            let width = UnicodeWidthStr::width(display_line.as_str());
            let mut annotation = vec![' '; width];

View on GitHub (pinned to 16e94cbf7f)

Solutions

1. Convert input to valid UTF-8 before extraction (e.g. read with a lossy decode or transcode).
2. Skip or filter out non-UTF-8 files before passing them to the extractor.
3. On the caller side, use String::from_utf8_lossy to sanitize, or validate with std::str::from_utf8 and handle the Err.

Example fix

// before — panics on invalid UTF-8
pre_processor.process(bytes)

// after — sanitize first
let content = String::from_utf8_lossy(&bytes).into_owned();
pre_processor.process(content.as_bytes())

// Rust caller: validate UTF-8 before invoking the pre-processor
match std::str::from_utf8(input) {
    Ok(s) => pre_processor.process(s.as_bytes()),
    Err(_) => {
        // skip or lossy-convert
        let lossy = String::from_utf8_lossy(input).into_owned();
        pre_processor.process(lossy.as_bytes())
    }
}

fn isUtf8(input: &[u8]) -> bool {
    std::str::from_utf8(input).is_ok()
}

// Wrap the extractor; replace .expect with graceful handling upstream
let content = String::from_utf8(input.to_vec())
    .unwrap_or_else(|_| String::from_utf8_lossy(input).into_owned());
pre_processor.process(content.as_bytes())

When it happens

Trigger: Feeding the oxide extractor content that is not valid UTF-8 — a legacy-encoded file, a binary file mistakenly treated as source, or bytes with invalid sequences. from_utf8 returns Err and .expect panics with this message.

Common situations: Scanning a project that contains non-UTF-8 source files (older encodings, mojibake). Accidentally passing binary/mixed content to the candidate extractor. Reading a file as bytes without validating encoding first.

Related errors

• Failed to read file
• <stdin> has no metadata
• unsupported platform
• walkdir: same_file_system option not supported on this platf

AI-assisted analysis of tailwindlabs/tailwindcss@16e94cbf7f (2026-08-12). Data as JSON: /api/errors/ba2fe789a8a28719. Report an issue: GitHub.