{"record":{"id":"ba2fe789a8a28719","repo":"tailwindlabs/tailwindcss","slug":"input-must-be-valid-utf-8","errorCode":null,"errorMessage":"Input must be valid UTF-8","messagePattern":"Input must be valid UTF-8","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/oxide/src/extractor/pre_processors/pre_processor.rs","lineNumber":126,"sourceCode":"            })\n            .collect::<Vec<_>>();\n\n        // Convert byte ranges to (line, start_col, end_col)\n        let mut annotations = byte_ranges\n            .into_iter()\n            .map(|(start, end)| {\n                let (line, start_col) = byte_offset_to_line_and_column(input, start);\n                let (_, end_col) = byte_offset_to_line_and_column(input, end);\n                (line, start_col, end_col)\n            })\n            .collect::<Vec<_>>();\n\n        // Sort for safe insertion\n        annotations.sort_by(|a, b| b.0.cmp(&a.0).then(b.1.cmp(&a.1)));\n\n        // Convert input to lines\n        let mut lines = std::str::from_utf8(input)\n            .expect(\"Input must be valid UTF-8\")\n            .lines()\n            .map(|line| line.to_string())\n            .collect::<Vec<_>>();\n\n        // Group annotations per line\n        let mut grouped = BTreeMap::<usize, Vec<(usize, usize)>>::new();\n        for (line, start_char, end_char) in annotations {\n            grouped\n                .entry(line)\n                .or_default()\n                .push((start_char, end_char));\n        }\n\n        // Inject annotation lines\n        for (line_idx, spans) in grouped.into_iter().rev() {\n            let display_line = &lines[line_idx];\n            let width = UnicodeWidthStr::width(display_line.as_str());\n            let mut annotation = vec![' '; width];","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/tailwindlabs/tailwindcss/blob/16e94cbf7f965c5ad697e90e940b5e178efad67c/crates/oxide/src/extractor/pre_processors/pre_processor.rs#L108-L144","documentation":"Panics (via .expect) in the oxide extractor's pre_processor when std::str::from_utf8(input) fails on the raw bytes being annotated. The pre_processor computes source positions and splits input into lines; it assumes valid UTF-8. Any non-UTF-8 bytes (e.g. a Latin-1 or binary file) cause a panic.","triggerScenarios":"Feeding the oxide extractor content that is not valid UTF-8 — a legacy-encoded file, a binary file mistakenly treated as source, or bytes with invalid sequences. from_utf8 returns Err and .expect panics with this message.","commonSituations":"Scanning a project that contains non-UTF-8 source files (older encodings, mojibake). Accidentally passing binary/mixed content to the candidate extractor. Reading a file as bytes without validating encoding first.","solutions":["Convert input to valid UTF-8 before extraction (e.g. read with a lossy decode or transcode).","Skip or filter out non-UTF-8 files before passing them to the extractor.","On the caller side, use String::from_utf8_lossy to sanitize, or validate with std::str::from_utf8 and handle the Err."],"exampleFix":"// before — panics on invalid UTF-8\npre_processor.process(bytes)\n\n// after — sanitize first\nlet content = String::from_utf8_lossy(&bytes).into_owned();\npre_processor.process(content.as_bytes())","handlingStrategy":"validation","validationCode":"// Rust caller: validate UTF-8 before invoking the pre-processor\nmatch std::str::from_utf8(input) {\n    Ok(s) => pre_processor.process(s.as_bytes()),\n    Err(_) => {\n        // skip or lossy-convert\n        let lossy = String::from_utf8_lossy(input).into_owned();\n        pre_processor.process(lossy.as_bytes())\n    }\n}","typeGuard":"fn isUtf8(input: &[u8]) -> bool {\n    std::str::from_utf8(input).is_ok()\n}","tryCatchPattern":"// Wrap the extractor; replace .expect with graceful handling upstream\nlet content = String::from_utf8(input.to_vec())\n    .unwrap_or_else(|_| String::from_utf8_lossy(input).into_owned());\npre_processor.process(content.as_bytes())","preventionTips":["Validate input encoding before extraction; skip non-UTF-8 files.","Use String::from_utf8_lossy to sanitize on the caller side.","Keep binary and legacy-encoded files out of the extractor input set."],"tags":["oxide","rust","utf-8","extractor","panic","encoding"],"backgroundTag":null,"analyzedSha":"16e94cbf7f965c5ad697e90e940b5e178efad67c","analyzedAt":"2026-08-12T06:02:42.469Z","schemaVersion":2},"datasetVersion":"2026-08-12T13:17:24.610Z"}