{"record":{"id":"c0001bb5fe1cf69d","repo":"tursodatabase/turso","slug":"error-file-is-not-valid-utf-8-text","errorCode":null,"errorMessage":"Error: file \"{}\" is not valid UTF-8 text – {}","messagePattern":"Error: file \"(.+?)\" is not valid UTF-8 text – (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cli/app.rs","lineNumber":2092,"sourceCode":"        let target = db.connect()?;\n\n        let mut applier = ApplyWriter::new(&target);\n        Self::dump_database_from_conn(false, self.conn.clone(), &mut applier, StderrProgress)?;\n        applier.finish()?;\n        Ok(())\n    }\n\n    fn read_sql_file(&mut self, path: &str) -> anyhow::Result<()> {\n        let file =\n            File::open(path).map_err(|e| anyhow!(\"Error: cannot open \\\"{}\\\" – {}\", path, e))?;\n        let reader = BufReader::new(file);\n\n        let mut query_buffer = String::new();\n        let mut state = ReadState::default();\n\n        for line in reader.lines() {\n            let line = line\n                .map_err(|e| anyhow!(\"Error: file \\\"{}\\\" is not valid UTF-8 text – {}\", path, e))?;\n\n            if !query_buffer.is_empty() {\n                query_buffer.push('\\n');\n            }\n            query_buffer.push_str(&line);\n\n            state.process(&line);\n\n            if state.is_complete() {\n                self.run_query(&query_buffer);\n                query_buffer.clear();\n                state = ReadState::default();\n            }\n        }\n\n        let remaining = query_buffer.trim();\n        if !remaining.is_empty() {\n            self.run_query(remaining);","sourceCodeStart":2074,"sourceCodeEnd":2110,"githubUrl":"https://github.com/tursodatabase/turso/blob/492c4a71cd7c2649e7df83da1471b74f4b1c7aa9/cli/app.rs#L2074-L2110","documentation":"Thrown by read_sql_file when BufRead::lines hits a byte sequence that is not valid UTF-8 while reading the script line by line (cli/app.rs:2076). The file opened successfully, but the CLI requires UTF-8 because it accumulates lines into a String query buffer.","triggerScenarios":"`.read file.sql` where the file is Latin-1/CP1252/UTF-16 encoded or contains binary garbage; UTF-16 dumps with BOMs from Windows tools; scripts concatenated with binary files.","commonSituations":"SQL exports saved by Windows editors as UTF-16; smart quotes pasted from word processors in a legacy codepage; files mangled by incorrect iconv conversions.","solutions":["Identify the encoding: file script.sql","Convert to UTF-8: iconv -f UTF-16 -t UTF-8 in.sql > out.sql (use the real source encoding)","Strip a UTF-8 BOM if present: sed -i '1s/^\\xEF\\xBB\\xBF//' script.sql","Fix the export settings of the tool that produced the file so it writes UTF-8"],"exampleFix":"# before\n.read dump-utf16.sql\n# after (convert once, then read)\niconv -f UTF-16 -t UTF-8 dump-utf16.sql > dump.sql\n.read dump.sql","handlingStrategy":"try-catch","validationCode":"let bytes = std::fs::read(path)?;\nlet text = std::str::from_utf8(&bytes)\n    .map_err(|_| anyhow!(\"{path} is not UTF-8; convert with iconv\"))?;","typeGuard":"fn is_utf8(bytes: &[u8]) -> bool {\n    std::str::from_utf8(bytes).is_ok()\n}","tryCatchPattern":"match std::fs::read_to_string(path) {\n    Ok(sql) => run_script(&sql),\n    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => convert_then_retry(path),\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Save SQL scripts as UTF-8 without BOM","Convert Windows-generated dumps with iconv before reading them","Verify encodings in CI with a from_utf8 check on checked-in .sql files"],"tags":["cli","dot-command","encoding","utf-8","file-io","rust"],"backgroundTag":"invalid-file-encoding","analyzedSha":"492c4a71cd7c2649e7df83da1471b74f4b1c7aa9","analyzedAt":"2026-08-20T07:02:18.389Z","contentChangedAt":"2026-08-20T07:02:18.389Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}