tursodatabase/turso · error

Error: file "{}" is not valid UTF-8 text – {}

Error message

Error: file "{}" is not valid UTF-8 text – {}

What it means

Thrown by read_sql_file when BufRead::lines hits a byte sequence that is not valid UTF-8 while reading the script line by line (cli/app.rs:2076). The file opened successfully, but the CLI requires UTF-8 because it accumulates lines into a String query buffer.

Source

Thrown at cli/app.rs:2092

        let target = db.connect()?;

        let mut applier = ApplyWriter::new(&target);
        Self::dump_database_from_conn(false, self.conn.clone(), &mut applier, StderrProgress)?;
        applier.finish()?;
        Ok(())
    }

    fn read_sql_file(&mut self, path: &str) -> anyhow::Result<()> {
        let file =
            File::open(path).map_err(|e| anyhow!("Error: cannot open \"{}\" – {}", path, e))?;
        let reader = BufReader::new(file);

        let mut query_buffer = String::new();
        let mut state = ReadState::default();

        for line in reader.lines() {
            let line = line
                .map_err(|e| anyhow!("Error: file \"{}\" is not valid UTF-8 text – {}", path, e))?;

            if !query_buffer.is_empty() {
                query_buffer.push('\n');
            }
            query_buffer.push_str(&line);

            state.process(&line);

            if state.is_complete() {
                self.run_query(&query_buffer);
                query_buffer.clear();
                state = ReadState::default();
            }
        }

        let remaining = query_buffer.trim();
        if !remaining.is_empty() {
            self.run_query(remaining);

View on GitHub (pinned to 492c4a71cd)

Solutions

  1. Identify the encoding: file script.sql
  2. Convert to UTF-8: iconv -f UTF-16 -t UTF-8 in.sql > out.sql (use the real source encoding)
  3. Strip a UTF-8 BOM if present: sed -i '1s/^\xEF\xBB\xBF//' script.sql
  4. Fix the export settings of the tool that produced the file so it writes UTF-8

Example fix

# before
.read dump-utf16.sql
# after (convert once, then read)
iconv -f UTF-16 -t UTF-8 dump-utf16.sql > dump.sql
.read dump.sql
Defensive patterns

Strategy: try-catch

Validate before calling

let bytes = std::fs::read(path)?;
let text = std::str::from_utf8(&bytes)
    .map_err(|_| anyhow!("{path} is not UTF-8; convert with iconv"))?;

Type guard

fn is_utf8(bytes: &[u8]) -> bool {
    std::str::from_utf8(bytes).is_ok()
}

Try / catch

match std::fs::read_to_string(path) {
    Ok(sql) => run_script(&sql),
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => convert_then_retry(path),
    Err(e) => return Err(e.into()),
}

Prevention

When it happens

Trigger: `.read file.sql` where the file is Latin-1/CP1252/UTF-16 encoded or contains binary garbage; UTF-16 dumps with BOMs from Windows tools; scripts concatenated with binary files.

Common situations: SQL exports saved by Windows editors as UTF-16; smart quotes pasted from word processors in a legacy codepage; files mangled by incorrect iconv conversions.

Related errors

AI-assisted analysis of tursodatabase/turso@492c4a71cd (2026-08-20). Data as JSON: /api/errors/c0001bb5fe1cf69d. Report an issue: GitHub.