astral-sh/ruff · error

truth source directory `{path}` contains invalid UTF-8

Error message

truth source directory `{path}` contains invalid UTF-8

What it means

`TaskSource::all` enumerates the subdirectories of `truth/` and converts each to a `SystemPath`; any directory whose name is not valid UTF-8 aborts the entire eval run with the offending path echoed via `display()`. This guards the whole harness up front so later steps can assume Unicode paths.

Source

Thrown at crates/ty_completion_eval/src/main.rs:416

}

impl TaskSource {
    fn all(src_dir: &SystemPath) -> anyhow::Result<Vec<TaskSource>> {
        let mut sources = vec![];
        let read_dir = src_dir
            .as_std_path()
            .read_dir()
            .with_context(|| format!("failed to read directory entries in `{src_dir}`"))?;
        for result in read_dir {
            let dent = result
                .with_context(|| format!("failed to get directory entry from `{src_dir}`"))?;
            let path = dent.path();
            if !path.is_dir() {
                continue;
            }

            let dir = SystemPath::from_std_path(&path).ok_or_else(|| {
                anyhow::anyhow!(
                    "truth source directory `{path}` contains invalid UTF-8",
                    path = path.display()
                )
            })?;
            sources.push(TaskSource::new(dir)?);
        }
        // Sort our sources so that we always run in the same order.
        // And also so that the CSV output is deterministic across
        // all platforms.
        sources.sort_by(|source1, source2| source1.name.cmp(&source2.name));
        Ok(sources)
    }

    fn new(dir: &SystemPath) -> anyhow::Result<TaskSource> {
        let name = dir.file_name().ok_or_else(|| {
            anyhow::anyhow!("truth source directory `{dir}` does not contain a base name")
        })?;

View on GitHub (pinned to 672bb4edf0)

Solutions

1. Find the offender: `find crates/ty_completion_eval/truth -mindepth 1 -maxdepth 1 -print0 | while IFS= read -r -d '' d; do printf '%s' "$d" | iconv -f utf-8 -t utf-8 >/dev/null 2>&1 || echo "non-UTF-8: $d"; done`
2. Remove or rename the offending directory (`git status crates/ty_completion_eval/truth` usually shows it as untracked)
3. Avoid unpacking third-party archives directly into the truth tree

Example fix

# before: run aborts with `truth source directory ... contains invalid UTF-8`
# after: remove the stray directory, then re-run
rm -r "$(printf 'crates/ty_completion_eval/truff/bad\xff')" 2>/dev/null || true
git clean -fd crates/ty_completion_eval/truth
cargo run -p ty_completion_eval -- all

find crates/ty_completion_eval/truth -mindepth 1 -maxdepth 1 -type d -print0 \
| while IFS= read -r -d '' d; do
    printf '%s' "$d" | iconv -f utf-8 -t utf-8 >/dev/null 2>&1 \
      || echo "non-UTF-8 fixture dir: $d" >&2
  done

When it happens

Trigger: A stray directory with non-UTF-8 bytes in its name inside `crates/ty_completion_eval/truth/` — OS droppings, editor artifacts, or extraction residue — gets enumerated and rejected.

Common situations: Developers or tools accidentally creating byte-named directories inside the repo; archives unpacked into the truth tree carrying mangled names.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• path `{}` is not valid UTF-8
• The current working directory `{}` contains non-Unicode char
• The current working directory `{}` contains non-Unicode char
• Detected project directory `{path}` contains non-Unicode cha
• {truth} does not exist: ty's completion evaluation must be r

AI-assisted analysis of astral-sh/ruff@672bb4edf0 (2026-08-16). Data as JSON: /api/errors/2623bd27b5549a6e. Report an issue: GitHub.