{"record":{"id":"8cf957adc1258c30","repo":"abhigyanpatwari/GitNexus","slug":"sandbox-copy-path-is-not-valid-utf-8-relative-s","errorCode":null,"errorMessage":"sandbox_copy path is not valid UTF-8: {relative!s}","messagePattern":"sandbox_copy path is not valid UTF-8: (.+?)","errorType":"exception","errorClass":"SandboxError","httpStatus":null,"severity":"error","filePath":"eval/workflow_bench/task_assets.py","lineNumber":892,"sourceCode":"        view = view[written:]\n\n\ndef _mutation_identity(metadata: os.stat_result) -> tuple[int, int, int, int, int, int]:\n    return (\n        metadata.st_dev,\n        metadata.st_ino,\n        metadata.st_mode,\n        metadata.st_size,\n        metadata.st_mtime_ns,\n        metadata.st_ctime_ns,\n    )\n\n\ndef _validate_manifest_path(relative: PurePosixPath) -> None:\n    try:\n        path_bytes = len(relative.as_posix().encode(\"utf-8\"))\n    except UnicodeEncodeError as exc:\n        raise SandboxError(f\"sandbox_copy path is not valid UTF-8: {relative!s}\") from exc\n    if path_bytes > MAX_TASK_ASSET_PATH_BYTES:\n        raise SandboxError(\"sandbox_copy exceeds the path byte limit\")\n\n\ndef _manifest_digest(entries: tuple[AssetManifestEntry, ...]) -> str:\n    payload = [\n        {\n            \"kind\": entry.kind,\n            \"link_target\": entry.link_target,\n            \"mode\": entry.mode,\n            \"path\": entry.path.as_posix(),\n            \"sha256\": entry.sha256,\n            \"size\": entry.size,\n        }\n        for entry in entries\n    ]\n    return hashlib.sha256(json.dumps(payload, sort_keys=True, separators=(\",\", \":\")).encode()).hexdigest()\n","sourceCodeStart":874,"sourceCodeEnd":910,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/d540b00184d71a896261ee02670da9a92d59d8f7/eval/workflow_bench/task_assets.py#L874-L910","documentation":"Raised by _validate_manifest_path when relative.as_posix().encode('utf-8') throws UnicodeEncodeError. sandbox_copy paths must round-trip cleanly through UTF-8 so the manifest JSON and digests stay deterministic; any path that cannot be UTF-8 encoded (e.g. containing surrogateescape codepoints from os.listdir on Linux) is rejected before it enters the manifest.","triggerScenarios":"A sandbox_copy declaration whose source/target path was constructed from raw os.scandir/os.listdir bytes decoded with surrogateescape, or assembled from a bytes path that included invalid UTF-8 sequences, then passed through PurePosixPath. The surrogate codepoints (U+DC80..U+DCFF) survive into PurePosixPath but fail .encode('utf-8').","commonSituations":"Mixing bytes-based path handling with str PurePosixPath; copying assets from a filesystem with legacy non-UTF-8 filenames (Latin-1,Shift-JIS); test fixtures that build paths from arbitrary bytes; cross-platform path joining that injects lone surrogates.","solutions":["Normalize the sandbox_copy path to a clean str before building the declaration: decode bytes with the actual filesystem encoding and re-encode strict utf-8, replacing or rejecting non-decodable names.","Rename the offending source file on disk to a valid UTF-8 name so the declaration path is clean.","If you genuinely need non-UTF-8 paths, they are unsupported by this module — exclude them from sandbox_copy or pre-process them into UTF-8 equivalents."],"exampleFix":"# before: raw bytes path leaks surrogates into the declaration\nraw = os.fsdecode(os.listdir(b'/repo')[0])   # may contain surrogates\nsource = PurePosixPath(raw)\n\n# after: enforce clean UTF-8 at the boundary\nraw = os.listdir('/repo')[0]\nsource = PurePosixPath(raw)\nassert source.as_posix().encode('utf-8')   # fails fast on bad input","handlingStrategy":"validation","validationCode":"from pathlib import PurePosixPath\n\ndef assert_utf8_path(relative: PurePosixPath) -> None:\n    posix = relative.as_posix()\n    try:\n        posix.encode('utf-8')\n    except UnicodeEncodeError:\n        raise ValueError(f'sandbox_copy path is not valid UTF-8: {posix!r}') from None\n\n# Run on every declared source/target before building the task dict.","typeGuard":"from pathlib import PurePosixPath\n\ndef is_utf8_path(relative: PurePosixPath) -> bool:\n    try:\n        relative.as_posix().encode('utf-8')\n        return True\n    except UnicodeEncodeError:\n        return False","tryCatchPattern":"from .proposer_sandbox import SandboxError\n\ntask_paths = [PurePosixPath(d['source']) for d in task.get('sandbox_copy', [])]\nif not all(is_utf8_path(p) for p in task_paths):\n    # sanitize / rename offending files before capture instead of catching post-hoc\n    task['sandbox_copy'] = [d for d in task['sandbox_copy'] if is_utf8_path(PurePosixPath(d['source']))]\n\ntry:\n    stage_task_assets(task, ...)\nexcept SandboxError as exc:\n    if 'not valid UTF-8' in str(exc):\n        # rename the offending source file to clean UTF-8, then retry prepare\n        ...\n    raise","preventionTips":["Build all sandbox_copy paths from clean str, never from surrogateescape-decoded bytes.","Reject or rename non-UTF-8 filenames at the repo boundary, before they reach sandbox_copy declarations.","Run assert_utf8_path over declared paths in task-validation, upstream of the asset module."],"tags":["validation","paths","encoding","sandbox"],"backgroundTag":null,"analyzedSha":"d540b00184d71a896261ee02670da9a92d59d8f7","analyzedAt":"2026-08-12T19:50:25.132Z","schemaVersion":2},"datasetVersion":"2026-08-14T00:17:13.853Z"}