{"record":{"id":"4beda739e6eb6fd1","repo":"GraphiteEditor/Graphite","slug":"in-place-hash-to-id-rewrite-produced-invalid-utf-8","errorCode":null,"errorMessage":"in-place hash-to-ID rewrite produced invalid UTF-8","messagePattern":"in-place hash-to-ID rewrite produced invalid UTF-8","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"editor/src/messages/portfolio/document_migration.rs","lineNumber":1143,"sourceCode":"\tlet mut hash_to_id: HashMap<ResourceHash, ResourceId> = HashMap::new();\n\tfor hash in resources_by_hash.keys() {\n\t\t#[allow(clippy::unwrap_or_default)]\n\t\thash_to_id.entry(*hash).or_insert_with(ResourceId::new);\n\t}\n\n\t// Each range is 66 bytes (64 hex + 2 quotes) and a ResourceId serializes to at most 20 ASCII digits, so the ID always fits.\n\t// Overwrite each hash in place with its ID and pad the leftover bytes with spaces, which JSON deserialization discards.\n\tlet mut bytes = document_serialized_content.into_bytes();\n\tfor (hash, ranges) in &resources_by_hash {\n\t\tlet id_str = format!(\"{}\", hash_to_id[hash]);\n\t\tlet id_bytes = id_str.as_bytes();\n\t\tfor range in ranges {\n\t\t\tbytes[range.start..range.start + id_bytes.len()].copy_from_slice(id_bytes);\n\t\t\tbytes[range.start + id_bytes.len()..range.end].fill(b' ');\n\t\t}\n\t}\n\n\tlet out = String::from_utf8(bytes).expect(\"in-place hash-to-ID rewrite produced invalid UTF-8\");\n\n\t(out, hash_to_id)\n}\n\npub fn document_migration_upgrades(document: &mut DocumentMessageHandler, reset_node_definitions_on_open: bool) {\n\tdocument.network_interface.migrate_path_modify_node();\n\tdocument.network_interface.document_network_mut().normalize_stored_types();\n\n\tlet network = document.network_interface.document_network().clone();\n\n\t// Apply string and node replacements to each node\n\tlet mut replacements = HashMap::<&str, ProtoNodeIdentifier>::new();\n\tIterator::chain(\n\t\tNODE_REPLACEMENTS.iter().flat_map(|NodeReplacement { node, aliases }| aliases.iter().map(|old| (*old, node.clone()))),\n\t\tREPLACEMENTS.iter().map(|(old, new)| (*old, ProtoNodeIdentifier::new(new))),\n\t)\n\t.for_each(|(old, new)| {\n\t\tif replacements.insert(old, new).is_some() {","sourceCodeStart":1125,"sourceCodeEnd":1161,"githubUrl":"https://github.com/GraphiteEditor/Graphite/blob/c507b356453361e31638b8bff8f6d46b6da2961e/editor/src/messages/portfolio/document_migration.rs#L1125-L1161","documentation":"During document migration, hash-based resource references in the serialized JSON are rewritten in place at the byte level: each 66-byte quoted hash range is overwritten with a numeric ID padded with spaces. String::from_utf8(bytes).expect(...) then asserts the spliced bytes are still valid UTF-8. All written bytes are ASCII, so this fails only when the byte ranges are misaligned relative to the real hash positions (overlapping matches, off-by-one ranges, or ranges computed against a different serialization than the one being spliced), which can split a multi-byte character and produce an invalid sequence.","triggerScenarios":"The regex/range scan that locates hashes drifts from the actual byte offsets - overlapping hash occurrences, ranges captured on a differently formatted string, or edits to document_serialized_content between scanning and splicing - so a replacement boundary lands inside a multi-byte UTF-8 character.","commonSituations":"Changing the serialization format (pretty-printing, key ordering) without updating the range scanner; introducing Unicode content where ranges overlap; migration code refactors that reuse stale ranges.","solutions":["Debug-assert before splicing that every range is ASCII-only and non-overlapping (bytes[r].is_ascii() for all bytes in each range)","Recompute ranges and splice in one pass over the same byte buffer so scan and write cannot diverge","As a robust alternative, deserialize to a value, replace the IDs via serde, and re-serialize - slower but immune to byte-offset drift"],"exampleFix":"// before\nlet out = String::from_utf8(bytes).expect(\"in-place hash-to-ID rewrite produced invalid UTF-8\");\n\n// after\nlet out = String::from_utf8(bytes).unwrap_or_else(|e| {\n\tlog::error!(\"UTF-8 invalid after hash-to-ID rewrite: {e}\");\n\tString::from_utf8_lossy(e.as_bytes()).into_owned()\n});","handlingStrategy":"validation","validationCode":"// before splicing, prove every range is pure ASCII and ranges do not overlap\nlet mut last_end = 0;\nfor range in ranges.iter().sorted_by_key(|r| r.start) {\n\tassert!(range.start >= last_end, \"overlapping hash ranges\");\n\tassert!(bytes[range.clone()].iter().all(u8::is_ascii), \"non-ASCII hash range\");\n\tlast_end = range.end;\n}","typeGuard":null,"tryCatchPattern":"match String::from_utf8(bytes) {\n\tOk(out) => out,\n\tErr(e) => {\n\t\tlog::error!(\"hash-to-ID rewrite broke UTF-8: {e}\");\n\t\tString::from_utf8_lossy(e.as_bytes()).into_owned()\n\t}\n}","preventionTips":["Scan and splice the same byte buffer in one pass so offsets cannot drift","Assert ranges are ASCII-aligned and non-overlapping before mutating bytes","Prefer deserialize-modify-reserialize over byte surgery when format changes are frequent"],"tags":["rust","document-migration","utf8","byte-manipulation","serialization"],"backgroundTag":"invalid-utf8-byte-sequence","analyzedSha":"c507b356453361e31638b8bff8f6d46b6da2961e","analyzedAt":"2026-08-16T21:57:18.596Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}