GraphiteEditor/Graphite · error

in-place hash-to-ID rewrite produced invalid UTF-8

Error message

in-place hash-to-ID rewrite produced invalid UTF-8

What it means

During document migration, hash-based resource references in the serialized JSON are rewritten in place at the byte level: each 66-byte quoted hash range is overwritten with a numeric ID padded with spaces. String::from_utf8(bytes).expect(...) then asserts the spliced bytes are still valid UTF-8. All written bytes are ASCII, so this fails only when the byte ranges are misaligned relative to the real hash positions (overlapping matches, off-by-one ranges, or ranges computed against a different serialization than the one being spliced), which can split a multi-byte character and produce an invalid sequence.

Source

Thrown at editor/src/messages/portfolio/document_migration.rs:1143

	let mut hash_to_id: HashMap<ResourceHash, ResourceId> = HashMap::new();
	for hash in resources_by_hash.keys() {
		#[allow(clippy::unwrap_or_default)]
		hash_to_id.entry(*hash).or_insert_with(ResourceId::new);
	}

	// Each range is 66 bytes (64 hex + 2 quotes) and a ResourceId serializes to at most 20 ASCII digits, so the ID always fits.
	// Overwrite each hash in place with its ID and pad the leftover bytes with spaces, which JSON deserialization discards.
	let mut bytes = document_serialized_content.into_bytes();
	for (hash, ranges) in &resources_by_hash {
		let id_str = format!("{}", hash_to_id[hash]);
		let id_bytes = id_str.as_bytes();
		for range in ranges {
			bytes[range.start..range.start + id_bytes.len()].copy_from_slice(id_bytes);
			bytes[range.start + id_bytes.len()..range.end].fill(b' ');
		}
	}

	let out = String::from_utf8(bytes).expect("in-place hash-to-ID rewrite produced invalid UTF-8");

	(out, hash_to_id)
}

pub fn document_migration_upgrades(document: &mut DocumentMessageHandler, reset_node_definitions_on_open: bool) {
	document.network_interface.migrate_path_modify_node();
	document.network_interface.document_network_mut().normalize_stored_types();

	let network = document.network_interface.document_network().clone();

	// Apply string and node replacements to each node
	let mut replacements = HashMap::<&str, ProtoNodeIdentifier>::new();
	Iterator::chain(
		NODE_REPLACEMENTS.iter().flat_map(|NodeReplacement { node, aliases }| aliases.iter().map(|old| (*old, node.clone()))),
		REPLACEMENTS.iter().map(|(old, new)| (*old, ProtoNodeIdentifier::new(new))),
	)
	.for_each(|(old, new)| {
		if replacements.insert(old, new).is_some() {

View on GitHub (pinned to c507b35645)

Solutions

1. Debug-assert before splicing that every range is ASCII-only and non-overlapping (bytes[r].is_ascii() for all bytes in each range)
2. Recompute ranges and splice in one pass over the same byte buffer so scan and write cannot diverge
3. As a robust alternative, deserialize to a value, replace the IDs via serde, and re-serialize - slower but immune to byte-offset drift

Example fix

// before
let out = String::from_utf8(bytes).expect("in-place hash-to-ID rewrite produced invalid UTF-8");

// after
let out = String::from_utf8(bytes).unwrap_or_else(|e| {
	log::error!("UTF-8 invalid after hash-to-ID rewrite: {e}");
	String::from_utf8_lossy(e.as_bytes()).into_owned()
});

// before splicing, prove every range is pure ASCII and ranges do not overlap
let mut last_end = 0;
for range in ranges.iter().sorted_by_key(|r| r.start) {
	assert!(range.start >= last_end, "overlapping hash ranges");
	assert!(bytes[range.clone()].iter().all(u8::is_ascii), "non-ASCII hash range");
	last_end = range.end;
}

match String::from_utf8(bytes) {
	Ok(out) => out,
	Err(e) => {
		log::error!("hash-to-ID rewrite broke UTF-8: {e}");
		String::from_utf8_lossy(e.as_bytes()).into_owned()
	}
}

When it happens

Trigger: The regex/range scan that locates hashes drifts from the actual byte offsets - overlapping hash occurrences, ranges captured on a differently formatted string, or edits to document_serialized_content between scanning and splicing - so a replacement boundary lands inside a multi-byte UTF-8 character.

Common situations: Changing the serialization format (pretty-printing, key ordering) without updating the range scanner; introducing Unicode content where ranges overlap; migration code refactors that reuse stale ranges.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• InvalidData
• Failed to parse input type for #[implementation(...)]. Expec
• Expected `->` arrow after input type in #[implementations(..
• Failed to parse output type for #[implementation(...)]. Expe
• Failed to parse node_fn attributes: {e}

AI-assisted analysis of GraphiteEditor/Graphite@c507b35645 (2026-08-16). Data as JSON: /api/errors/4beda739e6eb6fd1. Report an issue: GitHub.