siyuan-note/siyuan · error · obsidianUserError

344

344

Error message

Markdown [%s] is not valid UTF-8

What it means

Thrown in analyzeObsidianDocuments after readStableObsidianFile succeeds but utf8.Valid(data) is false. It is reported via newObsidianUserError with language code 344 ('Markdown file [%s] is not UTF-8 encoded'), so the UI surfaces the offending relative path. SiYuan's Markdown pipeline (Lute) expects UTF-8; bytes outside that encoding cannot be parsed meaningfully.

Source

Thrown at kernel/model/import_obsidian.go:862

}

func analyzeObsidianDocuments(ctx context.Context, vault *obsidianVaultContext, progress func(int, string)) error {
	sourceCount := countObsidianSourceDocs(vault.Docs)
	processed := 0
	for _, doc := range vault.Docs {
		if doc.Synthetic {
			continue
		}
		if err := ctx.Err(); err != nil {
			return err
		}
		data, err := readStableObsidianFile(doc.Source)
		if err != nil {
			return newObsidianReadUserError(doc.Source, err)
		}
		if !utf8.Valid(data) {
			return newObsidianUserError(344, doc.Source.RelPath,
				fmt.Errorf("Markdown [%s] is not valid UTF-8", doc.Source.RelPath))
		}
		scan := scanObsidianSource(data)
		for _, blockID := range scan.BlockIDs {
			if scan.Duplicates[blockID] {
				doc.DuplicateBlocks[blockID] = true
			}
			if doc.BlockIDs[blockID] == "" {
				doc.BlockIDs[blockID] = ast.NewNodeID()
			}
		}
		tree, _, _, _ := parseStdMd(data)
		if tree == nil {
			return newObsidianUserError(347, doc.Source.RelPath,
				fmt.Errorf("parse Markdown [%s] failed", doc.Source.RelPath))
		}
		buildObsidianHeadingIndex(doc, tree)
		vault.Analysis.WikiLinkCount += countObsidianNonEmbedTokens(scan.Wikis)
		vault.Analysis.EmbedCount += countObsidianEmbedTokens(scan.Wikis)

View on GitHub (pinned to 251596fc0d)

Solutions

1. Identify the offending file from the reported [%s] relative path and re-save it as UTF-8 (iconv -f GBK -t UTF-8 file.md, or open in an editor and choose 'Save as UTF-8').
2. Batch-convert the whole vault: detect per-file encoding (e.g. with chardet/enca) then iconv each non-UTF-8 file in place, preserving backups.
3. Remove or move the non-UTF-8 file out of the vault if it is not needed, then re-analyse.
4. Verify with a quick check (file -i or a UTF-8 validator) that no other markdown in the vault is non-UTF-8 before retrying.

Example fix

// before: file saved as GBK
//   iconv -f GBK -t UTF-8 notes/old.md -o notes/old.md.utf8

// after: convert in place and retry analysis
iconv -f GBK -t UTF-8 notes/old.md | sponge notes/old.md
StartObsidianVaultAnalysis(vaultPath)

// Reject non-UTF-8 markdown before analysis.
func vaultAllMarkdownUTF8(root string) (string, error) {
    var bad string
    err := filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
        if err != nil || d.IsDir() || !strings.HasSuffix(p, ".md") { return err }
        b, e := os.ReadFile(p)
        if e != nil { return e }
        if !utf8.Valid(b) { bad = p; return errors.New("non-utf8") }
        return nil
    })
    return bad, err
}

null

null

When it happens

Trigger: A .md file in the vault is encoded in a legacy single-byte or CJK encoding: GBK/GB2312/Big5 (common for older Chinese vaults), Shift-JIS (Japanese), Latin-1/Windows-1252, or a file with a binary prefix/corruption that produces invalid byte sequences. Any single non-UTF-8 markdown file aborts the whole analysis pass.

Common situations: Vault exported from an older note app that defaulted to the system codepage; files edited in a Windows editor saved as ANSI/GBK; mixed-encoding vaults merged from different sources; a .md file accidentally concatenated with a binary asset.

Related errors

• 347
• --id is required
• empty result
• data block DOM failed: %w
• Do not include Markdown syntax marker [%s]

AI-assisted analysis of siyuan-note/siyuan@251596fc0d (2026-08-12). Data as JSON: /api/errors/e23dc87a05cc9819. Report an issue: GitHub.