{"record":{"id":"e23dc87a05cc9819","repo":"siyuan-note/siyuan","slug":"344","errorCode":"344","errorMessage":"Markdown [%s] is not valid UTF-8","messagePattern":"Markdown \\[(.+?)\\] is not valid UTF-8","errorType":"error_code","errorClass":"obsidianUserError","httpStatus":null,"severity":"error","filePath":"kernel/model/import_obsidian.go","lineNumber":862,"sourceCode":"}\n\nfunc analyzeObsidianDocuments(ctx context.Context, vault *obsidianVaultContext, progress func(int, string)) error {\n\tsourceCount := countObsidianSourceDocs(vault.Docs)\n\tprocessed := 0\n\tfor _, doc := range vault.Docs {\n\t\tif doc.Synthetic {\n\t\t\tcontinue\n\t\t}\n\t\tif err := ctx.Err(); err != nil {\n\t\t\treturn err\n\t\t}\n\t\tdata, err := readStableObsidianFile(doc.Source)\n\t\tif err != nil {\n\t\t\treturn newObsidianReadUserError(doc.Source, err)\n\t\t}\n\t\tif !utf8.Valid(data) {\n\t\t\treturn newObsidianUserError(344, doc.Source.RelPath,\n\t\t\t\tfmt.Errorf(\"Markdown [%s] is not valid UTF-8\", doc.Source.RelPath))\n\t\t}\n\t\tscan := scanObsidianSource(data)\n\t\tfor _, blockID := range scan.BlockIDs {\n\t\t\tif scan.Duplicates[blockID] {\n\t\t\t\tdoc.DuplicateBlocks[blockID] = true\n\t\t\t}\n\t\t\tif doc.BlockIDs[blockID] == \"\" {\n\t\t\t\tdoc.BlockIDs[blockID] = ast.NewNodeID()\n\t\t\t}\n\t\t}\n\t\ttree, _, _, _ := parseStdMd(data)\n\t\tif tree == nil {\n\t\t\treturn newObsidianUserError(347, doc.Source.RelPath,\n\t\t\t\tfmt.Errorf(\"parse Markdown [%s] failed\", doc.Source.RelPath))\n\t\t}\n\t\tbuildObsidianHeadingIndex(doc, tree)\n\t\tvault.Analysis.WikiLinkCount += countObsidianNonEmbedTokens(scan.Wikis)\n\t\tvault.Analysis.EmbedCount += countObsidianEmbedTokens(scan.Wikis)","sourceCodeStart":844,"sourceCodeEnd":880,"githubUrl":"https://github.com/siyuan-note/siyuan/blob/251596fc0de2f9528c00c224252fd073a99973f4/kernel/model/import_obsidian.go#L844-L880","documentation":"Thrown in analyzeObsidianDocuments after readStableObsidianFile succeeds but utf8.Valid(data) is false. It is reported via newObsidianUserError with language code 344 ('Markdown file [%s] is not UTF-8 encoded'), so the UI surfaces the offending relative path. SiYuan's Markdown pipeline (Lute) expects UTF-8; bytes outside that encoding cannot be parsed meaningfully.","triggerScenarios":"A .md file in the vault is encoded in a legacy single-byte or CJK encoding: GBK/GB2312/Big5 (common for older Chinese vaults), Shift-JIS (Japanese), Latin-1/Windows-1252, or a file with a binary prefix/corruption that produces invalid byte sequences. Any single non-UTF-8 markdown file aborts the whole analysis pass.","commonSituations":"Vault exported from an older note app that defaulted to the system codepage; files edited in a Windows editor saved as ANSI/GBK; mixed-encoding vaults merged from different sources; a .md file accidentally concatenated with a binary asset.","solutions":["Identify the offending file from the reported [%s] relative path and re-save it as UTF-8 (iconv -f GBK -t UTF-8 file.md, or open in an editor and choose 'Save as UTF-8').","Batch-convert the whole vault: detect per-file encoding (e.g. with chardet/enca) then iconv each non-UTF-8 file in place, preserving backups.","Remove or move the non-UTF-8 file out of the vault if it is not needed, then re-analyse.","Verify with a quick check (file -i or a UTF-8 validator) that no other markdown in the vault is non-UTF-8 before retrying."],"exampleFix":"// before: file saved as GBK\n//   iconv -f GBK -t UTF-8 notes/old.md -o notes/old.md.utf8\n\n// after: convert in place and retry analysis\niconv -f GBK -t UTF-8 notes/old.md | sponge notes/old.md\nStartObsidianVaultAnalysis(vaultPath)","handlingStrategy":"validation","validationCode":"// Reject non-UTF-8 markdown before analysis.\nfunc vaultAllMarkdownUTF8(root string) (string, error) {\n    var bad string\n    err := filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {\n        if err != nil || d.IsDir() || !strings.HasSuffix(p, \".md\") { return err }\n        b, e := os.ReadFile(p)\n        if e != nil { return e }\n        if !utf8.Valid(b) { bad = p; return errors.New(\"non-utf8\") }\n        return nil\n    })\n    return bad, err\n}","typeGuard":"null","tryCatchPattern":"null","preventionTips":["Standardise vault files on UTF-8 (no BOM) with a batch iconv pass.","Configure editors to always save Markdown as UTF-8.","Run a UTF-8 validator over the vault before invoking the importer."],"tags":["obsidian-import","encoding","utf-8","markdown"],"backgroundTag":null,"analyzedSha":"251596fc0de2f9528c00c224252fd073a99973f4","analyzedAt":"2026-08-12T21:18:37.123Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}