{"record":{"id":"5730ce2bc460432b","repo":"Tencent/WeKnora","slug":"invalid-json-content","errorCode":null,"errorMessage":"invalid JSON content","messagePattern":"invalid JSON content","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/infrastructure/docparser/json_converter.go","lineNumber":35,"sourceCode":"// when the current chunk has reached at least this size.\nvar minJSONChunkSize = defaultJSONChunkSize - 200\n\n// jsonToMarkdown converts raw JSON bytes into markdown text\n//\n// Key properties:\n//   - Every output chunk is a **valid JSON object** (not a fragment).\n//   - Nested paths from root to leaf are **fully preserved** in each chunk.\n//   - Arrays are converted to index-keyed dicts so the algorithm is uniform.\n//   - Small objects that fit within maxChunkSize are kept intact (not split).\n//   - The output is a series of fenced ```json code blocks separated by \\n\\n,\n//     which the downstream text chunker can split at block boundaries.\nfunc jsonToMarkdown(data []byte) (string, error) {\n\tdata = trimBOM(data)\n\tif len(data) == 0 {\n\t\treturn \"\", fmt.Errorf(\"empty JSON content\")\n\t}\n\tif !json.Valid(data) {\n\t\treturn \"\", fmt.Errorf(\"invalid JSON content\")\n\t}\n\n\tvar parsed interface{}\n\tif err := json.Unmarshal(data, &parsed); err != nil {\n\t\treturn \"\", fmt.Errorf(\"failed to parse JSON: %w\", err)\n\t}\n\n\t// Normalize: convert top-level arrays to index-keyed dicts\n\tnormalized := listToDictPreprocess(parsed)\n\n\t// If the whole thing fits in one chunk, just format it\n\twholeSize := jsonSize(normalized)\n\tif wholeSize <= defaultJSONChunkSize {\n\t\tformatted := formatValue(normalized)\n\t\treturn wrapCodeBlock(formatted), nil\n\t}\n\n\t// Recursive split","sourceCodeStart":17,"sourceCodeEnd":53,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/infrastructure/docparser/json_converter.go#L17-L53","documentation":"After trimming the BOM, jsonToMarkdown runs json.Valid on the raw bytes; if the content is not syntactically valid JSON, it returns \"invalid JSON content\". This fast check runs before json.Unmarshal to give a clear, non-wrapped failure reason.","triggerScenarios":"Read (json_converter) called with bytes that do not parse as JSON: trailing commas, single quotes, comments, concatenated JSON objects, or entirely non-JSON text (HTML/CSV/log lines) in a .json file.","commonSituations":"Hand-edited JSON introducing syntax errors; server returning HTML error page saved as .json; JSONL (newline-delimited) files fed to a single-document JSON parser; encoding issues (UTF-16 files); truncated downloads.","solutions":["Run the content through `json.Valid` or a linter (jq, python -m json.tool) to locate the syntax error and fix it.","Confirm the file is UTF-8 JSON, not UTF-16 or HTML masquerading as .json.","For JSONL, convert to a JSON array (one JSON value per line -> wrap in [ ... ] with commas) before parsing.","Re-download/re-export the file if it is truncated."],"exampleFix":"// before: invalid JSONL\n{\"a\":1}\n{\"a\":2}\n// after: convert to a valid single JSON document\n[{\"a\":1},{\"a\":2}]","handlingStrategy":"validation","validationCode":"data, err := os.ReadFile(path)\nif err != nil { return err }\ndata = bytes.TrimPrefix(data, []byte(\"\\xef\\xbb\\xbf\")) // strip BOM\nif !json.Valid(data) {\n    return fmt.Errorf(\"%s is not valid JSON; validate with jq before parsing\", path)\n}","typeGuard":"func isValidJSON(b []byte) bool {\n    return json.Valid(bytes.TrimPrefix(b, []byte(\"\\xef\\xbb\\xbf\")))\n}","tryCatchPattern":"md, err := converter.Read(ctx, req)\nif err != nil {\n    if strings.Contains(err.Error(), \"invalid JSON content\") {\n        return fmt.Errorf(\"file %s failed JSON validation; run `jq . %s` to locate the syntax error\", req.Path, req.Path)\n    }\n    return err\n}","preventionTips":["Add a json.Valid/jq lint step to your ingestion pipeline before conversion.","Never hand-edit production JSON without validating afterwards.","Convert JSONL to a JSON array before single-document parsing.","Store JSON as UTF-8; avoid UTF-16 exports from Windows tools."],"tags":["json","validation","syntax"],"backgroundTag":"invalid-content-json","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}