Tencent/WeKnora · error

invalid JSON content

Error message

invalid JSON content

What it means

After trimming the BOM, jsonToMarkdown runs json.Valid on the raw bytes; if the content is not syntactically valid JSON, it returns "invalid JSON content". This fast check runs before json.Unmarshal to give a clear, non-wrapped failure reason.

Source

Thrown at internal/infrastructure/docparser/json_converter.go:35

// when the current chunk has reached at least this size.
var minJSONChunkSize = defaultJSONChunkSize - 200

// jsonToMarkdown converts raw JSON bytes into markdown text
//
// Key properties:
//   - Every output chunk is a **valid JSON object** (not a fragment).
//   - Nested paths from root to leaf are **fully preserved** in each chunk.
//   - Arrays are converted to index-keyed dicts so the algorithm is uniform.
//   - Small objects that fit within maxChunkSize are kept intact (not split).
//   - The output is a series of fenced ```json code blocks separated by \n\n,
//     which the downstream text chunker can split at block boundaries.
func jsonToMarkdown(data []byte) (string, error) {
	data = trimBOM(data)
	if len(data) == 0 {
		return "", fmt.Errorf("empty JSON content")
	}
	if !json.Valid(data) {
		return "", fmt.Errorf("invalid JSON content")
	}

	var parsed interface{}
	if err := json.Unmarshal(data, &parsed); err != nil {
		return "", fmt.Errorf("failed to parse JSON: %w", err)
	}

	// Normalize: convert top-level arrays to index-keyed dicts
	normalized := listToDictPreprocess(parsed)

	// If the whole thing fits in one chunk, just format it
	wholeSize := jsonSize(normalized)
	if wholeSize <= defaultJSONChunkSize {
		formatted := formatValue(normalized)
		return wrapCodeBlock(formatted), nil
	}

	// Recursive split

View on GitHub (pinned to 988cbb0330)

Solutions

1. Run the content through `json.Valid` or a linter (jq, python -m json.tool) to locate the syntax error and fix it.
2. Confirm the file is UTF-8 JSON, not UTF-16 or HTML masquerading as .json.
3. For JSONL, convert to a JSON array (one JSON value per line -> wrap in [ ... ] with commas) before parsing.
4. Re-download/re-export the file if it is truncated.

Example fix

// before: invalid JSONL
{"a":1}
{"a":2}
// after: convert to a valid single JSON document
[{"a":1},{"a":2}]

data, err := os.ReadFile(path)
if err != nil { return err }
data = bytes.TrimPrefix(data, []byte("\xef\xbb\xbf")) // strip BOM
if !json.Valid(data) {
    return fmt.Errorf("%s is not valid JSON; validate with jq before parsing", path)
}

func isValidJSON(b []byte) bool {
    return json.Valid(bytes.TrimPrefix(b, []byte("\xef\xbb\xbf")))
}

md, err := converter.Read(ctx, req)
if err != nil {
    if strings.Contains(err.Error(), "invalid JSON content") {
        return fmt.Errorf("file %s failed JSON validation; run `jq . %s` to locate the syntax error", req.Path, req.Path)
    }
    return err
}

When it happens

Trigger: Read (json_converter) called with bytes that do not parse as JSON: trailing commas, single quotes, comments, concatenated JSON objects, or entirely non-JSON text (HTML/CSV/log lines) in a .json file.

Common situations: Hand-edited JSON introducing syntax errors; server returning HTML error page saved as .json; JSONL (newline-delimited) files fed to a single-document JSON parser; encoding issues (UTF-16 files); truncated downloads.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• empty JSON content
• invite code has expired
• join request not found
• failed to retrieve: %s
• opensearch: index not found

AI-assisted analysis of Tencent/WeKnora@988cbb0330 (2026-09-02). Data as JSON: /api/errors/5730ce2bc460432b. Report an issue: GitHub.