Tencent/WeKnora · error

failed to parse JSON: %w

Error message

failed to parse JSON: %w

What it means

Even when json.Valid passes, json.Unmarshal can still fail (e.g. numbers out of range for interface{} decoding); jsonToMarkdown wraps that underlying error as "failed to parse JSON: %w". It means the content looked like JSON but could not be materialized into Go values.

Source

Thrown at internal/infrastructure/docparser/json_converter.go:40

// Key properties:
//   - Every output chunk is a **valid JSON object** (not a fragment).
//   - Nested paths from root to leaf are **fully preserved** in each chunk.
//   - Arrays are converted to index-keyed dicts so the algorithm is uniform.
//   - Small objects that fit within maxChunkSize are kept intact (not split).
//   - The output is a series of fenced ```json code blocks separated by \n\n,
//     which the downstream text chunker can split at block boundaries.
func jsonToMarkdown(data []byte) (string, error) {
	data = trimBOM(data)
	if len(data) == 0 {
		return "", fmt.Errorf("empty JSON content")
	}
	if !json.Valid(data) {
		return "", fmt.Errorf("invalid JSON content")
	}

	var parsed interface{}
	if err := json.Unmarshal(data, &parsed); err != nil {
		return "", fmt.Errorf("failed to parse JSON: %w", err)
	}

	// Normalize: convert top-level arrays to index-keyed dicts
	normalized := listToDictPreprocess(parsed)

	// If the whole thing fits in one chunk, just format it
	wholeSize := jsonSize(normalized)
	if wholeSize <= defaultJSONChunkSize {
		formatted := formatValue(normalized)
		return wrapCodeBlock(formatted), nil
	}

	// Recursive split
	chunks := recursiveJSONSplit(normalized, nil, nil)

	// Convert each chunk dict to a fenced code block
	blocks := make([]string, 0, len(chunks))
	for _, chunk := range chunks {

View on GitHub (pinned to 988cbb0330)

Solutions

1. Inspect the wrapped cause in the error message (%w chain) to see the exact offset/reason reported by encoding/json.
2. Serialize very large integer identifiers as JSON strings instead of bare numbers.
3. Re-export the JSON with a serializer that emits float64-safe numbers (e.g. cap precision or use strings for big values).
4. If you control parsing, use json.Decoder with UseNumber() — otherwise this library path cannot be changed, so fix the input data.

Example fix

// before
{"id": 922337203685477580712345}
// after
{"id": "922337203685477580712345"}

var probe interface{}
if err := json.Unmarshal(data, &probe); err != nil {
    return fmt.Errorf("content will fail conversion: %w", err)
}

var jsonNumErr *json.UnmarshalTypeError
md, err := converter.Read(ctx, req)
if err != nil {
    if errors.As(err, &jsonNumErr) || strings.Contains(err.Error(), "failed to parse JSON") {
        log.Errorf("JSON decode failed at offset %v: %v", jsonNumErr, err)
        return fmt.Errorf("fix the JSON value reported in the wrapped error and retry")
    }
    return err
}

When it happens

Trigger: Read (json_converter) called with syntactically valid JSON that json.Unmarshal rejects — most commonly numeric literals outside float64 range (huge integers/exponents) or exotic literals that pass a shallow validity check but fail decoding.

Common situations: Machine-generated JSON with 64-bit+ integer IDs serialized as bare numbers; scientific-notation numbers beyond float64; content valid per json.Valid but rejected during decode due to Go's number handling.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• args must be a string or an array of strings: %w
• logical operator %s requires an array of conditions
• failed to unmarshal condition at index %d: %w
• unmarshal page: %w
• unmarshal database: %w

AI-assisted analysis of Tencent/WeKnora@988cbb0330 (2026-09-02). Data as JSON: /api/errors/2b995fd4a8affc2c. Report an issue: GitHub.