What causes "cachebench: observation line %d: duplicate or invalid JSON" in JuliusBrussee/caveman?

Fix or remove the malformed line identified by the line number.

cachebench: observation line %d: duplicate or invalid JSON

Error message

cachebench: observation line %d: duplicate or invalid JSON

What it means

An observation line failed validUniqueJSONObject — it must be a single valid JSON object with no duplicate keys. The check runs before decoding so malformed or ambiguous lines (duplicate keys would make decoding order-dependent) are rejected deterministically with the 1-based line number.

Source

Thrown at cacheengine/cachebench/observed.go:116

	}
	scanner := bufio.NewScanner(reader)
	initial := 64 * 1024
	if limits.MaxLineBytes < initial {
		initial = limits.MaxLineBytes
	}
	scanner.Buffer(make([]byte, initial), limits.MaxLineBytes)
	seen := map[string]bool{}
	var records []ObservationRecord
	for line := 1; scanner.Scan(); line++ {
		raw := bytes.TrimSpace(scanner.Bytes())
		if len(raw) == 0 {
			continue
		}
		if len(records) >= limits.MaxRecords {
			return nil, fmt.Errorf("cachebench: observations exceed record limit %d", limits.MaxRecords)
		}
		if !validUniqueJSONObject(raw) {
			return nil, fmt.Errorf("cachebench: observation line %d: duplicate or invalid JSON", line)
		}
		decoder := json.NewDecoder(bytes.NewReader(raw))
		decoder.DisallowUnknownFields()
		var record ObservationRecord
		if err := decoder.Decode(&record); err != nil {
			return nil, fmt.Errorf("cachebench: observation line %d: %w", line, err)
		}
		var trailing any
		if err := decoder.Decode(&trailing); err != io.EOF {
			return nil, fmt.Errorf("cachebench: observation line %d: trailing JSON", line)
		}
		if record.Schema != ObservationSchema {
			return nil, fmt.Errorf("cachebench: observation line %d: schema %q", line, record.Schema)
		}
		if strings.TrimSpace(record.RequestID) == "" || seen[record.RequestID] {
			return nil, fmt.Errorf("cachebench: observation line %d: empty or duplicate request_id", line)
		}
		seen[record.RequestID] = true

View on GitHub (pinned to 27d5a3981a)

Solutions

Fix or remove the malformed line identified by the line number.
Re-emit observations from the source tool rather than post-processing with scripts that can duplicate keys.
Validate the file with a strict JSON linter (duplicate-key detection) before loading.

Defensive patterns

Strategy: validation

Validate before calling

func validateObservationFile(path string) error {
	data, err := os.ReadFile(path)
	if err != nil {
		return err
	}
	for i, line := range strings.Split(string(data), "\n") {
		line = strings.TrimSpace(line)
		if line == "" {
			continue
		}
		if !validUniqueJSONObject([]byte(line)) {
			return fmt.Errorf("observation line %d: duplicate or invalid JSON", i+1)
		}
	}
	return nil
}

Type guard

func isJSONObject(b []byte) bool {
	t := bytes.TrimSpace(b)
	return len(t) > 0 && t[0] == '{' && json.Valid(t)
}

Prevention

Write observations with encoding/json Marshal (one object per line).
Avoid post-processing observation files with tools that merge keys.
Check for line truncation if logs pass through size-limited shippers.

When it happens

Trigger: Reading observations where line N is not a JSON object (array, bare string, truncated JSON) or contains repeated keys like two 'request_id' members.

Common situations: Log shipping that truncates long lines; observation emitters that merge maps and duplicate keys; files with BOM or garbage interleaved into lines.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/e709c16c574fcf08. Report an issue: GitHub.