weaviate/weaviate · error

unmarshal new reindex payload: %w

Error message

unmarshal new reindex payload: %w

What it means

ReindexProvider.CheckConflict — the distributed-task ConflictDetector run on every node during RAFT-apply of AddTask — first json.Unmarshals the new task's payload into ReindexTaskPayload. If that fails it returns 'unmarshal new reindex payload: %w'. Because the function must be a pure, FSM-deterministic transform of its arguments, a payload that cannot be parsed is rejected outright rather than guessed at.

Source

Thrown at adapters/repos/db/reindex_conflict.go:146

// CheckConflict implements [distributedtask.ConflictDetector] for the
// reindex namespace. Called under [Manager.mu] from the RAFT-apply
// AddTask path BEFORE the new task is appended to FSM-stored state.
// Returns a non-nil error iff `newPayload` would conflict with an
// already-STARTED task in `existingTasks`.
//
// FSM-determinism: every node applies the same RAFT log entry, sees
// the same `existingTasks` snapshot, and runs this same function — so
// every node reaches the same accept/reject decision. The function
// must remain a pure transform of its arguments.
//
// Conflict rule: any two reindex migrations on overlapping properties
// of the same collection conflict, regardless of which bucket type
// they primarily write to. See [typesConflictReason] for the
// rationale.
func (p *ReindexProvider) CheckConflict(newPayload []byte, existingTasks []*distributedtask.Task) error {
	var newP ReindexTaskPayload
	if err := json.Unmarshal(newPayload, &newP); err != nil {
		return fmt.Errorf("unmarshal new reindex payload: %w", err)
	}
	if newP.Collection == "" || newP.MigrationType == "" {
		return fmt.Errorf("new reindex payload missing Collection or MigrationType")
	}

	for _, task := range existingTasks {
		// PREPARING and SWAPPING are the subtle ones: every unit has
		// reached terminal state, but the post-completion callbacks have
		// not yet committed. A new migration on the same property could
		// land before the schema flip commits, leaving it and the
		// unfinished swap racing on the same bucket pointers.
		if !task.Status.IsActive() {
			continue
		}

		var existP ReindexTaskPayload
		if err := json.Unmarshal(task.Payload, &existP); err != nil {
			// Existing task has an unparseable payload. We can't prove

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Marshal the payload via the ReindexTaskPayload type (json.Marshal) instead of hand-building JSON, and validate the round-trip before submitting.
2. Log/print the wrapped json error — it names the exact offset/type mismatch; fix that field in the payload.
3. Check Weaviate version skew across the cluster; align versions so all nodes parse the same payload schema.

Example fix

// before
payload := []byte(fmt.Sprintf(`{"collection":%s}`, collection)) // malformed JSON
// after
type ReindexTaskPayload struct{ Collection string; MigrationType string; Properties []string }
newP := ReindexTaskPayload{Collection: collection, MigrationType: mt, Properties: props}
payload, err := json.Marshal(newP)
if err != nil { return err }

func validReindexPayload(b []byte) error {
    var p ReindexTaskPayload
    if err := json.Unmarshal(b, &p); err != nil { return err }
    if p.Collection == "" || p.MigrationType == "" { return errors.New("missing Collection or MigrationType") }
    return nil
}

if err := provider.CheckConflict(payload, tasks); err != nil {
    var jsonErr *json.SyntaxError
    if errors.As(err, &jsonErr) && strings.HasPrefix(err.Error(), "unmarshal new reindex payload") {
        return fmt.Errorf("malformed reindex payload at offset %d: %w", jsonErr.Offset, err)
    }
    return err
}

When it happens

Trigger: Submitting a reindex (distributed) task whose payload bytes are not valid JSON or do not match ReindexTaskPayload's shape (wrong field types, truncated payload, non-JSON bytes).

Common situations: Client tooling building payloads by hand with wrong JSON shape; version skew where a newer node writes payload fields an older decoder rejects; corrupted task payload in the RAFT log.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• new reindex payload missing Collection or MigrationType
• unmarshal pre-calculated usage from disk: %w
• unknown data type ArrayEach %v
• unknown data type %v
• could not cast generative result additional prop

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/62fa163eacb3ee61. Report an issue: GitHub.