jaegertracing/jaeger · error

%d of %d bulk items rejected: %s

Error message

%d of %d bulk items rejected: %s

What it means

The transport succeeded and the response parsed, but the backend rejected one or more documents in the chunk (non-2xx per-item status, error object, or malformed item result). The error lists rejected/total counts plus a bounded sample of per-item reasons (index, status, truncated backend error). The returned count reflects items that were durably stored; the caller retries the batch.

Source

Thrown at internal/storage/elasticsearch/esclient/sync_bulk.go:235

	// Drop mode: discard poison (terminal) items so the batch can complete. If the
	// only failures are terminal, the chunk succeeds (offset advances) with the
	// poison logged out-of-band. Transient failures — or fail mode — still error and
	// retry the whole batch; any terminal items ride along and are re-dropped each
	// retry until the transient ones clear.
	if w.dropPoison && out.terminal > 0 {
		w.logger.Warn("dropping poison-pill documents the backend rejected terminally",
			zap.Int("dropped", out.terminal), zap.Int("total", count), zap.String("sample", msg))
	}
	if w.dropPoison && out.transient == 0 {
		return count - out.terminal, nil
	}
	rejected := failed
	if w.dropPoison {
		rejected = out.transient // terminal items were dropped above, not retried
	}
	w.logger.Error("synchronous bulk write had rejected items",
		zap.Int("rejected", rejected), zap.Int("total", count))
	return count - failed, fmt.Errorf("%d of %d bulk items rejected: %s", rejected, count, msg)
}

// encodeBulkItem renders one document as its two NDJSON lines: the action line
// ({"index":{"_index":…}} or {"create":…}) and the source line.
func encodeBulkItem(item BulkItem) ([]byte, error) {
	action := string(item.OpType)
	if action == "" {
		action = string(es.WriteOpIndex)
	}
	meta := map[string]map[string]string{action: {"_index": item.Index}}
	if item.ID != "" {
		meta[action]["_id"] = item.ID
	}
	var marshalErrors []error
	metaLine, err := json.Marshal(meta)
	marshalErrors = append(marshalErrors, err)
	source, err := json.Marshal(item.Body)
	marshalErrors = append(marshalErrors, err)

View on GitHub (pinned to 806f444784)

Solutions

1. Read the per-item reason in the error message (index=..., status=..., error=...) and fix the specific backend cause — most often a mapping conflict: delete/reindex the offending index or update the template.
2. If status is 429, scale the cluster or reduce batch size/worker concurrency to absorb backpressure.
3. If status is 404 index_not_found, ensure index templates/rollover init ran (e.g. jaeger init or the index cleaner job).
4. For deterministic 4xx poison-pill documents, consider enabling poison handling (drop mode) so terminal rejections don't stall the pipeline.
5. Retry transient (429/5xx) failures with backoff; the writer treats them as retryable.

Example fix

// before: field type changed, items rejected with mapper_parsing_exception
// jaeger-span-2026-08: duration mapped as long, new docs send "12.5ms"
// after: reindex the old index or fix the template so types match
curl -X PUT http://es:9200/_template/jaeger-span -d @jaeger-span-template.json

// check the target index and its mapping before writing
resp, _ := http.Get(server + "/_index_template/jaeger-span")
if resp.StatusCode != 200 {
    log.Fatal("jaeger index template not installed; run jaeger init / index cleaner")
}
resp.Body.Close()

// inspect per-item statuses from the sample text the error carries
// status 429/5xx -> transient, safe to retry; other 4xx -> terminal,
// fix mapping/document before retrying or enable poison-drop mode
func isTransientBulkRejection(err error) bool {
    for _, s := range []string{"status=429", "status=500", "status=502", "status=503", "status=504"} {
        if strings.Contains(err.Error(), s) { return true }
    }
    return false
}

err := writer.WriteBatch(ctx, items)
if err != nil && strings.Contains(err.Error(), "bulk items rejected") {
    if isTransientBulkRejection(err) {
        retryWithBackoff(ctx, items) // 429/5xx: cluster will recover
    } else {
        // terminal: log the per-item reason (index=..., error=...)
        // and fix the mapping/document; retrying is pointless
    }
}

When it happens

Trigger: Per-item rejections such as mapping errors (e.g. a string field clashing with an existing numeric mapping), index_not_found, version conflicts under op_type:create, 429 es_rejected_execution on backpressure, 5xx during shard failures, or oversized fields. Behavior differs by PoisonHandling: with dropPoison, terminal (4xx) items are dropped and only transient items (429/5xx/malformed) produce this error; without it, any rejection fails the batch.

Common situations: Index template/mapping drift after upgrading Jaeger (old indices with incompatible mappings); cluster under load returning 429; indices deleted while writes are in flight; duplicate-create 409s (these are treated as benign/durable, not this error); Rollover/IASS pipelines not applied so target indices are missing.

Related errors

• bulk request failed: %w
• failed to parse bulk response: %w
• malformed bulk response: %d item results for %d documents
• file must begin with '['
• max spans count reached

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/60a112d3bfa309c5. Report an issue: GitHub.