{"record":{"id":"60a112d3bfa309c5","repo":"jaegertracing/jaeger","slug":"d-of-d-bulk-items-rejected-s","errorCode":null,"errorMessage":"%d of %d bulk items rejected: %s","messagePattern":"(.+?) of (.+?) bulk items rejected: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/elasticsearch/esclient/sync_bulk.go","lineNumber":235,"sourceCode":"\t// Drop mode: discard poison (terminal) items so the batch can complete. If the\n\t// only failures are terminal, the chunk succeeds (offset advances) with the\n\t// poison logged out-of-band. Transient failures — or fail mode — still error and\n\t// retry the whole batch; any terminal items ride along and are re-dropped each\n\t// retry until the transient ones clear.\n\tif w.dropPoison && out.terminal > 0 {\n\t\tw.logger.Warn(\"dropping poison-pill documents the backend rejected terminally\",\n\t\t\tzap.Int(\"dropped\", out.terminal), zap.Int(\"total\", count), zap.String(\"sample\", msg))\n\t}\n\tif w.dropPoison && out.transient == 0 {\n\t\treturn count - out.terminal, nil\n\t}\n\trejected := failed\n\tif w.dropPoison {\n\t\trejected = out.transient // terminal items were dropped above, not retried\n\t}\n\tw.logger.Error(\"synchronous bulk write had rejected items\",\n\t\tzap.Int(\"rejected\", rejected), zap.Int(\"total\", count))\n\treturn count - failed, fmt.Errorf(\"%d of %d bulk items rejected: %s\", rejected, count, msg)\n}\n\n// encodeBulkItem renders one document as its two NDJSON lines: the action line\n// ({\"index\":{\"_index\":…}} or {\"create\":…}) and the source line.\nfunc encodeBulkItem(item BulkItem) ([]byte, error) {\n\taction := string(item.OpType)\n\tif action == \"\" {\n\t\taction = string(es.WriteOpIndex)\n\t}\n\tmeta := map[string]map[string]string{action: {\"_index\": item.Index}}\n\tif item.ID != \"\" {\n\t\tmeta[action][\"_id\"] = item.ID\n\t}\n\tvar marshalErrors []error\n\tmetaLine, err := json.Marshal(meta)\n\tmarshalErrors = append(marshalErrors, err)\n\tsource, err := json.Marshal(item.Body)\n\tmarshalErrors = append(marshalErrors, err)","sourceCodeStart":217,"sourceCodeEnd":253,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/elasticsearch/esclient/sync_bulk.go#L217-L253","documentation":"The transport succeeded and the response parsed, but the backend rejected one or more documents in the chunk (non-2xx per-item status, error object, or malformed item result). The error lists rejected/total counts plus a bounded sample of per-item reasons (index, status, truncated backend error). The returned count reflects items that were durably stored; the caller retries the batch.","triggerScenarios":"Per-item rejections such as mapping errors (e.g. a string field clashing with an existing numeric mapping), index_not_found, version conflicts under op_type:create, 429 es_rejected_execution on backpressure, 5xx during shard failures, or oversized fields. Behavior differs by PoisonHandling: with dropPoison, terminal (4xx) items are dropped and only transient items (429/5xx/malformed) produce this error; without it, any rejection fails the batch.","commonSituations":"Index template/mapping drift after upgrading Jaeger (old indices with incompatible mappings); cluster under load returning 429; indices deleted while writes are in flight; duplicate-create 409s (these are treated as benign/durable, not this error); Rollover/IASS pipelines not applied so target indices are missing.","solutions":["Read the per-item reason in the error message (index=..., status=..., error=...) and fix the specific backend cause — most often a mapping conflict: delete/reindex the offending index or update the template.","If status is 429, scale the cluster or reduce batch size/worker concurrency to absorb backpressure.","If status is 404 index_not_found, ensure index templates/rollover init ran (e.g. jaeger init or the index cleaner job).","For deterministic 4xx poison-pill documents, consider enabling poison handling (drop mode) so terminal rejections don't stall the pipeline.","Retry transient (429/5xx) failures with backoff; the writer treats them as retryable."],"exampleFix":"// before: field type changed, items rejected with mapper_parsing_exception\n// jaeger-span-2026-08: duration mapped as long, new docs send \"12.5ms\"\n// after: reindex the old index or fix the template so types match\ncurl -X PUT http://es:9200/_template/jaeger-span -d @jaeger-span-template.json","handlingStrategy":"retry","validationCode":"// check the target index and its mapping before writing\nresp, _ := http.Get(server + \"/_index_template/jaeger-span\")\nif resp.StatusCode != 200 {\n    log.Fatal(\"jaeger index template not installed; run jaeger init / index cleaner\")\n}\nresp.Body.Close()","typeGuard":"// inspect per-item statuses from the sample text the error carries\n// status 429/5xx -> transient, safe to retry; other 4xx -> terminal,\n// fix mapping/document before retrying or enable poison-drop mode\nfunc isTransientBulkRejection(err error) bool {\n    for _, s := range []string{\"status=429\", \"status=500\", \"status=502\", \"status=503\", \"status=504\"} {\n        if strings.Contains(err.Error(), s) { return true }\n    }\n    return false\n}","tryCatchPattern":"err := writer.WriteBatch(ctx, items)\nif err != nil && strings.Contains(err.Error(), \"bulk items rejected\") {\n    if isTransientBulkRejection(err) {\n        retryWithBackoff(ctx, items) // 429/5xx: cluster will recover\n    } else {\n        // terminal: log the per-item reason (index=..., error=...)\n        // and fix the mapping/document; retrying is pointless\n    }\n}","preventionTips":["Keep index templates and rollover jobs in sync with the Jaeger version to avoid mapping conflicts on upgrade.","Size the cluster for peak write rate; 429 es_rejected_execution means backpressure.","Never delete active indices while writers are running.","Enable poison-pill drop handling when a single malformed document should not stall the whole pipeline.","Read the error's sample reasons (index/status/error fields) before choosing retry vs. fix."],"tags":["elasticsearch","bulk-index","mapping-conflict","retryable"],"backgroundTag":"bulk-item-rejected","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}