{"record":{"id":"372feecb274f21c1","repo":"Tencent/WeKnora","slug":"failed-to-do-bulk-w","errorCode":null,"errorMessage":"failed to do bulk: %w","messagePattern":"failed to do bulk: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/application/repository/retriever/elasticsearch/v8/repository.go","lineNumber":212,"sourceCode":"\tlog.Infof(\"[Elasticsearch] Batch saving %d indices\", len(embeddingList))\n\tindexRequest := e.client.Bulk().Index(e.index)\n\n\t// Add each document to the bulk request\n\tfor _, embedding := range embeddingList {\n\t\tembeddingDB := elasticsearchRetriever.ToDBVectorEmbedding(embedding, additionalParams)\n\t\terr := indexRequest.CreateOp(types.CreateOperation{Index_: &e.index}, embeddingDB)\n\t\tif err != nil {\n\t\t\tlog.Errorf(\"[Elasticsearch] Failed to create bulk operation: %v\", err)\n\t\t\treturn fmt.Errorf(\"failed to create op: %w\", err)\n\t\t}\n\t\tlog.Debugf(\"[Elasticsearch] Added chunk ID %s to bulk request\", embedding.ChunkID)\n\t}\n\n\t// Execute the bulk request\n\t_, err := indexRequest.Do(ctx)\n\tif err != nil {\n\t\tlog.Errorf(\"[Elasticsearch] Failed to execute bulk operation: %v\", err)\n\t\treturn fmt.Errorf(\"failed to do bulk: %w\", err)\n\t}\n\n\tlog.Infof(\"[Elasticsearch] Successfully batch saved %d indices\", len(embeddingList))\n\treturn nil\n}\n\n// DeleteByChunkIDList removes documents from the index based on chunk IDs\n// Returns an error if the delete operation fails\nfunc (e *elasticsearchRepository) DeleteByChunkIDList(ctx context.Context, chunkIDList []string, dimension int, knowledgeType string) error {\n\tlog := logger.GetLogger(ctx)\n\tif len(chunkIDList) == 0 {\n\t\tlog.Warn(\"[Elasticsearch] Empty chunk ID list provided for deletion, skipping\")\n\t\treturn nil\n\t}\n\n\tlog.Infof(\"[Elasticsearch] Deleting indices by chunk IDs, count: %d\", len(chunkIDList))\n\t// Use DeleteByQuery to delete all documents matching the chunk IDs\n\t_, err := e.client.DeleteByQuery(e.index).Query(&types.Query{","sourceCodeStart":194,"sourceCodeEnd":230,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/application/repository/retriever/elasticsearch/v8/repository.go#L194-L230","documentation":"BatchSave wraps any error returned by the go-elasticsearch v8 bulk IndexRequest.Do call with 'failed to do bulk: %w'. The bulk API sends all indexed documents to Elasticsearch in one request, so a failure here means the whole batch was rejected or the request failed (network, mapping, server error). The original ES error is preserved for unwrapping via errors.Is/errors.As.","triggerScenarios":"Calling BatchSave when the Elasticsearch cluster is unreachable, the index mapping rejects a document (e.g. embedding field type conflict), a document is malformed, or the bulk request Do(ctx) returns a connection/timeout/4xx-5xx error.","commonSituations":"Elasticsearch down or misconfigured host/port in env; version mismatch between go-elasticsearch v8 client and an older/newer server; embedding vector dimension mismatch with the index mapping causing mapper_parsing_exception; disk watermark exceeded leading to 429/403 rejection of writes.","solutions":["Check Elasticsearch health (GET /_cluster/health) and connectivity from the app host; verify host/credentials config used to build the client.","Inspect the wrapped error (errors.Unwrap or %v in logs) — a bulk response failure typically names the offending document and reason.","Verify the embedding dimension matches the index mapping (e.g. dense_vector dims); delete/recreate the index if the mapping changed.","Check disk watermark and index write block settings (GET /<index>/_settings) and clear blocks if present.","Retry the batch once connectivity is restored; the operation is idempotent per document ID."],"exampleFix":"// before\nerr := retriever.BatchSave(ctx, items) // panic/log-and-ignore\n// after\nif err := retriever.BatchSave(ctx, items); err != nil {\n\tvar esErr *elastic.Error\n\tif errors.As(err, &esErr) {\n\t\tlog.Printf(\"ES bulk failed status=%d: %s\", esErr.Status, esErr.Error())\n\t}\n\treturn fmt.Errorf(\"batch save embeddings: %w\", err)\n}","handlingStrategy":"try-catch","validationCode":"if len(embeddingList) == 0 { return nil }\nif err := pingElasticsearch(ctx, client); err != nil { return fmt.Errorf(\"es unavailable: %w\", err) }","typeGuard":null,"tryCatchPattern":"if err := retriever.BatchSave(ctx, items); err != nil {\n\tlog.Errorf(\"es bulk save failed: %v\", err) // wrapped root cause preserved\n\treturn err\n}","preventionTips":["Health-check the ES cluster before large batch writes.","Keep embedding dimensions consistent with the index mapping; version the index when mappings change.","Use bounded batch sizes to avoid oversized bulk payloads.","Alert on 429/503 responses (disk watermark, load) rather than retrying blindly."],"tags":["elasticsearch","go","bulk-index","network"],"backgroundTag":"elasticsearch-bulk-failed","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}