{"record":{"id":"4a31d224361d1920","repo":"Tencent/WeKnora","slug":"failed-to-index-questions-w","errorCode":null,"errorMessage":"failed to index questions: %w","messagePattern":"failed to index questions: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/application/service/knowledge_process.go","lineNumber":1758,"sourceCode":"\t\t\t\tSourceID:        sourceID,\n\t\t\t\tSourceType:      types.ChunkSourceType,\n\t\t\t\tChunkID:         chunk.ID,\n\t\t\t\tKnowledgeID:     knowledge.ID,\n\t\t\t\tKnowledgeBaseID: knowledge.KnowledgeBaseID,\n\t\t\t\tIsEnabled:       true,\n\t\t\t})\n\t\t}\n\t\tlogger.Debugf(ctx, \"Generated %d questions for chunk %s\", len(questions), chunk.ID)\n\t}\n\tindexEntriesPrepared = len(indexInfoList)\n\n\t// Index generated questions\n\tif len(indexInfoList) > 0 {\n\t\tindexBatchAttempted = true\n\t\tif err := retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList); err != nil {\n\t\t\texitStatus = \"index_questions_failed\"\n\t\t\tlogger.Errorf(ctx, \"Failed to index generated questions: %v\", err)\n\t\t\treturn fmt.Errorf(\"failed to index questions: %w\", err)\n\t\t}\n\t\tindexBatchSucceeded = true\n\t\tlogger.Infof(ctx, \"Successfully indexed %d generated questions for knowledge: %s\", len(indexInfoList), payload.KnowledgeID)\n\t}\n\n\treturn nil\n}\n\n// processQuestionGenerationForChunks generates questions for a batch (window)\n// of text chunks. This is the batched fan-out path (one asynq task per\n// questionGenChunkBatchSize chunks), aligned with the graph-extract\n// TypeChunkExtract pattern: independent retry, per-batch cancellation, and a\n// postprocess.question.batch[i] subspan. The payload carries only chunk ids\n// (never content); content is read fresh here, and all questions for the batch\n// are indexed in a single embedding BatchIndex call.\nfunc (s *knowledgeService) processQuestionGenerationForChunks(ctx context.Context, t *asynq.Task, payload types.QuestionGenerationPayload) (retErr error) {\n\ttaskStartedAt := time.Now()\n\tretryCount, _ := asynq.GetRetryCount(ctx)","sourceCodeStart":1740,"sourceCodeEnd":1776,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/application/service/knowledge_process.go#L1740-L1776","documentation":"Returned by processQuestionGenerationForKnowledge after LLM question generation succeeded but retrieveEngine.BatchIndex failed to embed and upsert the generated question entries into the vector store. BatchIndex embeds indexInfoList with the KB's embedding model and writes them to kb.VectorStoreID; any embedding API or vector-store write failure surfaces here wrapped with %w. The chunk metadata was already updated, so questions exist on chunks but are not searchable.","triggerScenarios":"retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList) errors when len(indexInfoList) > 0 — e.g. embedding provider API failure (rate limit, quota, bad key), vector store bulk-write failure, dimension mismatch between embeddingModel and the collection, or an empty/invalid embedding model resolved from kb.EmbeddingModelID.","commonSituations":"Embedding API key expired or quota exhausted; embedding model changed on the KB so new vectors don't match collection dimension; vector store temporarily down during bulk indexing; oversized batch rejected by the store.","solutions":["Read the wrapped error to distinguish embedding-provider failure vs vector-store write failure","Verify kb.EmbeddingModelID's model matches the vector store collection dimension","Check embedding provider API key, quota, and rate limits","Retry the task — asynq retry will re-run generation; BatchIndex upserts are idempotent per SourceID","Retry with a smaller batch if the store rejected bulk size"],"exampleFix":"// before: single large batch, all-or-nothing\nif err := retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList); err != nil {\n\treturn fmt.Errorf(\"failed to index questions: %w\", err)\n}\n// after: chunked batches with retry\nfor i := 0; i < len(indexInfoList); i += 100 {\n\tend := i + 100\n\tif end > len(indexInfoList) {\n\t\tend = len(indexInfoList)\n\t}\n\tif err := retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList[i:end]); err != nil {\n\t\treturn fmt.Errorf(\"failed to index questions (batch %d-%d): %w\", i, end, err)\n\t}\n}","handlingStrategy":"retry","validationCode":"// preflight: embedding model must resolve and match collection dimension\nembModel, err := s.modelService.GetEmbeddingModel(ctx, kb.EmbeddingModelID)\nif err != nil {\n\treturn fmt.Errorf(\"embedding model unavailable: %w\", err)\n}\nif dim, _ := retrieveEngine.CollectionDimension(ctx); dim != embModel.Dimension() {\n\treturn fmt.Errorf(\"dimension mismatch: collection=%d model=%d — reindex KB\", dim, embModel.Dimension())\n}","typeGuard":"func indexable(items []*types.IndexInfo) bool {\n\tfor _, it := range items {\n\t\tif it == nil || strings.TrimSpace(it.Content) == \"\" || it.SourceID == \"\" {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn len(items) > 0\n}","tryCatchPattern":"if err := retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList); err != nil {\n\tif isRateLimitError(err) || isTimeoutError(err) {\n\t\treturn fmt.Errorf(\"failed to index questions (retryable): %w\", err) // asynq retries\n\t}\n\tlogger.Errorf(ctx, \"permanent index failure: %v\", err)\n\treturn fmt.Errorf(\"failed to index questions: %w\", err)\n}","preventionTips":["Pin kb.EmbeddingModelID — never change it without a full KB reindex","Set up alerting on embedding provider quota/rate limits before fan-out batches run","Use bounded batch sizes in BatchIndex calls","Verify batch upserts are idempotent by SourceID so retries are safe"],"tags":["embedding","vector-store","indexing","batch-write"],"backgroundTag":"vector-index-write-failed","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}