Tencent/WeKnora · error

failed to index questions: %w

Error message

failed to index questions: %w

What it means

Returned by processQuestionGenerationForKnowledge after LLM question generation succeeded but retrieveEngine.BatchIndex failed to embed and upsert the generated question entries into the vector store. BatchIndex embeds indexInfoList with the KB's embedding model and writes them to kb.VectorStoreID; any embedding API or vector-store write failure surfaces here wrapped with %w. The chunk metadata was already updated, so questions exist on chunks but are not searchable.

Source

Thrown at internal/application/service/knowledge_process.go:1758

				SourceID:        sourceID,
				SourceType:      types.ChunkSourceType,
				ChunkID:         chunk.ID,
				KnowledgeID:     knowledge.ID,
				KnowledgeBaseID: knowledge.KnowledgeBaseID,
				IsEnabled:       true,
			})
		}
		logger.Debugf(ctx, "Generated %d questions for chunk %s", len(questions), chunk.ID)
	}
	indexEntriesPrepared = len(indexInfoList)

	// Index generated questions
	if len(indexInfoList) > 0 {
		indexBatchAttempted = true
		if err := retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList); err != nil {
			exitStatus = "index_questions_failed"
			logger.Errorf(ctx, "Failed to index generated questions: %v", err)
			return fmt.Errorf("failed to index questions: %w", err)
		}
		indexBatchSucceeded = true
		logger.Infof(ctx, "Successfully indexed %d generated questions for knowledge: %s", len(indexInfoList), payload.KnowledgeID)
	}

	return nil
}

// processQuestionGenerationForChunks generates questions for a batch (window)
// of text chunks. This is the batched fan-out path (one asynq task per
// questionGenChunkBatchSize chunks), aligned with the graph-extract
// TypeChunkExtract pattern: independent retry, per-batch cancellation, and a
// postprocess.question.batch[i] subspan. The payload carries only chunk ids
// (never content); content is read fresh here, and all questions for the batch
// are indexed in a single embedding BatchIndex call.
func (s *knowledgeService) processQuestionGenerationForChunks(ctx context.Context, t *asynq.Task, payload types.QuestionGenerationPayload) (retErr error) {
	taskStartedAt := time.Now()
	retryCount, _ := asynq.GetRetryCount(ctx)

View on GitHub (pinned to 988cbb0330)

Solutions

1. Read the wrapped error to distinguish embedding-provider failure vs vector-store write failure
2. Verify kb.EmbeddingModelID's model matches the vector store collection dimension
3. Check embedding provider API key, quota, and rate limits
4. Retry the task — asynq retry will re-run generation; BatchIndex upserts are idempotent per SourceID
5. Retry with a smaller batch if the store rejected bulk size

Example fix

// before: single large batch, all-or-nothing
if err := retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList); err != nil {
	return fmt.Errorf("failed to index questions: %w", err)
}
// after: chunked batches with retry
for i := 0; i < len(indexInfoList); i += 100 {
	end := i + 100
	if end > len(indexInfoList) {
		end = len(indexInfoList)
	}
	if err := retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList[i:end]); err != nil {
		return fmt.Errorf("failed to index questions (batch %d-%d): %w", i, end, err)
	}
}

// preflight: embedding model must resolve and match collection dimension
embModel, err := s.modelService.GetEmbeddingModel(ctx, kb.EmbeddingModelID)
if err != nil {
	return fmt.Errorf("embedding model unavailable: %w", err)
}
if dim, _ := retrieveEngine.CollectionDimension(ctx); dim != embModel.Dimension() {
	return fmt.Errorf("dimension mismatch: collection=%d model=%d — reindex KB", dim, embModel.Dimension())
}

func indexable(items []*types.IndexInfo) bool {
	for _, it := range items {
		if it == nil || strings.TrimSpace(it.Content) == "" || it.SourceID == "" {
			return false
		}
	}
	return len(items) > 0
}

if err := retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList); err != nil {
	if isRateLimitError(err) || isTimeoutError(err) {
		return fmt.Errorf("failed to index questions (retryable): %w", err) // asynq retries
	}
	logger.Errorf(ctx, "permanent index failure: %v", err)
	return fmt.Errorf("failed to index questions: %w", err)
}

When it happens

Trigger: retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList) errors when len(indexInfoList) > 0 — e.g. embedding provider API failure (rate limit, quota, bad key), vector store bulk-write failure, dimension mismatch between embeddingModel and the collection, or an empty/invalid embedding model resolved from kb.EmbeddingModelID.

Common situations: Embedding API key expired or quota exhausted; embedding model changed on the KB so new vectors don't match collection dimension; vector store temporarily down during bulk indexing; oversized batch rejected by the store.

Related errors

• failed to index chunk: %w
• failed to delete chunk vectors: %w
• failed to get embedding model: %w

AI-assisted analysis of Tencent/WeKnora@988cbb0330 (2026-09-02). Data as JSON: /api/errors/4a31d224361d1920. Report an issue: GitHub.