ruvnet/ruflo · error

Batch ${batchIndex} failed after ${attempt} attempts: ${last

Error message

Batch ${batchIndex} failed after ${attempt} attempts: ${lastError?.message}

What it means

Thrown by processSingleBatch in the ruvector streaming pipeline after the retry budget for one batch is exhausted. Each batch is attempted up to options.maxRetries with exponential backoff (1s, 2s, 4s... capped at 10s); a 'batch_error' event is emitted per failed attempt. When the processor callback keeps failing (or retryOnFailure is false, failing on the first attempt), the loop breaks and this aggregate error is thrown, wrapping the last underlying error message.

Source

Thrown at v3/@claude-flow/plugins/src/integrations/ruvector/streaming.ts:1491

      attempt++;
      try {
        const results = await processor(batch);
        this.emit('batch_complete', { batchIndex, attempt, success: true });
        return results;
      } catch (error) {
        lastError = error as Error;
        this.emit('batch_error', { batchIndex, attempt, error: lastError });

        if (!this.options.retryOnFailure || attempt >= this.options.maxRetries) {
          break;
        }

        // Exponential backoff
        await this.sleep(Math.min(1000 * Math.pow(2, attempt - 1), 10000));
      }
    }

    throw new Error(`Batch ${batchIndex} failed after ${attempt} attempts: ${lastError?.message}`);
  }

  /**
   * Execute a single search query.
   */
  private async executeSingleSearch(
    options: VectorSearchOptions
  ): Promise<VectorSearchResult[]> {
    const client = await this.pool.connect();
    try {
      const { sql, params } = this.buildSearchQuery(options);
      const result = await client.query<{
        id: string | number;
        distance: number;
        [key: string]: unknown;
      }>(sql, params);

      const metric = options.metric ?? 'cosine';

View on GitHub (pinned to fa13ee4ad6)

Solutions

1. Read the trailing lastError message in the thrown text - it carries the real DB cause (dimension error, connection refused, undefined table); fix that root cause first
2. For transient outages, raise maxRetries and keep retryOnFailure true so the exponential backoff spans the outage window
3. Subscribe to the 'batch_error' event to log each attempt and distinguish always-failing validation errors from flaky infrastructure errors
4. Verify vector dimensions match the column type (e.g. vector(1536)) and that the target table/schema exists before streaming
5. For huge ingest jobs, use a smaller batchSize with onBatchComplete checkpoints so a single bad batch does not abort the entire stream

Example fix

// before
const streamer = new StreamingVectorClient(pool, {
  batchSize: 1000,
  maxRetries: 1, // single attempt, any transient failure throws
});

// after
const streamer = new StreamingVectorClient(pool, {
  batchSize: 500,
  maxRetries: 5,
  retryOnFailure: true,
});
streamer.on('batch_error', e =>
  logger.warn(`batch ${e.batchIndex} attempt ${e.attempt}: ${e.error.message}`)
);

// Pre-flight the ingest target so retries are only spent on transient issues
// (e.g. confirm the pgvector column dimension matches your data before streaming)
const expectedDim = vectors[0]?.length;
const colDim = await probeColumnDimension(client, tableName);
if (expectedDim !== undefined && colDim !== expectedDim) {
  throw new Error(`dimension mismatch: column ${colDim} vs data ${expectedDim}`);
}

try {
  for await (const r of streamer.streamInsert(vectors, opts)) { /* ... */ }
} catch (err) {
  if (err instanceof Error && /Batch \d+ failed after \d+ attempts/.test(err.message)) {
    // inspect the trailing cause; checkpoint and resume from last completed batch
    await resumeFromCheckpoint(lastCompletedBatchIndex);
  } else {
    throw err;
  }
}

When it happens

Trigger: Running streaming batch processing (e.g. streamInsert with concurrent batches) where the processor callback hits persistent DB errors: connection dropped by Postgres, malformed vector literal (dimension mismatch), unique-constraint violation with upsert off, target table missing. Also triggered deterministically when options.retryOnFailure is false and the first attempt fails.

Common situations: Vector dimension mismatch between embedded data and the pgvector column (e.g. 1536 vs 768 dims) so every retry fails identically; Postgres restarted or connection pool exhausted mid-ingest so all retries hit the same dead connection; maxRetries misconfigured to 1 while the database is briefly unavailable; ingesting into a table that does not exist in the configured schema.

Related errors

• Transaction is not active. Call begin() first.
• Can only resume paused agent
• Only the original claimant can contest the steal
• No contest to resolve
• Concurrent write detected on aggregate '${aggregateId}'. Res

AI-assisted analysis of ruvnet/ruflo@fa13ee4ad6 (2026-08-18). Data as JSON: /api/errors/68222f1bbb5ddb61. Report an issue: GitHub.