Observation watermark bump skipped — partial write, row…

Error message

Observation watermark bump skipped — partial write, row marked pending

What it means

syncObservation writes multiple documents to Chroma; if addDocuments writes fewer than requested (partial write), the observation watermark is NOT advanced (a high-water mark would permanently skip the row) and the row is marked pending so a later backfill retries it.

Solutions

Inspect why addDocuments returned fewer writes (Chroma client logs) and fix the batch error.
Rely on the pending mark: the next backfill re-syncs this observation automatically.
Reduce batch size for syncObservation to lower partial-failure likelihood.
Verify Chroma collection health/connectivity if partials recur.

Defensive patterns

Strategy: fallback

Validate before calling

const written = await this.addDocuments(documents);
if (written !== documents.length) console.warn(`partial write ${written}/${documents.length}; expect pending re-sync`);

Try / catch

try { await syncObservation(...); }
catch (e) { /* row remains pending; backfill retries — safe to ignore transient errors */ }

Prevention

Keep observation batch sizes small
Monitor Chroma client errors for recurring partial failures
Ensure pending rows are periodically re-scanned by backfill
Validate document content before batching

When it happens

Trigger: addDocuments returns written < documents.length — e.g. Chroma rejects some embeddings, a batch item fails, or the client returns partial success for the batch.

Common situations: Large batches hitting payload limits, intermittent Chroma client errors mid-batch, malformed document content in one row of the batch.

Related errors

AI-assisted analysis of thedotmack/claude-mem@d8bc9755e7 (2026-09-17). Data as JSON: /api/errors/6c10f71954af861f. Report an issue: GitHub.

Appendix: source

Thrown at src/services/sync/ChromaSync.ts:518

      documentCount: documents.length,
      project
    });

    // Only advance the watermark on a confirmed full write. addDocuments() now
    // returns a written count and tolerates per-batch failures, so a transient
    // Chroma error must NOT mark this observation as synced — otherwise the
    // backfill pass on next boot will skip past it (CodeRabbit review on PR
    // #2282).
    const written = await this.addDocuments(documents);
    if (written === documents.length) {
      ChromaSyncState.clearPending(project, 'observations', [observationId]);
      ChromaSyncState.bump(project, 'observations', observationId);
    } else {
      // Not bumping is not enough: the watermark is a high-water mark, so the
      // next row that does write would skip past this one for good (#3917).
      // Record it as pending so the backfill retries it, like backfillKind().
      ChromaSyncState.markPending(project, 'observations', [observationId]);
      logger.warn('CHROMA_SYNC', 'Observation watermark bump skipped — partial write, row marked pending', {
        observationId,
        project,
        requested: documents.length,
        written
      });
    }
  }

  async syncSummary(
    summaryId: number,
    memorySessionId: string,
    project: string,
    summary: ParsedSummary,
    promptNumber: number,
    createdAtEpoch: number,
    platformSource?: string
  ): Promise<void> {
    const stored: StoredSummary = {

View on GitHub (pinned to d8bc9755e7)