{"record":{"id":"efb77c8c29c9ca0e","repo":"shareAI-lab/learn-claude-code","slug":"consolidation-returned-empty-or-duplicate-records","errorCode":null,"errorMessage":"consolidation returned empty or duplicate records","messagePattern":"consolidation returned empty or duplicate records","errorType":"console","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"s09_memory/code.py","lineNumber":487,"sourceCode":"        if len(catalog) > CONSOLIDATE_INPUT_CHAR_LIMIT:\n            raise ValueError(\n                \"memory store is too large for one consolidation pass\"\n            )\n        response = client.messages.create(\n            model=MODEL,\n            messages=[{\"role\": \"user\", \"content\": prompt}],\n            max_tokens=3000,\n        )\n        consolidated = [\n            validated\n            for item in extract_json_array(\n                message_text({\"content\": response.content})\n            )\n            if (validated := validate_memory_record(item)) is not None\n        ]\n        slugs = [memory_slug(record[\"name\"]) for record in consolidated]\n        if not consolidated or len(slugs) != len(set(slugs)):\n            raise ValueError(\n                \"consolidation returned empty or duplicate records\"\n            )\n\n        snapshot = {\n            record[\"filename\"]: memory_path(record[\"filename\"]).read_text()\n            for record in records\n        }\n        try:\n            for path in MEMORY_DIR.glob(\"*.md\"):\n                if path.name != MEMORY_INDEX.name:\n                    try:\n                        memory_path(path.name).unlink()\n                    except ValueError:\n                        continue\n            for record in consolidated:\n                path = memory_path(f\"{memory_slug(record['name'])}.md\")\n                path.write_text(memory_document(\n                    record[\"name\"],","sourceCodeStart":469,"sourceCodeEnd":505,"githubUrl":"https://github.com/shareAI-lab/learn-claude-code/blob/985456f4adea6f4df8fbad4112245dbd97444eae/s09_memory/code.py#L469-L505","documentation":"Raised during memory consolidation in s09_memory/code.py:487 as a self-check on the model's response. After the consolidating LLM returns a JSON array, each item is validated with validate_memory_record(); if every item fails validation (consolidated is empty) or two surviving records slugify to the same filename (duplicate names), the routine aborts before deleting any existing files. The duplicate check exists because each record is written as {slug}.md and duplicates would overwrite each other.","triggerScenarios":"The model returns malformed items that all fail validate_memory_record (missing name/type/description/body, unknown type); the model returns an empty array; the model merges two records but keeps identical or slug-colliding names (e.g. 'Editor prefs' and 'editor-prefs'), making len(slugs) != len(set(slugs)). It only triggers after a successful API call.","commonSituations":"Non-deterministic model output drifting outside the requested schema; names differing only in punctuation/case colliding after memory_slug normalization; weaker models ignoring the 'at most 30 records, fields name/type/description/body' instruction.","solutions":["Retry consolidation once or twice — output variance often resolves a schema miss; consider lowering the record count or adding an explicit schema echo to the prompt.","Strengthen the prompt: require each item to have exactly name, type in {user,feedback,project,reference}, description, body, and unique names.","Post-process instead of failing: on duplicate slugs, merge the two records deterministically (append bodies) rather than aborting.","Switch to a model/temperature setting that adheres to JSON structure more reliably."],"exampleFix":"# before\nif not consolidated or len(slugs) != len(set(slugs)):\n    raise ValueError('consolidation returned empty or duplicate records')\n\n# after: retry once, then merge duplicates\nfor attempt in range(2):\n    consolidated = call_consolidate(client, prompt)\n    slugs = [memory_slug(r['name']) for r in consolidated]\n    if consolidated and len(slugs) == len(set(slugs)):\n        break\nelse:\n    merged = merge_by_slug(consolidated)  # append bodies on slug collision\n    consolidated = merged","handlingStrategy":"retry","validationCode":"def consolidation_is_usable(records) -> bool:\n    if not records:\n        return False\n    slugs = [memory_slug(r['name']) for r in records]\n    return len(slugs) == len(set(slugs))","typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    try:\n        return run_consolidation(client, records)\n    except ValueError as e:\n        if 'empty or duplicate' not in str(e) or attempt == 2:\n            raise\n        time.sleep(1)  # vary output on retry","preventionTips":["Validate model output with validate_memory_record() and re-request on schema misses.","Post-merge duplicate slugs deterministically instead of aborting the pass.","Keep a snapshot of the store (the code already snapshots before deleting) until the new set validates."],"tags":["memory","llm","consolidation","data-integrity"],"backgroundTag":null,"analyzedSha":"985456f4adea6f4df8fbad4112245dbd97444eae","analyzedAt":"2026-08-14T22:02:26.028Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}