{"record":{"id":"5ec02da12f30f55a","repo":"HumanSignal/label-studio","slug":"remove-duplicates-failed-operation-is-not-finishe","errorCode":null,"errorMessage":"Remove duplicates failed, operation is not finished: queryset count {queryset.count()} != removing {len(removing)}. It means that some of duplicated tasks have been annotated twice or more.","messagePattern":"Remove duplicates failed, operation is not finished: queryset count (.+?) != removing (.+?)\\. It means that some of duplicated tasks have been annotated twice or more\\.","errorType":"validation","errorClass":"ValidationError","httpStatus":400,"severity":"error","filePath":"label_studio/data_manager/actions/remove_duplicates.py","lineNumber":94,"sourceCode":"                one_task_saved = True\n            else:\n                new_root.append(task)\n\n        for task in new_root:\n            # keep the first task in safety\n            if not one_task_saved:\n                one_task_saved = True\n            # remove all other tasks\n            else:\n                removing.append(task['id'])\n\n    # get the final queryset for removing tasks\n    queryset = queryset.filter(id__in=removing, annotations__isnull=True)\n    kept = queryset.exclude(id__in=removing, annotations__isnull=True)\n\n    # check that we don't remove tasks with annotations\n    if queryset.count() != len(removing):\n        raise ValidationError(\n            f'Remove duplicates failed, operation is not finished: '\n            f'queryset count {queryset.count()} != removing {len(removing)}. '\n            'It means that some of duplicated tasks have been annotated twice or more.'\n        )\n\n    delete_tasks(project, queryset)\n    logger.info(f'Removed {len(removing)} duplicated tasks')\n    return kept\n\n\ndef move_annotations(duplicates):\n    \"\"\"Move annotations to the first task from duplicated tasks\"\"\"\n    total_moved_annotations = 0\n\n    for data in duplicates:\n        root = duplicates[data]\n        if len(root) == 1:\n            continue","sourceCodeStart":76,"sourceCodeEnd":112,"githubUrl":"https://github.com/HumanSignal/label-studio/blob/0b49e9b53917880baf1dd85d574fe5541a9aafb2/label_studio/data_manager/actions/remove_duplicates.py#L76-L112","documentation":"remove_duplicated_tasks deduplicates unannotated tasks, but before deleting it re-filters the queryset to id__in=removing with annotations__isnull=True and asserts the count equals the planned removal set. If some planned duplicates gained annotations concurrently (or were removed), it aborts with ValidationError rather than deleting annotated tasks.","triggerScenarios":"Running the remove_duplicates action when, between planning and execution, tasks in the 'removing' set receive annotations (labelers working concurrently) so queryset.filter(id__in=removing, annotations__isnull=True).count() != len(removing).","commonSituations":"Labeling activity running during dedup on a live project; race between the action's planning phase and final delete; stale precomputed duplicates after new annotations arrive.","solutions":["Pause labeling (set project to a state where annotators cannot submit) and re-run remove_duplicates","Re-run the action — the removal set is recomputed from current annotations","If you must proceed, annotate/delete the newly annotated duplicates deliberately, then re-run"],"exampleFix":"// before\n# dedup while labelers are active -> count mismatch abort\n// after\nproject.update(is_published=False)  # pause labeling\nrun remove_duplicates action\nproject.update(is_published=True)","handlingStrategy":"retry","validationCode":"// before running dedup\nconst annotated = removing.filter(id => annotationsExist(id));\nif (annotated.length) throw new Error('planned duplicates now have annotations');","typeGuard":"null","tryCatchPattern":"try {\n  await dm.removeDuplicates(projectId);\n} catch (e) {\n  if (String(e).includes('operation is not finished')) {\n    await sleep(RETRY_DELAY);\n    return dm.removeDuplicates(projectId); // recomputed removal set\n  }\n  throw e;\n}","preventionTips":["Pause or gate labeling while dedup runs to avoid concurrent annotations","Re-run the action after a mismatch; the removal set is recomputed safely","Schedule dedup during low-activity windows"],"tags":["label-studio","data-manager","race-condition","validation"],"backgroundTag":"concurrent-modification","analyzedSha":"0b49e9b53917880baf1dd85d574fe5541a9aafb2","analyzedAt":"2026-08-29T00:39:52.578Z","schemaVersion":2},"datasetVersion":"2026-08-29T02:17:18.158Z"}