HumanSignal/label-studio · error · ValidationError

Remove duplicates failed, operation is not finished: queryse

Error message

Remove duplicates failed, operation is not finished: queryset count {queryset.count()} != removing {len(removing)}. It means that some of duplicated tasks have been annotated twice or more.

What it means

remove_duplicated_tasks deduplicates unannotated tasks, but before deleting it re-filters the queryset to id__in=removing with annotations__isnull=True and asserts the count equals the planned removal set. If some planned duplicates gained annotations concurrently (or were removed), it aborts with ValidationError rather than deleting annotated tasks.

Source

Thrown at label_studio/data_manager/actions/remove_duplicates.py:94

                one_task_saved = True
            else:
                new_root.append(task)

        for task in new_root:
            # keep the first task in safety
            if not one_task_saved:
                one_task_saved = True
            # remove all other tasks
            else:
                removing.append(task['id'])

    # get the final queryset for removing tasks
    queryset = queryset.filter(id__in=removing, annotations__isnull=True)
    kept = queryset.exclude(id__in=removing, annotations__isnull=True)

    # check that we don't remove tasks with annotations
    if queryset.count() != len(removing):
        raise ValidationError(
            f'Remove duplicates failed, operation is not finished: '
            f'queryset count {queryset.count()} != removing {len(removing)}. '
            'It means that some of duplicated tasks have been annotated twice or more.'
        )

    delete_tasks(project, queryset)
    logger.info(f'Removed {len(removing)} duplicated tasks')
    return kept


def move_annotations(duplicates):
    """Move annotations to the first task from duplicated tasks"""
    total_moved_annotations = 0

    for data in duplicates:
        root = duplicates[data]
        if len(root) == 1:
            continue

View on GitHub (pinned to 0b49e9b539)

Solutions

1. Pause labeling (set project to a state where annotators cannot submit) and re-run remove_duplicates
2. Re-run the action — the removal set is recomputed from current annotations
3. If you must proceed, annotate/delete the newly annotated duplicates deliberately, then re-run

Example fix

// before
# dedup while labelers are active -> count mismatch abort
// after
project.update(is_published=False)  # pause labeling
run remove_duplicates action
project.update(is_published=True)

// before running dedup
const annotated = removing.filter(id => annotationsExist(id));
if (annotated.length) throw new Error('planned duplicates now have annotations');

null

try {
  await dm.removeDuplicates(projectId);
} catch (e) {
  if (String(e).includes('operation is not finished')) {
    await sleep(RETRY_DELAY);
    return dm.removeDuplicates(projectId); // recomputed removal set
  }
  throw e;
}

When it happens

Trigger: Running the remove_duplicates action when, between planning and execution, tasks in the 'removing' set receive annotations (labelers working concurrently) so queryset.filter(id__in=removing, annotations__isnull=True).count() != len(removing).

Common situations: Labeling activity running during dedup on a live project; race between the action's planning phase and final delete; stale precomputed duplicates after new annotations arrive.

Related errors

• Can't find '{action_id}' in registered actions
• sample() does not accept arguments.
• random(min, max) requires two arguments.
• choices(values:list, weights:list) requires one or two argum
• replace(old_value, new_value) requires two arguments.

AI-assisted analysis of HumanSignal/label-studio@0b49e9b539 (2026-08-29). Data as JSON: /api/errors/5ec02da12f30f55a. Report an issue: GitHub.