vxcontrol/pentagi · error

failed to remove container: %w

Error message

failed to remove container: %w

What it means

Wraps a non-NotFound error from the Docker ContainerRemove call in RemoveContainer (backend/pkg/docker/client.go), which forces removal with volumes. NotFound is tolerated (container already gone); any other daemon error (permission, dependency, connection failure) is returned wrapped and the container row is not marked deleted.

Source

Thrown at backend/pkg/docker/client.go:765

	return nil
}

func (dc *dockerClient) RemoveContainer(ctx context.Context, containerID string, dbID int64) error {
	logger := dc.logger.WithContext(ctx).WithField("local_id", containerID)
	logger.Info("removing container and associated resources")

	if err := dc.StopContainer(ctx, containerID, dbID); err != nil {
		return fmt.Errorf("failed to stop container: %w", err)
	}

	options := client.ContainerRemoveOptions{
		RemoveVolumes: true,
		Force:         true,
	}
	if _, err := dc.client.ContainerRemove(ctx, containerID, options); err != nil {
		if !cerrdefs.IsNotFound(err) {
			return fmt.Errorf("failed to remove container: %w", err)
		}
		// already gone (removed manually, or a prior call already succeeded);
		// still mark it deleted below so the database row does not go stale.
		logger.WithError(err).Warn("container not found")
	}

	_, err := dc.db.UpdateContainerStatus(ctx, database.UpdateContainerStatusParams{
		Status: database.ContainerStatusDeleted,
		ID:     dbID,
	})
	if err != nil {
		return fmt.Errorf("failed to update container status to deleted: %w", err)
	}

	logger.Info("container removed")

	return nil
}

View on GitHub (pinned to ea665308ba)

Solutions

1. Retry the remove after a short delay — transient mount/busy conditions often clear
2. Check `docker inspect -f '{{.State.Status}}' <id>`; if 'dead' or 'restarting', kill then remove manually
3. Try `docker rm -f <id>` on the host to see the daemon's raw error message
4. Check dockerd logs and volume-driver health if RemoveVolumes is the failing part; consider removing with RemoveVolumes:false and cleaning volumes separately

Example fix

// before
if _, err := dc.client.ContainerRemove(ctx, containerID, options); err != nil {
    if !cerrdefs.IsNotFound(err) {
        return fmt.Errorf("failed to remove container: %w", err)
    }
}
// after
if _, err := dc.client.ContainerRemove(ctx, containerID, options); err != nil && !cerrdefs.IsNotFound(err) {
    logger.WithError(err).Warn("remove failed, retrying once")
    if _, err := dc.client.ContainerRemove(ctx, containerID, options); err != nil && !cerrdefs.IsNotFound(err) {
        return fmt.Errorf("failed to remove container: %w", err)
    }
}

// only attempt removal of containers that exist
inspect, err := dockerCli.ContainerInspect(ctx, id)
if err != nil && cerrdefs.IsNotFound(err) {
    return nil
}

if err := dc.RemoveContainer(ctx, id, dbID); err != nil {
    if strings.Contains(err.Error(), "failed to remove container") {
        time.Sleep(2 * time.Second)
        err = dc.RemoveContainer(ctx, id, dbID) // one retry
    }
}

When it happens

Trigger: dc.client.ContainerRemove returns a non-NotFound error: container still running and force-kill failed, volume/volume-driver removal error (RemoveVolumes:true), device-mount busy, or daemon storage-driver error.

Common situations: Container in 'restarting' or 'dead' state resisting force removal; dangling mounts (e.g. NFS/iSCSI volume driver unresponsive) that block RemoveVolumes; storage-driver corruption on the host; container created by a different daemon context (e.g. after Docker restart with live-restore).

Related errors

• container shutdown failed: %w
• failed to stop container: %w
• failed to create list exec for '%s': %w
• target container is not operational
• failed to purge container '%s': %w

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/16fe4b135a4bb147. Report an issue: GitHub.