{"record":{"id":"16fe4b135a4bb147","repo":"vxcontrol/pentagi","slug":"failed-to-remove-container-w","errorCode":null,"errorMessage":"failed to remove container: %w","messagePattern":"failed to remove container: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"backend/pkg/docker/client.go","lineNumber":765,"sourceCode":"\n\treturn nil\n}\n\nfunc (dc *dockerClient) RemoveContainer(ctx context.Context, containerID string, dbID int64) error {\n\tlogger := dc.logger.WithContext(ctx).WithField(\"local_id\", containerID)\n\tlogger.Info(\"removing container and associated resources\")\n\n\tif err := dc.StopContainer(ctx, containerID, dbID); err != nil {\n\t\treturn fmt.Errorf(\"failed to stop container: %w\", err)\n\t}\n\n\toptions := client.ContainerRemoveOptions{\n\t\tRemoveVolumes: true,\n\t\tForce:         true,\n\t}\n\tif _, err := dc.client.ContainerRemove(ctx, containerID, options); err != nil {\n\t\tif !cerrdefs.IsNotFound(err) {\n\t\t\treturn fmt.Errorf(\"failed to remove container: %w\", err)\n\t\t}\n\t\t// already gone (removed manually, or a prior call already succeeded);\n\t\t// still mark it deleted below so the database row does not go stale.\n\t\tlogger.WithError(err).Warn(\"container not found\")\n\t}\n\n\t_, err := dc.db.UpdateContainerStatus(ctx, database.UpdateContainerStatusParams{\n\t\tStatus: database.ContainerStatusDeleted,\n\t\tID:     dbID,\n\t})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to update container status to deleted: %w\", err)\n\t}\n\n\tlogger.Info(\"container removed\")\n\n\treturn nil\n}","sourceCodeStart":747,"sourceCodeEnd":783,"githubUrl":"https://github.com/vxcontrol/pentagi/blob/ea665308baaff015b226f308438a68d929d0f29b/backend/pkg/docker/client.go#L747-L783","documentation":"Wraps a non-NotFound error from the Docker ContainerRemove call in RemoveContainer (backend/pkg/docker/client.go), which forces removal with volumes. NotFound is tolerated (container already gone); any other daemon error (permission, dependency, connection failure) is returned wrapped and the container row is not marked deleted.","triggerScenarios":"dc.client.ContainerRemove returns a non-NotFound error: container still running and force-kill failed, volume/volume-driver removal error (RemoveVolumes:true), device-mount busy, or daemon storage-driver error.","commonSituations":"Container in 'restarting' or 'dead' state resisting force removal; dangling mounts (e.g. NFS/iSCSI volume driver unresponsive) that block RemoveVolumes; storage-driver corruption on the host; container created by a different daemon context (e.g. after Docker restart with live-restore).","solutions":["Retry the remove after a short delay — transient mount/busy conditions often clear","Check `docker inspect -f '{{.State.Status}}' <id>`; if 'dead' or 'restarting', kill then remove manually","Try `docker rm -f <id>` on the host to see the daemon's raw error message","Check dockerd logs and volume-driver health if RemoveVolumes is the failing part; consider removing with RemoveVolumes:false and cleaning volumes separately"],"exampleFix":"// before\nif _, err := dc.client.ContainerRemove(ctx, containerID, options); err != nil {\n    if !cerrdefs.IsNotFound(err) {\n        return fmt.Errorf(\"failed to remove container: %w\", err)\n    }\n}\n// after\nif _, err := dc.client.ContainerRemove(ctx, containerID, options); err != nil && !cerrdefs.IsNotFound(err) {\n    logger.WithError(err).Warn(\"remove failed, retrying once\")\n    if _, err := dc.client.ContainerRemove(ctx, containerID, options); err != nil && !cerrdefs.IsNotFound(err) {\n        return fmt.Errorf(\"failed to remove container: %w\", err)\n    }\n}","handlingStrategy":"retry","validationCode":"// only attempt removal of containers that exist\ninspect, err := dockerCli.ContainerInspect(ctx, id)\nif err != nil && cerrdefs.IsNotFound(err) {\n    return nil\n}","typeGuard":null,"tryCatchPattern":"if err := dc.RemoveContainer(ctx, id, dbID); err != nil {\n    if strings.Contains(err.Error(), \"failed to remove container\") {\n        time.Sleep(2 * time.Second)\n        err = dc.RemoveContainer(ctx, id, dbID) // one retry\n    }\n}","preventionTips":["Avoid volume drivers that can hang during RemoveVolumes removal","Keep the storage driver healthy; watch dockerd logs for driver errors","Retry removal with backoff — many failures are transient mount-busy states","Fall back to `docker rm -f` runbook steps for stuck 'dead' containers"],"tags":["docker","container-lifecycle","volumes"],"backgroundTag":"docker-container-remove-failed","analyzedSha":"ea665308baaff015b226f308438a68d929d0f29b","analyzedAt":"2026-09-01T14:16:31.421Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}