What causes "failed to create container(%s): %v" in hashicorp/nomad?

Check disk space and permissions on the alloc directory (and alloc/container) for the Nomad user

failed to create container(%s): %v

Error message

failed to create container(%s): %v

What it means

Launch wraps failures from libcontainer.Create, which creates the container factory/directory at <alloc_dir>/container and instantiates the container with the prepared config. If the container cannot be created (bad path, factory error, id conflict), the launch aborts with this error including the container id and underlying cause.

Source

Thrown at drivers/shared/executor/executor_linux_cgo.go:207

			NomadResources: &structs.AllocatedTaskResources{},
		}
	}

	l.command = command

	// A container groups processes under the same isolation enforcement
	containerCfg, err := l.newLibcontainerConfig(command)
	if err != nil {
		return nil, fmt.Errorf("failed to configure container(%s): %v", l.id, err)
	}

	if err := l.cleanOldProcessesInCGroup(containerCfg.Cgroups.Path); err != nil {
		return nil, err
	}

	container, err := libcontainer.Create(path.Join(command.TaskDir, "../alloc/container"), l.id, containerCfg)
	if err != nil {
		return nil, fmt.Errorf("failed to create container(%s): %v", l.id, err)
	}
	l.container = container

	// Look up the binary path and make it executable
	taskPath, hostPath, err := lookupTaskBin(command)
	if err != nil {
		return nil, err
	}
	if err := makeExecutable(hostPath); err != nil {
		return nil, err
	}

	combined := append([]string{taskPath}, command.Args...)
	stdout, err := command.Stdout()
	if err != nil {
		return nil, err
	}
	stderr, err := command.Stderr()

View on GitHub (pinned to 482b49bf1a)

Solutions

Check disk space and permissions on the alloc directory (and alloc/container) for the Nomad user
Verify the alloc dir wasn't garbage-collected or deleted while the task was starting; inspect client logs around the launch
Clear stale state under <alloc_dir>/container and retry the allocation
Confirm TaskDir layout matches Nomad expectations (TaskDir sits inside alloc_dir with sibling alloc/ directory)

Defensive patterns

Strategy: validation

Validate before calling

allocDir := filepath.Dir(cmd.TaskDir) // TaskDir must be <alloc>/<name>
if err := os.MkdirAll(filepath.Join(allocDir, "container"), 0o755); err != nil {
    return fmt.Errorf("alloc/container not creatable: %w", err)
}
if stat, err := os.Stat(allocDir); err != nil || !isWritable(stat) {
    return fmt.Errorf("alloc dir missing or not writable")
}

Try / catch

if _, err := executor.Launch(cmd); err != nil && strings.Contains(err.Error(), "failed to create container") {
    log.Printf("container create failed: %v — check disk/alloc dir", err)
}

Prevention

Keep free disk space on the host volume backing alloc dirs
Prevent alloc GC from racing task launches (check client logs for GC timing)
Clean stale alloc/container state before retrying allocations

When it happens

Trigger: libcontainer.Create(path.Join(command.TaskDir, "../alloc/container"), l.id, containerCfg) errors: alloc/container directory cannot be created or written, factory initialization fails, or the container id is unusable.

Common situations: Alloc dir permissions broken or disk full so the container directory cannot be created; alloc directory removed mid-launch by GC; task dir not laid out under the expected alloc structure (custom/odd TaskDir); leftover stale container state in alloc/container.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/933140b53c041ff3. Report an issue: GitHub.