hashicorp/nomad · error

error building alloc dir for previous alloc %q: %w

Error message

error building alloc dir for previous alloc %q: %w

What it means

Nomad's remotePrevAlloc.migrateAllocDir creates a local directory for the previous allocation (via allocdir.NewAllocDir + Build()) before streaming the old node's snapshot into it. When Build() fails (e.g. the client's alloc/mounts root is unwritable or malformed), the error is wrapped as 'error building alloc dir for previous alloc %q: %w' and the migration aborts.

Source

Thrown at client/allocwatcher/alloc_watcher.go:525

	if resp.Node == nil {
		return "", fmt.Errorf("node %q not found", nodeID)
	}

	scheme := "http://"
	if resp.Node.TLSEnabled {
		scheme = "https://"
	}
	return scheme + resp.Node.HTTPAddr, nil
}

// migrate a remote alloc dir to local node. Caller is responsible for calling
// Destroy on the returned allocdir if no error occurs.
func (p *remotePrevAlloc) migrateAllocDir(ctx context.Context, nodeAddr string) (*allocdir.AllocDir, error) {
	// Create the previous alloc dir
	prevAllocDir := allocdir.NewAllocDir(p.logger, p.config.AllocDir, p.config.AllocMountsDir, p.prevAllocID)
	if err := prevAllocDir.Build(); err != nil {
		return nil, fmt.Errorf("error building alloc dir for previous alloc %q: %w", p.prevAllocID, err)
	}

	// Create an API client
	apiConfig := nomadapi.DefaultConfig()
	apiConfig.Address = nodeAddr
	apiConfig.TLSConfig = &nomadapi.TLSConfig{
		CACert:        p.config.TLSConfig.CAFile,
		ClientCert:    p.config.TLSConfig.CertFile,
		ClientKey:     p.config.TLSConfig.KeyFile,
		TLSServerName: fmt.Sprintf("client.%s.nomad", p.config.Region),
	}
	apiClient, err := nomadapi.NewClient(apiConfig)
	if err != nil {
		return nil, err
	}

	url := fmt.Sprintf("/v1/client/allocation/%v/snapshot", p.prevAllocID)
	qo := &nomadapi.QueryOptions{AuthToken: p.migrateToken}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the client's alloc_dir/data_dir exists, is a directory, and is writable by the Nomad agent user (ls -ld, touch test); fix permissions or path
2. Free disk space / remount the volume read-write if the filesystem is full or read-only
3. Remove any non-directory file or stale leftover at <alloc_dir>/<prevAllocID> then retry the allocation
4. Fix SELinux/AppArmor policies or run the agent with the intended user; then reschedule the allocation

Example fix

// before
alloc_dir = "/mnt/nomad"  // mounted read-only or unwritable by nomad user
// after
alloc_dir = "/opt/nomad/data"  # chown nomad:nomad /opt/nomad/data && chmod 750

import "os"

func allocDirWritable(dir string) error {
	if fi, err := os.Stat(dir); err != nil {
		return err
	} else if !fi.IsDir() {
		return fmt.Errorf("%s is not a directory", dir)
	}
	f, err := os.CreateTemp(dir, ".writecheck*")
	if err != nil {
		return err
	}
	f.Close()
	os.Remove(f.Name())
	return nil
}

prevAllocDir, err := migrateAllocDir(ctx, nodeAddr)
if err != nil {
	if strings.Contains(err.Error(), "error building alloc dir") {
		// inspect unwrapped cause: check disk space/permissions on alloc_dir
	}
	return err
}

When it happens

Trigger: Calling Migrate (via prevAllocWatcher) for a remote previous allocation when prevAllocDir.Build() fails: the client data directory (config.AllocDir or AllocMountsDir) does not exist or cannot be created, permissions deny mkdir, the filesystem is full or read-only, or the path collides with a non-directory file.

Common situations: Nomad client data_dir on a full disk or read-only volume; wrong permissions after running the agent as a different user; SELinux/AppArmor blocking directory creation; leftover file at the previous-alloc path blocking mkdir; misconfigured client { alloc_dir } in the agent config.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

• unable to move to %q - alloc dir is not built
• error moving data dir: %w
• error creating task %q dir: %w
• error moving task %q local dir: %w
• failed to remove alloc dir %q: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/61a0be14a57c7e4a. Report an issue: GitHub.