hashicorp/nomad · error

error getting snapshot from previous alloc %q: %w

Error message

error getting snapshot from previous alloc %q: %w

What it means

After building the local alloc dir, migrateAllocDir issues an HTTP GET to the OLD node's /v1/client/allocation/<id>/snapshot endpoint (with the migration token as AuthToken) to download a tar snapshot of the previous alloc dir. Any failure of that Raw().Response call (network error, old agent down, TLS mismatch, rejected token) is wrapped as 'error getting snapshot from previous alloc %q: %w'; the freshly built prevAllocDir is destroyed and migration fails.

Source

Thrown at client/allocwatcher/alloc_watcher.go:547

	apiConfig := nomadapi.DefaultConfig()
	apiConfig.Address = nodeAddr
	apiConfig.TLSConfig = &nomadapi.TLSConfig{
		CACert:        p.config.TLSConfig.CAFile,
		ClientCert:    p.config.TLSConfig.CertFile,
		ClientKey:     p.config.TLSConfig.KeyFile,
		TLSServerName: fmt.Sprintf("client.%s.nomad", p.config.Region),
	}
	apiClient, err := nomadapi.NewClient(apiConfig)
	if err != nil {
		return nil, err
	}

	url := fmt.Sprintf("/v1/client/allocation/%v/snapshot", p.prevAllocID)
	qo := &nomadapi.QueryOptions{AuthToken: p.migrateToken}
	resp, err := apiClient.Raw().Response(url, qo)
	if err != nil {
		prevAllocDir.Destroy()
		return nil, fmt.Errorf("error getting snapshot from previous alloc %q: %w", p.prevAllocID, err)
	}

	if err := p.streamAllocDir(ctx, resp, prevAllocDir.AllocDir); err != nil {
		prevAllocDir.Destroy()
		return nil, err
	}

	return prevAllocDir, nil
}

// stream remote alloc to dir to a local path. Caller should cleanup dest on
// error.
func (p *remotePrevAlloc) streamAllocDir(ctx context.Context, resp io.ReadCloser, dest string) error {
	p.logger.Debug("streaming snapshot of previous alloc", "destination", dest)
	tr := tar.NewReader(resp)
	defer resp.Close()

	// Cache effective uid as we only run Chown if we're root

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the previous node's agent is up and its HTTP API is reachable at nodeAddr (curl the address) and that the port/firewall allow it
2. Check client TLS configuration consistency (ca_cert/cert/key vs http address scheme) so the HTTPS request validates
3. Ensure the migration token is fresh: retry the allocation/reschedule so a new migrate token is minted
4. Confirm the previous node is still a cluster member and nodeAddr equals its advertised HTTP address; fix advertise/rpc/serf config if stale

Example fix

// before (client config)
http = "127.0.0.1:4646"  // unreachable from other nodes
// after
bind_addr = "0.0.0.0"
ports { http = 4646 }  # open port 4646 between client nodes for alloc migration

func prevNodeReachable(nodeAddr string) error {
	resp, err := http.Get(nodeAddr + "/v1/agent/health")
	if err != nil {
		return fmt.Errorf("previous node %s unreachable: %w", nodeAddr, err)
	}
	resp.Body.Close()
	return nil
}

alloc, err := watcher.Migrate()
if err != nil {
	if strings.Contains(err.Error(), "error getting snapshot from previous alloc") {
		// transient network/agent issue: retry with backoff before rescheduling elsewhere
	}
	return err
}

When it happens

Trigger: apiClient.Raw().Response("/v1/client/allocation/<prevAllocID>/snapshot", qo) returns an error: previous node's HTTP API unreachable (wrong nodeAddr), agent dead or restarted, connection refused/timeout, TLS certificate/CA mismatch, or the migrateToken is invalid/expired so the request is rejected.

Common situations: Previous node drained or decommissioned mid-migration; firewall between nodes blocking the client HTTP port; nodeAddr resolved to a stale/wrong advertise address; TLS configured on clients but API address uses http (or vice versa); migration token expired after long queue wait.

Related errors

• failed querying self endpoint: %s
• error streaming previous alloc %q for new alloc %q: %w
• unable to get validation keys from JWKS: %v
• unable to get validation keys from OIDC provider: %v
• invalid digest format

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/34a5daf6be836bf3. Report an issue: GitHub.