hashicorp/nomad · error

getter subprocess failed: %v: %v

Error message

getter subprocess failed: %v: %v

What it means

runCmd runs the go-getter as a subprocess with a per-artifact environment; if cmd.Run() returns non-zero, Nomad collects the subprocess log output and wraps both the error and the log message in an Error with Recoverable: true, meaning the task runner may retry the artifact fetch. The message intentionally pairs the exit error with the getter's own log output because go-getter failures (auth, DNS, TLS, checksum) surface only in the subprocess output.

Source

Thrown at client/allocrunner/taskrunner/getter/util.go:274

	// final method of ensuring subprocess termination
	ctx, cancel := subproc.Context(env.deadline())
	defer cancel()

	// start the subprocess, passing in parameters via stdin
	output := new(bytes.Buffer)
	cmd := exec.CommandContext(ctx, bin, SubCommand)
	cmd.Env = environment(env.TaskDir, env.SetEnvironmentVariables)
	cmd.Stdin = env.reader()
	cmd.Stdout = output
	cmd.Stderr = output

	// start & wait for the subprocess to terminate
	if err := cmd.Run(); err != nil {
		msg := subproc.Log(output, s.logger.Error)

		return &Error{
			URL:         env.Source,
			Err:         fmt.Errorf("getter subprocess failed: %v: %v", err, msg),
			Recoverable: true,
		}
	}
	subproc.Log(output, s.logger.Debug)

	// if no root has been defined, no inspection
	// is being performed so return now.
	if at == nil {
		return nil
	}

	// generate the inspector for the destination
	artifactInspector, err := genWalkInspector(env.Destination)
	if err != nil {
		return err
	}

	// inspect the contents to find any unwanted files

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the appended msg (subprocess log) in the error to find the root cause — it names the underlying getter failure
2. Verify the source URL is reachable from the Nomad client node (curl the URL as the nomad user)
3. Fix credentials: for git sources ensure SSH keys are in the client's ssh-agent or GIT_SSH_COMMAND is set; for s3/gcs verify provider config
4. Since Recoverable is true, simply retrying after fixing network/auth will usually resolve it; check client connectivity/DNS if transient

Example fix

// before (client without git access to private repo)
source = "git::ssh://git@private.example.com/repo.git"
// after (configure key on client, e.g. via client config or env)
// ssh-keyscan private.example.com >> ~nomad/.ssh/known_hosts
// load deploy key into nomad user's ssh-agent, then keep the same source

// pre-flight: check URL reachability from the client node
curl -sSIf -o /dev/null --max-time 10 https://example.com/artifact.tgz

// Recoverable=true: retry with backoff, log subprocess output
if e, ok := err.(*getter.Error); ok && e.Recoverable {
    backoff(e, e.Err) // includes getter subprocess log msg
}

When it happens

Trigger: The getter subprocess exits non-zero — unreachable host/DNS failure, HTTP 403/404, bad S3/GCS credentials, TLS certificate errors, unsupported URL scheme, checksum mismatch, or the source binary (git/ssh) missing on the client host

Common situations: Private repos without deploy keys on the Nomad client, expired cloud credentials, artifacts behind a proxy not configured on the node, S3 region/bucket typos, or a client missing git/curl-based getter dependencies

Related errors

• network namespace already exists but was misconfigured
• network already configured but not found in state
• no CNI network config found
• empty response from AWS metadata
• empty AZ Environment value

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/4cae13305a22048a. Report an issue: GitHub.