hashicorp/nomad · error

task memory requirements exceed driver hard limit of %d MB

Error message

task memory requirements exceed driver hard limit of %d MB

What it means

When the operator sets a driver-level memory_hard_limit in the docker plugin config, the task's requested memory (reservation) must not exceed that limit. If the computed reserved memory is greater than the hard limit (and limit > 0), container creation is rejected so the kernel OOM-killer does not immediately kill the container.

Source

Thrown at drivers/docker/driver.go:1040

	imageID string) (createContainerOptions, error) {

	logger := d.logger.With("task_name", task.Name)
	c := createContainerOptions{}

	// ensure that PortMap variables are populated early on
	task.Env = taskenv.SetPortMapEnvs(task.Env, driverConfig.PortMap)

	if task.Resources == nil {
		// Guard against missing resources. We should never have been able to
		// schedule a job without specifying this.
		logger.Error("task.Resources is empty")
		return c, fmt.Errorf("task.Resources is empty")
	}

	memory, memoryReservation := memoryLimits(driverConfig.MemoryHardLimit,
		task.Resources.NomadResources.Memory)
	if memory > 0 && memoryReservation > memory {
		return c, fmt.Errorf("task memory requirements exceed driver hard limit of %d MB",
			driverConfig.MemoryHardLimit)
	}

	binds, err := d.containerBinds(task, driverConfig)
	if err != nil {
		return c, err
	}
	logger.Trace("binding volumes", "volumes", binds)

	// create the config block that will later be consumed by go-dockerclient
	config := &containerapi.Config{
		Image:      imageID,
		Entrypoint: driverConfig.Entrypoint,
		Hostname:   driverConfig.Hostname,
		User:       task.User,
		Tty:        driverConfig.TTY,
		OpenStdin:  driverConfig.Interactive,
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Raise memory_hard_limit in the Nomad client docker plugin config to at least the task's requested memory.
2. Lower the job's resources.memory (or memory_hard task field) to fit under the plugin limit.
3. Target the task at clients where the plugin config permits the requested memory (use a constraint or client block).

Example fix

// job
// before
resources { memory = 512 }
// plugin has memory_hard_limit = 256
// after
resources { memory = 256 }  // or raise plugin memory_hard_limit to 512

function validateMemoryWithinHardLimit(jobMemoryMB, memoryHardLimitMB) {
  if (memoryHardLimitMB > 0 && jobMemoryMB > memoryHardLimitMB) {
    throw new Error(`job memory ${jobMemoryMB}MB exceeds driver hard limit ${memoryHardLimitMB}MB`);
  }
}

try {
  await client.jobs.submit(job);
} catch (err) {
  if (/exceed driver hard limit/.test(err.message)) {
    console.error('Reduce resources.memory or raise plugin memory_hard_limit');
  }
  throw err;
}

When it happens

Trigger: Job requests memory greater than the plugin's memory_hard_limit: e.g. plugin configured memory_hard_limit = 256 while the job's resources.memory = 512, or memory set via memory_limits/Hard MB mismatch.

Common situations: Cluster operators tightening memory_hard_limit on existing clients without updating job specs; a job spec copied from another cluster with a higher hard limit; MB vs MiB assumptions making the requested value larger than expected.

Related errors

• failed to parse 'image_delay' duration: %v
• failed to parse 'period' duration: %v
• failed to parse 'creation_grace' duration: %v
• creation_grace is less than minimum, %v
• failed to parse 'pull_activity_timeout' duration: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1302ec47c48825d1. Report an issue: GitHub.