apache/hadoop · error · YarnRuntimeException

ShuffleProvider Service: {} was NOT found in the list of aux

Error message

ShuffleProvider Service: {} was NOT found in the list of aux-services that are available in this NM. You may need to specify this ShuffleProvider as an aux-service in your yarn-site.xml

What it means

addExternalShuffleProviders throws YarnRuntimeException at container-launch setup when a name listed in mapreduce.job.shuffle.provider.services is not present in yarn.nodemanager.aux-services as known to the AM. Before launching task containers, the AM writes serviceData for each configured external shuffle provider; a provider missing from the NM's aux-service list cannot receive it, so the job is failed rather than silently shuffling wrong.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java:1105

          conf.getStringCollection(YarnConfiguration.NM_AUX_SERVICES);

      for (final String shuffleProvider : shuffleProviders) {
        if (shuffleProvider
            .equals(ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID)) {
          continue; // skip built-in shuffle-provider that was already inserted
                    // with shuffle secret key
        }
        if (auxNames.contains(shuffleProvider)) {
          LOG.info("Adding ShuffleProvider Service: " + shuffleProvider
              + " to serviceData");
          // This only serves for INIT_APP notifications
          // The shuffle service needs to be able to work with the host:port
          // information provided by the AM
          // (i.e. shuffle services which require custom location / other
          // configuration are not supported)
          serviceData.put(shuffleProvider, ByteBuffer.allocate(0));
        } else {
          throw new YarnRuntimeException("ShuffleProvider Service: "
              + shuffleProvider
              + " was NOT found in the list of aux-services that are "
              + "available in this NM. You may need to specify this "
              + "ShuffleProvider as an aux-service in your yarn-site.xml");
        }
      }
    }
  }

  static ContainerLaunchContext createContainerLaunchContext(
      Map<ApplicationAccessType, String> applicationACLs,
      Configuration conf, Token<JobTokenIdentifier> jobToken, Task remoteTask,
      final org.apache.hadoop.mapred.JobID oldJobId,
      WrappedJvmID jvmID,
      TaskAttemptListener taskAttemptListener,
      Credentials credentials) {

    synchronized (commonContainerSpecLock) {

View on GitHub (pinned to 2add963021)

Solutions

Add the provider to yarn.nodemanager.aux-services (plus its yarn.nodemanager.aux-services.<name>.class) in the NM yarn-site.xml on every NodeManager and restart them
Verify spelling matches exactly between mapreduce.job.shuffle.provider.services and the aux-services entry
Until rollout completes, remove the provider from the job's shuffle-provider list

Example fix

<!-- before: job asks for a provider NMs don't run -->
<property><name>mapreduce.job.shuffle.provider.services</name><value>mapred_shufflex</value></property>

<!-- after: NM yarn-site.xml on all nodes, then restart NM -->
<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle,mapred_shufflex</value></property>
<property><name>yarn.nodemanager.aux-services.mapred_shufflex.class</name><value>com.example.ShufflexService</value></property>

Defensive patterns

Strategy: validation

Validate before calling

// preflight: every configured shuffle provider must be an NM aux-service
Collection<String> providers = conf.getStringCollection("mapreduce.job.shuffle.provider.services");
providers.remove("mapreduce_shuffle"); // built-in, always present
Collection<String> aux = conf.getStringCollection("yarn.nodemanager.aux-services");
if (!aux.containsAll(providers)) {
  throw new IllegalArgumentException("Missing aux-services: " +
      Sets.difference(new HashSet<>(providers), new HashSet<>(aux)));
}

Prevention

Roll out new aux-services to all NMs (and restart them) before jobs reference them
Keep the job's mapreduce.job.shuffle.provider.services list spelled exactly like the aux-services entries
After NM config changes, verify with 'yarn node -status' that the aux-service shows up before submitting

When it happens

Trigger: Job sets mapreduce.job.shuffle.provider.services=mapred_shufflex but NMs only run mapreduce_shuffle; the AM's view of yarn.nodemanager.aux-services (mirrored from cluster config) lacks the provider; only some NMs were restarted after adding the service.

Common situations: Rolling out a new aux-service (e.g., an SSD/offheap shuffle plugin) where yarn-site.xml was updated on the job client but not on NMs; typo between the job config name and the aux-services entry; heterogeneous cluster after partial restart.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/16fa167afe9b34d8. Report an issue: GitHub.