apache/hadoop · error · IllegalArgumentException

Resource type PATTERN is not implemented yet. {}

Error message

Resource type PATTERN is not implemented yet. {}

What it means

LocalDistributedCacheManager emulates YARN distributed-cache localization when a job runs through LocalJobRunner (mapreduce.framework.name=local). During setup it classifies each LocalResource as FILE or ARCHIVE; LocalResourceType.PATTERN has no local implementation, so it throws IllegalArgumentException and the local job fails during setup.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java:153

        try {
          path = resourcesToPaths.get(resource).get();
        } catch (InterruptedException e) {
          throw new IOException(e);
        } catch (ExecutionException e) {
          throw new IOException(e);
        }
        String pathString = path.toUri().toString();
        String link = entry.getKey();
        String target = new File(path.toUri()).getPath();
        symlink(workDir, target, link);
        
        if (resource.getType() == LocalResourceType.ARCHIVE) {
          localArchives.add(pathString);
        } else if (resource.getType() == LocalResourceType.FILE) {
          localFiles.add(pathString);
        } else if (resource.getType() == LocalResourceType.PATTERN) {
          //PATTERN is not currently used in local mode
          throw new IllegalArgumentException("Resource type PATTERN is not " +
          		"implemented yet. " + resource.getResource());
        }
        Path resourcePath;
        try {
          resourcePath = resource.getResource().toPath();
        } catch (URISyntaxException e) {
          throw new IOException(e);
        }
        LOG.info(String.format("Localized %s as %s", resourcePath, path));
        String cp = resourcePath.toUri().getPath();
        if (classpaths.keySet().contains(cp)) {
          localClasspaths.add(path.toUri().getPath().toString());
        }
      }
    } finally {
      if (exec != null) {
        exec.shutdown();
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Replace the PATTERN resource with LocalResourceType.ARCHIVE (or FILE) for local runs
2. Run the test on MiniMRYarnCluster instead of the pure local runner
3. Filter PATTERN resources out before submitting locally rather than letting setup fail

Example fix

// before
LocalResource r = LocalResource.newInstance(url,
    LocalResourceType.PATTERN, LocalResourceVisibility.APPLICATION,
    10L, timestamp, "classes/.*.jar");

// after: local runner supports FILE and ARCHIVE only
LocalResource r = LocalResource.newInstance(url,
    LocalResourceType.ARCHIVE, LocalResourceVisibility.APPLICATION,
    10L, timestamp);

// Before running locally, scan the resource map for unsupported types
boolean hasPattern = resources.values().stream()
    .anyMatch(r -> r.getType() == LocalResourceType.PATTERN);
if (hasPattern && "local".equals(conf.get("mapreduce.framework.name"))) {
  throw new IllegalStateException(
      "LocalJobRunner does not support LocalResourceType.PATTERN; "
    + "convert to ARCHIVE/FILE or run on MiniMRYarnCluster");
}

static boolean isLocalRunnerCompatible(Map<String, LocalResource> rs) {
  return rs.values().stream().noneMatch(
      r -> r.getType() == LocalResourceType.PATTERN);
}

try {
  job.submit();
} catch (Exception e) {
  if (e.getCause() instanceof IllegalArgumentException iae
      && iae.getMessage().contains("PATTERN")) {
    // strip or convert PATTERN resources, then rerun locally or on a cluster
  }
  throw e;
}

When it happens

Trigger: A job whose resource map contains a LocalResource with LocalResourceType.PATTERN is run locally - typically code that builds LocalResources for a YARN deployment and is then reused in local unit tests or embedded local execution.

Common situations: Framework or application code that sets PATTERN for pattern-based classpath archives; tests switching between LocalJobRunner and MiniMRYarnCluster; running a YARN-style MR pipeline in local mode without adapting resources.

Related errors

• Error parsing local resource path. Path was not able to be c
• Error generating encrypted spill key
• Configured mapreduce.local.map.tasks.maximum must be >= 1
• Configured mapreduce.local.reduce.tasks.maximum must be >= 1
• Changing job priority in LocalJobRunner is not supported.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c9f45825b55fef20. Report an issue: GitHub.