{"record":{"id":"242a45037b8f4243","repo":"apache/hadoop","slug":"invalid-specification-for-distributed-cache-artifa","errorCode":null,"errorMessage":"Invalid specification for distributed-cache artifacts of type {} : #uris={} #timestamps={} #visibilities={}","messagePattern":"Invalid specification for distributed-cache artifacts of type (.+?) : #uris=(.+?) #timestamps=(.+?) #visibilities=(.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/LocalResourceBuilder.java","lineNumber":96,"sourceCode":"    this.sizes = s;\n  }\n\n  void setVisibilities(boolean[] v) {\n    this.visibilities = v;\n  }\n\n  void setSharedCacheUploadPolicies(Map<String, Boolean> policies) {\n    this.sharedCacheUploadPolicies = policies;\n  }\n\n  void createLocalResources(Map<String, LocalResource> localResources)\n      throws IOException {\n\n    if (uris != null) {\n      // Sanity check\n      if ((uris.length != timestamps.length) || (uris.length != sizes.length) ||\n          (uris.length != visibilities.length)) {\n        throw new IllegalArgumentException(\"Invalid specification for \" +\n            \"distributed-cache artifacts of type \" + type + \" :\" +\n            \" #uris=\" + uris.length +\n            \" #timestamps=\" + timestamps.length +\n            \" #visibilities=\" + visibilities.length\n            );\n      }\n\n      for (int i = 0; i < uris.length; ++i) {\n        URI u = uris[i];\n        Path p = new Path(u);\n        FileSystem remoteFS = p.getFileSystem(conf);\n        String linkName = null;\n\n        if (p.getName().equals(DistributedCache.WILDCARD)) {\n          p = p.getParent();\n          linkName = p.getName() + Path.SEPARATOR + DistributedCache.WILDCARD;\n        }\n","sourceCodeStart":78,"sourceCodeEnd":114,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/LocalResourceBuilder.java#L78-L114","documentation":"When a job is submitted, LocalResourceBuilder.createLocalResources turns the distributed-cache artifacts of one type (files, archives, libjars) into LocalResources using four parallel arrays populated from job conf keys: uris, timestamps, sizes, visibilities. If the arrays have different lengths the artifacts cannot be paired 1:1, so it fails fast with IllegalArgumentException naming the type and each count. These keys are normally filled in atomically by the submission API; mismatch means someone wrote them by hand or partially overwrote them.","triggerScenarios":"Setting conf keys like mapreduce.job.cache.files (or .archives / .libjars and their .timestamps/.sizes/.visibilities companions) directly with unequal counts; hand-editing a serialized job conf; copying only some of the four keys between configurations.","commonSituations":"Custom submission frameworks (Spark/Cascading-style) that manipulate the raw cache conf keys instead of the Job API; reusing an old conf object where files were appended but metadata arrays were not regenerated; XML conf templates that define only the files key.","solutions":["Use the public API (job.addCacheFile / addArchiveToClassPath / DistributedCache) instead of raw conf keys — it regenerates all four arrays consistently","If raw keys are unavoidable, always write the four keys together with identical element counts (clearing stale ones first)","Rebuild the job conf from a clean Job.getInstance instead of mutating a previously submitted conf"],"exampleFix":"// before (manual, mismatch-prone)\nconf.setStrings(\"mapreduce.job.cache.files\", \"hdfs://nn/a.jar,hdfs://nn/b.jar\");\n// mapreduce.job.cache.files.timestamps etc. left from a previous 1-file config -> throws\n\n// after\nJob job = Job.getInstance(conf);\njob.addCacheFile(new URI(\"hdfs://nn/a.jar\"));\njob.addCacheFile(new URI(\"hdfs://nn/b.jar\")); // submitter fills timestamps/sizes/visibilities","handlingStrategy":"validation","validationCode":"String[] uris = conf.getStrings(\"mapreduce.job.cache.files\");\nlong[] ts = conf.getLongs(\"mapreduce.job.cache.files.timestamps\");\nlong[] sz = conf.getLongs(\"mapreduce.job.cache.files.sizes\");\nboolean[] vis = parseVisibilities(conf); // same companion key\nif (uris != null && (ts == null || sz == null || vis == null\n    || uris.length != ts.length || uris.length != sz.length\n    || uris.length != vis.length)) {\n  throw new IOException(\"Distributed cache arrays out of sync — regenerate via job.addCacheFile\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  job.submit();\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().contains(\"distributed-cache artifacts\")) {\n    // clear all four companion keys and re-add cache entries via the Job API\n  }\n  throw e;\n}","preventionTips":["Never hand-write the cache .files/.timestamps/.sizes/.visibilities keys — use job.addCacheFile/addArchiveToClassPath","Start from a fresh Job.getInstance(conf) per submission instead of reusing submitted confs","Add a conf lint that asserts equal lengths of the four cache arrays before submit"],"tags":["hadoop","mapreduce","distributed-cache","job-submission","configuration"],"backgroundTag":"distributed-cache-misconfiguration","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}