juicedata/juicefs · error · DatasetIOException

Cannot load default config

Error message

Cannot load default config

What it means

In load(), KiteDataLoader calls FileSystem.getLocal(DefaultConfiguration.get()) purely to force eager loading of the default Hadoop configuration (including hdfs-site.xml). If that throws IOException, it wraps it in DatasetIOException with this message. The getLocal call is a config-validation probe, so failure means the Hadoop configuration could not be loaded at all.

Source

Thrown at sdk/java/src/main/java/io/juicefs/KiteDataLoader.java:60

      try {
        fs = FileSystem.get(fileSystemURI(match), conf);
      } catch (IOException e) {
        throw new DatasetIOException("Could not get a FileSystem", e);
      }
      return new FileSystemDatasetRepository.Builder()
              .configuration(new Configuration(conf)) // make a modifiable copy
              .rootDirectory(fs.makeQualified(root))
              .build();
    }
  }

  @Override
  public void load() {
    try {
      // load hdfs-site.xml by loading HdfsConfiguration
      FileSystem.getLocal(DefaultConfiguration.get());
    } catch (IOException e) {
      throw new DatasetIOException("Cannot load default config", e);
    }

    OptionBuilder<DatasetRepository> builder = new URIBuilder();
    Registration.register(
            new URIPattern("jfs:/*path"),
            new URIPattern("jfs:/*path/:namespace/:dataset"),
            builder);
  }

  private static URI fileSystemURI(Map<String, String> match) {
    try {
      return new URI(match.get(URIPattern.SCHEME), null,
              match.get(URIPattern.HOST), -1, "/", null, null);
    } catch (URISyntaxException ex) {
      throw new DatasetOperationException("[BUG] Could not build FS URI", ex);
    }
  }
}

View on GitHub (pinned to c9a67b23e8)

Solutions

1. Ensure HADOOP_CONF_DIR (and HADOOP_HOME) point to a directory containing valid core-site.xml/hdfs-site.xml in the runtime environment.
2. Inspect the wrapped IOException cause for the exact file that failed to load and repair its syntax/permissions.
3. If embedding programmatically, pre-load Configuration.addDefaultResource or set config paths before calling load().
4. Confirm the hadoop-common jar is on the classpath so FileSystem.getLocal can instantiate LocalFileSystem.

Example fix

// before: app launched with no conf -> DatasetIOException: Cannot load default config
// after
export HADOOP_CONF_DIR=/etc/hadoop/conf
spark-submit --driver-class-path $HADOOP_CONF_DIR ...

String confDir = System.getenv("HADOOP_CONF_DIR");
if (confDir == null || !new java.io.File(confDir, "core-site.xml").canRead()) {
  throw new IllegalStateException("HADOOP_CONF_DIR missing or unreadable: " + confDir);
}

try {
  loader.load();
} catch (DatasetIOException e) {
  throw new IllegalStateException("Hadoop default config failed to load: " + e.getCause().getMessage(), e);
}

When it happens

Trigger: Calling KiteDataLoader.load() when HADOOP_CONF_DIR/HADOOP_HOME point to missing or unreadable config directories, or when loading hdfs-site.xml/core-site.xml throws an IOException (parse/IO error), making DefaultConfiguration.get() or FileSystem.getLocal fail.

Common situations: Deploying the JuiceFS/Kite loader in a container or Spark job without the Hadoop conf directory mounted; malformed XML in core-site.xml/hdfs-site.xml; HADOOP_CONF_DIR unset so defaults are absent.

Related errors

• Could not get a FileSystem
• name is required
• Invalid start or len parameter
• stream was closed
• arguments: " + off + " " + len

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/6d6cbee95bddad8c. Report an issue: GitHub.