apple/pkl · error · VmException

cannotEvaluateNonFileBasedTestModule

cannotEvaluateNonFileBasedTestModule

Error message

cannotEvaluateNonFileBasedTestModule

What it means

Pkl's `examples` test block can only be evaluated for modules loaded from the local filesystem, because the test runner must read and write sibling `*-expected.pcf` / `*-actual.pcf` files next to the module file. When `TestRunner.runExamples` finds the module URI's scheme is not `file` (e.g. a module from a JAR, HTTP, or an in-memory/synthetic module), it throws this error instead of attempting file I/O on a non-file path.

Source

Thrown at pkl-core/src/main/java/org/pkl/core/runtime/TestRunner.java:145

                          err.getMessage(), err.toPklException(stackFrameTransformer, useColor));
                  resultBuilder.addError(error);
                }
                return true;
              });
          testResults.add(resultBuilder.build());
          return true;
        });
    return new TestSectionResults(TestSectionName.FACTS, Collections.unmodifiableList(testResults));
  }

  private TestSectionResults runExamples(VmTyped testModule, ModuleInfo info) {
    var examples = VmUtils.readMember(testModule, Identifier.EXAMPLES);
    if (examples instanceof VmNull)
      return new TestSectionResults(TestSectionName.EXAMPLES, List.of());

    var moduleUri = info.getModuleKey().getUri();
    if (!moduleUri.getScheme().equalsIgnoreCase("file")) {
      throw new VmExceptionBuilder()
          .evalError("cannotEvaluateNonFileBasedTestModule", moduleUri)
          .build();
    }

    var examplesMapping = (VmMapping) examples;
    var moduleFile = Path.of(moduleUri);
    var expectedOutputFile = moduleFile.resolveSibling(moduleFile.getFileName() + "-expected.pcf");
    var actualOutputFile = moduleFile.resolveSibling(moduleFile.getFileName() + "-actual.pcf");

    try {
      Files.deleteIfExists(actualOutputFile);
    } catch (IOException e) {
      throw new VmExceptionBuilder()
          .evalError("ioErrorWritingTestOutputFile", actualOutputFile)
          .withCause(e)
          .build();
    }
    try {

View on GitHub (pinned to f3efcbfc9b)

Solutions

1. Materialize the module to a real file on disk and run the test against that file path.
2. If the module comes from a remote source, download it into the project directory and reference it by relative path.
3. If using a custom ModuleResolver/synthetic ModuleKey, switch the test module to a FileModuleKey pointing at a temporary file.
4. Extract the `examples` block and validate it manually with `pkl eval` if filesystem-backed testing is not possible.

Example fix

// before: pkl test https://example.com/schema/MySchema.pkl
// after
curl -o MySchema.pkl https://example.com/schema/MySchema.pkl
pkl test MySchema.pkl

// check module scheme before running tests
URI uri = moduleKey.getUri();
if (!"file".equalsIgnoreCase(uri.getScheme())) {
    throw new IllegalArgumentException("pkl test requires a file-based module, got: " + uri);
}

boolean isFileBasedModule(ModuleKey key) { return "file".equalsIgnoreCase(key.getUri().getScheme()); }

When it happens

Trigger: Running `pkl test` (or TestRunner.run) against a module whose ModuleKey URI scheme is not `file` — e.g. a module resolved from an HTTP(S) URL, a classpath/JAR resource, or a synthetic in-memory module.

Common situations: Testing modules fetched from a remote registry or URL; testing modules embedded in an application via custom module resolvers; CI setups that load Pkl code over the network; running tests on synthetic modules built programmatically with the Embedding API.

Understand the failure class

Background: "Invalid URL" / "URL cannot be empty": fix the malformed or missing URL behind request-construction failures — this error's family across 50 libraries.

Related errors

• Is a directory
• ioErrorWritingTestOutputFile
• invalidOutputFileStructure
• ioErrorReadingTestOutputFile
• ioErrorLoadingModule

AI-assisted analysis of apple/pkl@f3efcbfc9b (2026-09-08). Data as JSON: /api/errors/b090fe8318068a53. Report an issue: GitHub.