stablyai/orca · error · Error

--higher-is-better requires a metric key

Error message

--higher-is-better requires a metric key

What it means

Thrown by parseBenchmarkComparisonArgs when the --higher-is-better flag is followed by zero non-flag arguments. The flag greedily consumes subsequent args until it hits another --flag or end of args; if nothing follows, the metric-key set is empty and the flag had no effect, which the parser rejects rather than silently defaulting to all-lower-is-better.

Source

Thrown at config/scripts/compare-benchmark-artifacts.mjs:48

      continue
    }
    if (flag === '--output') {
      parsed.outputPath = readRequiredValue(args, ++index, '--output requires a path')
      continue
    }
    if (flag === '--json-output') {
      parsed.jsonOutputPath = readRequiredValue(args, ++index, '--json-output requires a path')
      continue
    }
    if (flag === '--higher-is-better') {
      let consumed = 0
      while (args[index + 1] != null && !args[index + 1].startsWith('--')) {
        parsed.higherIsBetter.add(args[index + 1])
        index += 1
        consumed += 1
      }
      if (consumed === 0) {
        throw new Error('--higher-is-better requires a metric key')
      }
      continue
    }
    throw new Error(USAGE)
  }

  if (!parsed.baselinePath) {
    throw new Error(USAGE)
  }
  if (!parsed.candidatePath) {
    throw new Error(USAGE)
  }
  return parsed
}

function readRequiredValue(args, index, message) {
  const value = args[index]
  if (value == null || value.startsWith('--')) {

View on GitHub (pinned to 1136503c6a)

Solutions

1. Pass at least one metric key after the flag: --higher-is-better opsPerSec throughput.
2. If the metric key legitimately starts with '--', restructure (this parser treats any next arg starting with '--' as a new flag).
3. Remove the --higher-is-better flag entirely if all metrics are lower-is-better (the default).
4. In CI, guard the workflow so the flag is only emitted when the metric-key variable is non-empty.

Example fix

// before
// node config/scripts/compare-benchmark-artifacts.mjs --baseline a.json --candidate b.json --higher-is-better
// -> '--higher-is-better requires a metric key'

// after
// node config/scripts/compare-benchmark-artifacts.mjs --baseline a.json --candidate b.json --higher-is-better throughput

function parseHigherIsBetter(args, index) {
  const keys = []
  while (args[index + 1] != null && !args[index + 1].startsWith('--')) {
    keys.push(args[index + 1])
    index += 1
  }
  if (keys.length === 0) {
    throw new Error('--higher-is-better requires a metric key')
  }
  return { keys, nextIndex: index }
}
// use from parseBenchmarkComparisonArgs

When it happens

Trigger: Invoking compare-benchmark-artifacts.mjs --baseline a.json --candidate b.json --higher-is-better (no key after the flag); the key was itself a flag-like token starting with '--'; the flag was placed at the very end of the command line.

Common situations: Operator forgets the metric key; CI workflow templated the flag with an empty value; the metric key was meant to be passed via a variable that expanded to empty.

Related errors

• Usage: node config/scripts/compare-benchmark-artifacts.mjs -
• Missing value for ${arg}
• Unknown argument: ${arg}
• --boundary must be child or worker
• --trials must be a positive integer

AI-assisted analysis of stablyai/orca@1136503c6a (2026-08-12). Data as JSON: /api/errors/3c9ffa054bbed6f9. Report an issue: GitHub.