Variations

Users do not phrase a request the way the spec author did. variations runs a test with paraphrases of its trigger, and the test passes only if every phrasing does:

trigger: "I want to cancel my order"
variations: 3
expect:
  function_calls:
    - name: cancel_order

The output shows each phrasing that failed, and the test's title says how many ran: I want to cancel my order (3 variations).

Generate once, commit, run offline

Paraphrases are generated by an LLM once and kept in .highbar/variations.json. Generating on every run would test different sentences each time, and CI would go red on a commit that changed nothing.

export HIGHBAR_LLM_API_KEY=...
highbar variations
git add .highbar/variations.json

• The generator calls any OpenAI-compatible API: HIGHBAR_LLM_BASE_URL (default https://api.openai.com/v1) and HIGHBAR_LLM_MODEL (default gpt-4o-mini).
• highbar variations generates only what is missing; --force regenerates.
• highbar finds the cache by walking up from the current directory, so running from a subdirectory uses the same file.
• highbar run generates a missing entry when a key is set. In CI, run with --frozen: a phrasing that is not in the cache then fails instead of being generated.

Writing them by hand

Without an API key, write the entry yourself. Each key is the first 16 hex characters of the SHA-256 of the trigger and the count:

{
  "1a40c26135e2d5ff": {
    "trigger": "I want to cancel my order",
    "count": 3,
    "model": "by hand",
    "variations": [
      "Please cancel my order",
      "Can you cancel the order I placed?",
      "I need to cancel an order"
    ]
  }
}

import hashlib
import json

trigger, count = "I want to cancel my order", 3
key = json.dumps({"trigger": trigger, "count": count}, sort_keys=True)
print(hashlib.sha256(key.encode()).hexdigest()[:16])  # 1a40c26135e2d5ff

With runs

Each phrasing runs runs times and must reach min_pass on its own. A test with variations: 3 and runs: 5 sends 20 messages: the trigger and three paraphrases, five times each. Variations do not apply to multi-turn tests.