Docs

LvW-bench runs fully on your machine. Pick a model, it downloads what it needs from Hugging Face, starts a local server, scores three tracks and saves a result file tied to the exact weights.

Install

Terminal: macOS and Linux (Homebrew)

brew trust gitguy268/tap
brew install gitguy268/tap/lvw-bench     # the lvwb command
brew install llama.cpp uv                 # engines: GGUF and safetensors

Desktop app

brew install --cask gitguy268/tap/lvw-bench-app   # macOS

For Linux (.deb, tar.gz) and Windows (installer, zip), download the files from the releases page. The app uses the same engines, so install llama.cpp and uv as well.

From source

git clone https://github.com/Gitguy268/LvW-bench && cd LvW-bench
go build -o lvwb ./cmd/lvwb

Run

lvwb run Lucasgitguy-co/spot-snake-d1-2B-v0.1          # HF id: picks the GGUF, Q4_K_M by default
lvwb run https://huggingface.co/unsloth/Qwen3-0.6B-GGUF  # HF link
lvwb run ~/models/model-Q4_K_M.gguf                      # local GGUF
lvwb run ~/models/my-model/                              # local safetensors folder
lvwb run Lucasgitguy-co/minimind-mermaid-run2 --subfolder ep3
lvwb run http://127.0.0.1:8080/v1 --backend openai --server-model my-model
lvwb run llama3.2:1b --backend ollama
lvwb tui                                                 # terminal UI with presets

Flag	What it does
`--track decide,answer,reason`	Which tracks to run (default: all that the model supports)
`--profile quick\|standard\|full`	`quick` is a smoke test, `standard` is what the leaderboard uses, `full` runs everything
`--quant Q8_0`, `--file x.gguf`	Pick a GGUF in a repo that has several
`--format safetensors`	Use the safetensors weights when a repo has both
`--subfolder`, `--revision`	Weights in a subfolder, or a branch, tag, commit or `refs/pr/N`
`--trust-remote-code`	Allow custom model code from the repo (safetensors only)
`--seed N`	A fresh generated set that no model can have memorised
`--reason-budget 2048`	Output tokens for the reason track
`--mode specialist`	Mark a model that was trained on the `train` split or the same task
`--cpu`, `--parallel 4`	No GPU offload, requests in flight

Results go to the lvw-bench cache folder (--out to change it). lvwb results, lvwb leaderboard, lvwb compare a.json b.json, lvwb export --md|--csv|--eval-results and lvwb rescore work on them.

Tracks

System 1 · decide

Five workflows with typed questions: choice, score and noul. A decision model answers through /v1/systemone; a text model answers through the probabilities of its first token. Scored on accuracy, KL from gold, Brier and ECE, like typed-decisions.

System 2 · answer

Thinking off. Short answers get 32 tokens, and free-form ones (IFEval, JSON, Mermaid) get 768. Uses GSM8K, MMLU, MMLU-Pro, TruthfulQA, IFEval and a generated set.

System 2 · reason

Thinking on: native <think> when the template has it, step by step otherwise. 2,048-token budget. Only the final answer is scored, and a think block that never closes counts as wrong.

Engines

Input	Engine	Needs
GGUF (HF or local)	`llama-server` on a loopback port, GPU offload on	llama.cpp 0.9.0 or later (decision models need `/v1/systemone`)
safetensors (HF or local), PEFT adapters	A small Python helper on transformers, run with `uv` on Python 3.12	uv; the first run installs PyTorch in a cached environment
Server URL, Ollama	Your server, OpenAI-compatible	Logprobs for System 1, or a sampling fallback

Scoring notes

Why these scores differ from lm-eval

lm-eval's multiple-choice tasks compare the probabilities of the options and never let the model write. LvW-bench lets the model answer and then extracts the answer: the Answer: line, the leading letter, or the last number. That is closer to how a small model gets used, but the numbers are not comparable with acc_norm from lm-eval or TPN.

How the System 1 gold is made

Each case is generated from sampled facts, and its gold comes from fixed rules. A fact the state shows directly gets a sharp gold. A judgement call keeps a soft gold. No teacher model is involved. Train and test come from different seeds and never share a state.

Reproducing a score

Every result has a .predictions.jsonl next to it. lvwb rescore or python bench/score.py scores it again without the model. Go, Python and the browser share one set of golden tests.