Comparing models with `model_bakeoff.py`

arsgrammatica was developed with Claude Opus 5 hosted at the College of the Holy Cross. dspy lets us plug in other models and use the same code with them. What open-weight models hosted on Hugging Face, or running locally under Ollama could realistically run arsgrammatica?

model_bakeoff.py lets you compare multiple models against the same held-out slice of gold examples, and see not just whether each one is “good enough” but where it falls short and whether that performance can be improved. You can run model_bakeoff.py whenever you want to evaluate a candidate; it never affects the production pipeline.

Three stages of evaluation

“Can a given model run arsgrammatica?” is really three separate questions:

How good is the model with the instructions already written and tuned against Opus 5, without optimization?
How much does the performance improve if we let dspy.GEPA optimize instructions for the model?
How much does the performance improve if we use Opus 5 as a “teacher” model and modify the candidate model’s instructions?

You can answer each of those questions with the model_bakeoff.py script, selecting from these values for the --stages:

baseline : zero-shot testing answering question 1, how good is the model without optimization? The cheapest stage by far (one call per held-out example) and worth running alone first.
gepa — uses dspy.GEPA is allowed to rewrite the instructions for this candidate (question 2). The candidate still solves every example itself, both during optimization and at final scoring; only the wording changes.
bootstrap uses Opus as a “teacher” model, and attaches verified correct traces to the model’s instructions as few-shot examples. Here the candidate never has to solve a training example itself — only the held-out scoring examples afterward.

Comparing baseline against gepa and bootstrap distinguishes “Does this model already perform adequately?” from “How much can it be improved?”. Comparing gepa against bootstrap separates two different ways of improving the model (with or without a separate “teacher” model). A candidate that is mediocre at baseline but closes most of the gap at gepa or bootstrap might be a good prospect for optimization. A mediocre performance on all three stages probably reflects a real ceiling on the model’s capabilities.

Managing cost

gepa and bootstrap make many more LM calls than baseline (since they involve calls both to the candidate and the “teacher” model). so start with --stages baseline across every candidate you’re curious about, then spend the heavier stages only on the ones worth the API usage.

Providers

--provider huggingface (the default) routes each candidate through Hugging Face’s Inference Providers layer. You can run several candidates in one invocation, one after another, since each is just an API call. Be sure you have included a Hugging Face access token in your .env file.

HUGGINGFACE_API_KEY=hf_...

--provider ollama is for models you pull and run locally with ollama. This needs no special configuration since it talks to ollama’s default port of http://localhost:11434. (Set OLLAMA_API_BASE in .env, or pass --ollama-api-base, if Ollama is running somewhere else on your network.)

Realistically you can only have one model loaded in Ollama at a time, so this script’s shape is built around that constraint: --provider ollama refuses to run unless exactly one candidate is selected. The resulting workflow will look like this:

ollama pull llama3.1:8b            # or whichever candidate; make sure ollama serve is running
python model_bakeoff.py --provider ollama --candidates llama-3.1-8b --stages baseline
# ...stop that model, pull/load the next one...
python model_bakeoff.py --provider ollama --candidates gpt-oss-20b --stages baseline

Every invocation is a normal one-shot run, not a long-lived process . Scores are collected in a cumulative results file (see below), so you can score one model, save the results, swap models, and repeat, without overwriting earlier results.

The held-out evaluation set

optimize_gepa.py trains against every example in tests/fixtures/gold_examples.py, with no separate held-out set — a reasonable choice for tuning one model, but it would make cross-model comparison unreliable here: a candidate’s post-optimization score would partly reflect how well its own optimized prompt/demos fit the exact examples it’s judged on.

model_bakeoff.py instead carves out a fixed slice of ten gold examples (HELD_OUT_SLUGS, near the top of the file) that no candidate’s gepa/bootstrap stage ever trains against. Every candidate optimizes against the same remaining examples and is scored against the same untouched held-out set, so scores are actually comparable across models and across stages. The slice is deliberately stratified — a plain independent clause, a subordinating-conjunction and a relative-pronoun dependent clause, a coordinated-verb pair, an indirect statement, a circumstantial participle, a depth-2 nesting case, and the three newest relations (apposition, indirect question, complementary infinitive) — so a low score can be traced to a specific construction rather than just “worse overall.”

Reading the scores

Every stage reports a blended mean score in [0, 1] over the held-out set, plus three unblended sub-scores from arsgrammatica.gepa_metric.syntax_metric():

field_mean — basic per-token fields (tokentype, lemma, which verbal unit a token belongs to).
relation_mean — the actual dependency relations, weighted highest (0.5 of the blend) since they’re the heart of the scheme.
vu_mean — verbal-expression classification (syntactic_type/semantic_type).

A model that scores well on field_mean but collapses on relation_mean is failing at multi-hop structural reasoning specifically — a different problem (and probably less susceptible to fixing with prompts) than a model that is just generally worse across the board. The console output also prints, per held-out example, any that raised an outright error (a request that failed, or output that didn’t parse into the expected shape) rather than silently folding those into the average — worth reading before trusting a low score.

This script doesn’t compute a malformed-output rate as its own number (validate()/find_unanchored_coordinated_verbs() in the main package can do that over raw predictions if you want it), and it doesn’t do real weight-level fine-tuning — the bootstrap stage attaches Opus-solved examples to the candidate’s prompt, never touching its weights. dspy.BootstrapFinetune is real fine-tuning, but needs different infrastructure than either provider here and isn’t wired in.

Running it

# Cheap first pass across every Hugging Face candidate
python model_bakeoff.py --stages baseline

# Everything, every candidate (expensive -- gepa + bootstrap make many calls)
python model_bakeoff.py --stages baseline gepa bootstrap

# Just a couple of candidates
python model_bakeoff.py --candidates llama-3.1-8b gpt-oss-20b

# Skip gepa/bootstrap for anything that didn't clear a baseline threshold
python model_bakeoff.py --min-baseline-to-optimize 0.3

# Ollama, one candidate at a time (see workflow above)
python model_bakeoff.py --provider ollama --candidates llama-3.1-8b --stages baseline
python model_bakeoff.py --provider ollama --candidates llama-3.1-8b --stages gepa bootstrap

# A model not in CANDIDATES at all -- either provider
python model_bakeoff.py --provider ollama --model ollama_chat/llama3.1:8b-instruct-q8_0 --label llama-3.1-8b-q8

# Optimizer knobs
python model_bakeoff.py --auto medium                    # gepa's (and miprov2's) budget preset
python model_bakeoff.py --max-metric-calls 40             # exact gepa budget instead of --auto
python model_bakeoff.py --bootstrap-optimizer miprov2     # heavier alternative to bootstrap-fewshot
python model_bakeoff.py --max-bootstrapped-demos 4 --max-labeled-demos 4

python model_bakeoff.py --out results.csv                 # merge into a specific file

The full flag reference is always available with python model_bakeoff.py --help; the ones most worth knowing up front:

--provider {huggingface,ollama} — where candidates run (default huggingface).
--candidates LABEL [LABEL ...] — restrict to specific labels from CANDIDATES (default: all of them; --provider ollama requires exactly one).
--model / --label — an ad hoc pair for a model not in CANDIDATES at all; both or neither.
--stages {baseline,gepa,bootstrap} — which stage(s) to run this invocation (default: all three).
--bootstrap-optimizer {bootstrap-fewshot,miprov2} — which optimizer implements the bootstrap stage (default bootstrap-fewshot, cheaper).
--min-baseline-to-optimize — skip gepa/bootstrap for a candidate whose zero-shot mean is below this threshold, whether that baseline came from this invocation or a previously recorded row.
--out — the results file to merge into (default model_bakeoff_results.csv).

The results file

Results accumulate in a CSV file (model_bakeoff_results.csv by default), keyed by (label, provider, stage). Each invocation reads the contents, merges in new rows, and writes the whole thing back out, sorted by CANDIDATES order and then by stage. You could, for example, run a Hugging Face pass today and an Ollama pass tomorrow, or score one Ollama model per day as you cycle through them, and the results all accumulate in the same file. If you re-run the same combination of label, provider and stage, that overwrites only that single row; other rows are untouched.

Columns: label, provider, model (the resolved model string actually called), family, tier, stage, n (held-out example count), mean/min/max (blended score), field_mean/relation_mean/vu_mean, elapsed_s, n_calls, total_cost (blank when the provider doesn’t report per-call cost — Ollama never will, since it’s local), and error (set instead of scores if the candidate was skipped entirely, e.g. a missing API key).

Each optimized gepa/bootstrap run is also saved to disk as optimized_<label>_<provider>_<stage>.json, alongside the CSV row — load one of these into a dspy.ChainOfThought(SyntaxAnalysis) instance with .load(...) if you want to inspect or reuse the actual optimized program, not just its score.

The candidate roster

CANDIDATES, near the top of the file, spans roughly 3B to 120B+ parameters across several major open-weight families, plus one reasoning-distilled model (DeepSeek-R1-Distill-Llama-8B) at the same size as its plain counterpart (Llama-3.1-8B), to see whether chain-of-thought distillation specifically helps a multi-hop structural task like this one.

label	family	tier
`phi-4-mini`	Microsoft Phi	~4B
`llama-3.2-3b`	Meta Llama	~3B
`llama-3.1-8b`	Meta Llama	~8B
`deepseek-r1-distill-llama-8b`	DeepSeek (R1 distill)	~8B
`qwen-8b`	Alibaba Qwen	~7B
`gpt-oss-20b`	OpenAI (open weights)	~20B
`mistral-small-24b`	Mistral AI	~24B
`llama-3.3-70b`	Meta Llama	~70B
`deepseek-r1-distill-llama-70b`	DeepSeek (R1 distill)	~70B
`gpt-oss-120b`	OpenAI (open weights)	~120B

Model availability and naming move fast on both sides. This roster is just a snapshot, so recheck against Hugging Face’s Inference Providers docs, each model’s Hub page, and ollama list before running.

To add a new candidate permanently, add a dict(...) entry to CANDIDATES with at least label, model (the huggingface/org/repo string), and ollama_model (the exact Ollama pull tag); for a one-off you don’t want to add to the roster at all, use --model/--label instead (see the Running it examples above).