Optimizing with GEPA
One of dspy’s strengths is its streamlined path to optimize a program for a specific configured langauge model without you having to change any code.
Configuration
dspy allows you to configure two models: the “teacher” and the “student.”
arsgrammatica uses an .env file with the same values as the rest of arsgrammatica’s programs to configure the target (“student”) model: API_BASE, MODEL, and API_KEY.If you add a parallel trio REFLECTION_API_BASE, REFLECTION_MODEL, and REFLECTION_API_KEY to configure a more powerful reasoning model as the “teacher”, arsgrammatica will take that as the “teacher” model, and GEPA will reason with the “teacher” model while optimizing for the “student” mode. If you do not configure a second reflection model, GEPA will the same model for both “teacher” and “student.”
Running optimize_gepa
The script optimize_gepa.py runs one of dspy’s most advanced optimizers, GEPA (dspy.GEPA) to improve SyntaxAnalysis’s instructions against the gold examples in tests/fixtures/gold_examples.py. Since this requires making the specified request, and then a further request to a reflection model, this can potentially be expensive, so optimize_gepa lets you regulate this stage of work with these parameters:
python optimize_gepa.py # --auto light (cheapest; default)
python optimize_gepa.py --auto medium # more thorough, more expensive
python optimize_gepa.py --auto heavy # most thorough, most expensive
python optimize_gepa.py --max-metric-calls 40 # exact call budget instead of a preset
python optimize_gepa.py --skip-baseline # skip the pre-GEPA scoring pass (saves N calls)This optimizes SyntaxAnalysis by training on all the gold examples in GOLD_EXAMPLES (a small, hand-curated set built to exercise every documented construction at least once) with no separate held-out evaluation set. If you have files of analyses you have verified, you can add those to the pool optimize_gepa uses and specify the fraction to be used for an evaluation set.
The data directory of the arsgrammatica repository includes hand-vetted analyses you can use in optimizing the package for a new model.
# Include individual files with analyses:
python3 utilities/optimize_gepa.py --eval-file urn_cts_compnov_bible_genesis_vulgate_42_1.cex urn_cts_compnov_bible_genesis_vulgate_42_2.cex
# Include all files in a directory, and hold out 25% for evaluation set:
python3 utilities/optimize_gepa.py --eval-dir data/vulgate/ --val-fraction 0.25
# Include all files in a directory with no separate evaluation set:
python3 utilities/optimize_gepa.py --eval-dir data/vulgate/ --val-fraction 0 Scoring
arsgrammatica defines a syntax_metric. This compares the responses verbalunits and tokengraph against the gold answer and returns a score from 0 to 1. The score is computed weighing score for correct relations at 50%, score for classifying verbal-expressions at 30%, scoring for basic fields for each token fields at 20%. It includes specific, human-readable feedback naming every mismatched token/relation/classification, which GEPA’s reflection model can read and take advantage of.
Using the result
optimize_gepa.py saves the optimized program’s instructions to a JSON file named in the --out parameter or defaulting to optimized_syntax_analysis.json.
To use the saved model, load it right after importing analyze and before calling any analysis functions:
from arsgrammatica.latin_syntax_dspy import analyze
analyze.load("optimized_syntax_analysis.json")