Improving arsgrammatica by reading Latin
You can incorporate the results of reading real Latin texts with arsgrammatica into a repeatable development loop that constantly improves the package.
Failed analyses
When arsgrammatica fails to analyze the syntax of a passage correctly, we can use the results to identify gaps in the schema, or areas in need of better training.
First check whether syntax_model.md explicitly and clearly defines the construction that was not analyzed correctly. If it does not, modify syntax_model.md and follow the steps given in the reference section to extend the scheme for the new construction.
If the scheme is already clear, but the analysis failed anyway, use this recipe to add a GoldExample with a correct canned_answer in tests/fixtures/gold_examples.py. If you’re unsure between two defensible readings, say so explicitly in a comment above the fixture rather than silently committing to one.
Re-run pytest to confirm that the new or corrected fixture validates and that test_coverage.py is satisfied.
Successful analyses
When arsgrammatica correctly analyzes a passage, that is real evidence that the model is working correctly. We can save manually vetted examples as checks to ensure that future changes to a prompt or model can’t silently break this case, and to extend the evaluation pool for later optimization.
An unusual or difficult construction
If the response correctly analyzes an unusual or difficult construction that you think it might not always get right, this is worth reinforcing in the core training set. If you save the analysis to a file, you can use the script utilities/analysis_to_gold_example.py to write the necessary Python for you that you can paste in to gold_examples.py.
Ordinary constructions
If the analysis correctly identified something you already expected it to get right, such as a simple independent clause with direct object, this is still real evidence, but of low value as training signal. If we add more examples of easy cases that the model already gets right, it teaches the optimizer nothing new, and dilutes the training set with redundant coverage.
Instead, serialize and save the analysis in a directory you can later pass to optimize_gepa.py with the --eval-dir parameter as part of the larger training and evaluation set.
Optimization over time
Every addition to GOLD_EXAMPLES (whether a correction for a failure or a harvested success with a rare construction) becomes part of the training set for the next time optimize_gepa.py runs.
Every analysis you save expands the pool GEPA can use to optimize prompts for specific models.
Optimizing vs. overtraining
If you save a large number of correct analyses as you work through a corpus, and then include all those analyses in the training set for GEPA, the result might perform better on your particular corpus, but worse on a broader corpus of Latin. In one experiment, for example, an optimized model drawing on a large number of analyzed passages from the Latin Vulgate performed worse on a selection of passages from Livy than an unoptimized generic model. Presumably it would do better on further passages of the Vulgate.
Your specific strategy for optimization will depend on your goal: do want a more specialized system tailored to a particular corpus, or a more generally applicable system?
The github repository for arsgrammatica organizes a selection of hand-vetted analyses in subdirectories of the data directory by corpus, so they can easily be included or excluded with the --eval-dir parameter to optimize_gepa.py.