How to add a gold example

A fundamental part of the DSPy development are the GoldExample pairings of a Latin passage with a hand-verified canned_answer. These are defined in tests/fixtures/gold_examples.py. Every GoldExample entry lives in the single GOLD_EXAMPLES list at the bottom of that file. You can add examples there that you have either hand-written, or harvested from successful analyses.

Harvesting from a successful analysis

If you have analyzed a difficult construction that you want to add to the core training sets, you can save the analysis and use the analysis_to_gold_example.py script to compose the Python you need.

Pass it two arguments: the analysis file, and an identifying label for the analysis. The identifying label is a short, unique, descriptive id (e.g. cicero_proarchia_exordium) that serves as the pytest id for every test parametrized over GOLD_EXAMPLES. Include the --tags parameter to name constructions the example is meant to exercise, for a human skimming the file. Run analysis_to_gold_example -h to see all the optional parameters the script accepts.

Run it like this:

python3 utilities/analysis_to_gold_example.py analysisfile.cex cicero_proarchia_exordium --tags "complex subordination"

This prints to standard output a string in two parts . The first part is a Python dictionary, which in this example will be named _CICERO_PROARCHIA_EXORDIUM_ANSWER. The dictionary contains the complete analysis data (the model’s reasoning, the verbal expressions table, and the token graph).

The second part is a GoldExample object identifying the labelled analysis as the correct canned_answer for the passage. In our example, it would look like this:


GoldExample(
    slug="cicero_proarchia_exordium",
    passage="ac ne quis a nobis hoc ita dici forte miretur, quod alia quaedam in hoc facultas sit ingeni neque haec dicendi ratio aut disciplina, ne nos quidem huic uni studio penitus umquam dediti fuimus.",
    tags=["complex subordination"],
    canned_answer=_CICERO_PROARCHIA_EXORDIUM_ANSWER,
),

In tests/fixtures/gold_examples.py, you can paste the dict above the list beginning GOLD_EXAMPLES = [ and place the new GoldExample(...) entry inside the list.

Be sure to review the reasoning property of dict you’ve pasted in. It’s the model’s own recorded reasoning, and not guaranteed to read the way you would write it by hand.

Hand writing a GoldExample

If you want to add a small, deliberately-constructed example targeting one specific construction, refer to the existing examples and:

  1. Write a module-level _SOME_ANSWER = {...} dict: reasoning (a real explanation, not a placeholder), verbalunits (one entry per verbal expression – id/syntactic_type/semantic_type), tokengraph (one entry per token – id/token/tokentype, plus whichever of lemma/verbalunitid/relatedtoken1/relationship1/relatedtoken2/relationship2 actually apply; omit a field entirely rather than setting it to None, matching every existing entry). See arsgrammatica/models.py’s TokenAnalysis/VerbalExpression and syntax_model.md for what’s allowed, and notes/relationship_labels.md for the relation-label catalog.
  2. Add a GoldExample(slug=..., passage=..., tags=[...], canned_answer=_SOME_ANSWER) entry to GOLD_EXAMPLES, referencing that dict.

Before you add it

  • Run pytest -k <slug>: test_gold_examples.py checks validate() and a clean Mermaid render for every fixture automatically, no test-file changes needed.
  • test_coverage.py reports which relation labels / verbal-expression types / tokentypes still have no example – worth a look if you’re deliberately filling a gap.