Lexical tokens are composed only of valid Greek characters as specified for Greek strings, except white-space characters. Lexical tokens may be created directly from strings, and converted back to strings, following the same rules as for Greek strings.
Create a lexical token from a string, and convert it back to a string:
| Source string | Token as a String |
|---|---|
| MH=NIN | mh=nin |
| mh=nin | mh=nin |
| Mh=nin | mh=nin |
| *mh=nin | *mh=nin |
A Greek string may comprise more than one lexical token. White space characters that are permitted in GreekStrings but not in Greek words delimit tokens within a String. The GreekString class can create a list of Greek Words from a Greek String.
Tokenizing the following Unicode string
Ζεὺς δ' Ἔριδα προΐαλλε θοὰς ἐπὶ νῆας Ἀχαιῶν
yields this ordered list of GreekWords:
| Verb (with abbreviated prefix) |
|---|
| *zeu\s |
| d' |
| *)/erida |
| proi+/alle |
| qoa\s |
| e)pi\ |
| nh=as |
| *)axaiw=n |
Like GreekStrings, GreekWords can be represented as Unicode strings in NFC form.
| Source string | As Unicode |
|---|---|
| *zeu\s | Ζεὺς |
| d' | δʼ |
| *)/erida | Ἔριδα |
| proi+/alle | προΐαλλε |
| qoa\s | θοὰς |
| e)pi\ | ἐπὶ |
| nh=as | νῆας |
| *)axaiw=n | Ἀχαιῶν |