User's guide: tokenization

The Orthography module also defines a tokenize function.

using Orthography
using Lycian
ortho  = lycianAscii()
tokens = tokenize( "ebENnE : xopA : mE=ti prNnawatE", ortho)
tokens

# output

7-element Vector{OrthographicToken}:
 OrthographicToken("ebENnE", LexicalToken())
 OrthographicToken(":", PunctuationToken())
 OrthographicToken("xopA", LexicalToken())
 OrthographicToken(":", PunctuationToken())
 OrthographicToken("mE", LexicalToken())
 OrthographicToken("ti", LexicalToken())
 OrthographicToken("prNnawatE", LexicalToken())