Utilities to simplify analyzing a corpus

Analyzing a corpus

You can analyze an entire corpus in a CEX-format file with analyze_ctsdata_to_files.py. An analysis file is written for each sentence to an output directory you provide.

python utilities/analyze_ctsdata_to_files.py corpus.cex 
     --output-dir analyses/

Working with part of a corpus

You may not be able to analyze a large corpus all at once. The utilities directory includes scripts that do not require a language model to help you work with parts of a corpus.

Group cited passages by sentence boundaries

If you want to analyze a subset of a corpus where sentences and citation units do not always coincide (typical for Latin poetry cited by verse line), you can group passages into sequences that begin and end on sentence boundaries with group_ctsdata_by_sentence.py. This script writes to standard output a list of URNs for each group on a single line.

If we segemented a text of the Aeneid like this:

python utilities/group_ctsdata_by_sentence.py aeneid.cex

the first two lines would list passages for a sentence beginning at Aeneid 1.1 and a second sentence beginning at Aeneid 1.8*, like this:

urn:cts:latinLit:phi0690.phi003:1.1 urn:cts:latinLit:phi0690.phi003:1.2 urn:cts:latinLit:phi0690.phi003:1.3 urn:cts:latinLit:phi0690.phi003:1.4 urn:cts:latinLit:phi0690.phi003:1.5 urn:cts:latinLit:phi0690.phi003:1.6 urn:cts:latinLit:phi0690.phi003:1.7
urn:cts:latinLit:phi0690.phi003:1.8 urn:cts:latinLit:phi0690.phi003:1.9 urn:cts:latinLit:phi0690.phi003:1.10 urn:cts:latinLit:phi0690.phi003:1.11

Use normal IO redirection to save the results in a file:

python utilities/group_ctsdata_by_sentence.py aeneid.cex > aeneid-linegroups.txt

Define citable tokens and sentence units

It can be useful to cite tokens with the same stable identifiers produced by a syntactic analysis with arsgrammatica. The script tokenize_ctsdata.py reads a text corpus from a CEX-format file, and writes to standard output a delimited-text listing of all the tokens in the corpus, and a list of sentences defined by the starting and ending token of each sentence.

python utilities/tokenize_ctsdata.py corpus.cex > tokens-sentences.cex

For details of the output format, see the reference documentation