proycon / python-ucto Public

This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is regular-expression based, extensible, and advanced tokeniser writt…

29 stars 5 forks Branches Tags Activity

Notifications

Error
Looks like something went wrong!

About

This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is regular-expression based, extensible, and advanced tokeniser writt…

python nlp tokenizer computational-linguistics text-processing nlp-library folia

Report repository

Releases 21

Packages

No packages published

Languages