[HN Gopher] PyCantonese: Cantonese Linguistics and NLP in Python
       ___________________________________________________________________
        
       PyCantonese: Cantonese Linguistics and NLP in Python
        
       Author : teleforce
       Score  : 49 points
       Date   : 2022-06-21 14:05 UTC (8 hours ago)
        
 (HTM) web link (pycantonese.org)
 (TXT) w3m dump (pycantonese.org)
        
       | throwaway290 wrote:
       | Cantonese is in my opinion a richer and more interesting language
       | than Mandarin, yet much more difficult in terms of finding
       | resources and tools for working with it. I wish I could support
       | this project but for some reason none of Github's paid features,
       | including sponsoring, seem to work on my account.
        
       | contingencies wrote:
       | Whenever someone talks about Cantonese I instantly think of _Ah
       | To_ 's cartoons of Cantonese proverbs:
       | https://says.com/my/fun/81-chinese-cantonese-proverbs-in-one...
       | https://writecantonese8.wordpress.com/2014/02/25/cantonese-p...
        
       | fartcannon wrote:
       | Nice to see some work being done with Cantonese. Large part of
       | North American Chinese speak Cantonese but its rarely supported
       | officially in any tools.
        
         | kelvie wrote:
         | Google's cantonese IME is a godsend for us in this group (that
         | never learned other pingyin)
        
       | closed wrote:
       | As a person learning cantonese, pycantonese and
       | cantonese.sheik.co.uk were so helpful!
       | 
       | The trickiest thing to explain to people is that Cantonese is a
       | diaglossika. It's spoken, but to write you would use mandarin. So
       | there aren't a ton of giant written language corpuses to train
       | on.
       | 
       | AFAICT that's why tools like this often use simple heuristics to
       | do word segmentation. I remember going down a deep rabbit hole,
       | before finally just packaging a small tool to do cantonese word
       | segmentation using the A* algorithm:
       | 
       | https://github.com/machow/cantocut
        
       | totoglazer wrote:
       | Seems wild to build a new python NLP library totally disconnected
       | from the current ecosystem. Particularly spaCy, but really any of
       | NLTK, gensim, huggingface. Would love to understand the context
       | of that choice.
        
         | [deleted]
        
         | solarmist wrote:
         | Because Academia. "Lee, Jackson L., Litong Chen, Charles Lam,
         | Chaak Ming Lau, and Tsz-Him Tsui. 2022. PyCantonese: Cantonese
         | Linguistics and NLP in Python. Proceedings of the 13th Language
         | Resources and Evaluation Conference."
         | 
         | That said parsing is not a solved problem in a lot of languages
         | (particularly ones without spaces) and frameworks like spaCy
         | just integrate existing solutions into a common pipeline and
         | datastructures.
        
       ___________________________________________________________________
       (page generated 2022-06-21 23:02 UTC)