3

I am looking for a (rule-based) algorithm to segmentize Esperanto words selecting roots, affixes, and endings. It should also recognize compound words such as 'vaporŝipo' or even 'fivaporŝipegaro'. I just wanted to ask before I do it myself.

Thanks for all tips, programming language or format does not matter.

Improve this question

2 Answers 2

Reset to default
5

The online dictionary La Simpla Vortaro, while basing its content on the data of ReVo, also analyses the searched word for whether it could be a compound word. It can also handle compounds that have more than one interpretation, such as kolego.

The creator of simplavortaro.org, Vilfredo, explains the working principle of their approach or "algorithm" (as well as some challenges and how they solved them) in two blog posts:

The gist of it seems to be:

  1. remove the word type marker, if any (e.g. -o for nouns)
  2. generate all prefixes¹ of the thus shortened word
  3. of these prefixes, keep those that can be found in a list of word building parts (roots, affixes and complete² nouns and adjectives)
  4. for each of the kept prefixes, split it off the beginning of the word and repeat with the remainders from step 2

This generates a list of (potentially many) combinations on how the word might be built from word building parts. These combinations are weighted against each other according to the formula

badness := (number of word building parts) – 0.5 × (number of affixes)

If I'm not mistaken, as affixes are also word building parts, this is equivalent to

badness := 1 × (number of non-affix word building parts) + 0.5 × (number of affixes)

(A penalty function that is a linear combination with only positive weights might be easier to reason about.)

simplavortaro.org then displays only the two "least bad" options, with the "worse" of them second.

The website's (or web application's, if you will) code is free software (licensed under GNU AGPL 3). The part doing the segmentation and weighting seems to be the function parse_morphology(word) from the package vortaro.morphology.

The website's search functionality is also available through a simple RESTful API (with only a single endpoint, it seems), whose responses also include the segmentation results.

¹all strings that match the beginning of the word, not all of which necessarily are prefixes in the sense of Esperanto's word building system.
²"complete" as in: including the word type marker

Improve this answer
3
  • Thanks das-g a lot, it is very helpful! Alas, I have forgotten my old account and needed to start a new one, in which I am very new and cannot upvote :-(
    – Stacy Maimoon
    Commented Oct 20, 2022 at 7:47
  • after reading Vilfredo's postings. I created an Esperanto segmentizer myself (Python), to try to create all possible decompositions of a word into meaningful subwords (roots, affixes, endings, thematic insertions) and had to see that tis is not such a simple task, therefore exchange with like-minded individuals is valuable. My segmentizer is not perfekt, but so is Vilfredo's, alas. Eg if I take 'malvaporŝipego' as the probe, Vilfredo delivers 'mal'+'vapor'+'ŝi'+'peg'+'o' as an option, which is clearly invalid ('ŝi' cannot be there, imho).
    – Stacy Maimoon
    Commented Oct 20, 2022 at 8:09
  • I doubt if the problem can be solved 100% rule-based, especially if the accent be on 'really' meaningful combinations. BTW, my goal is partly an exercise in NLP, partly a preparation for a similar task in a non-constructed agglutinative language
    – Stacy Maimoon
    Commented Oct 20, 2022 at 8:13
2

There's not going to be any rule-based algorithm that can do the job 100%. There are common jokes in Esperanto with words like financo (finance), which is of course fi- (shameful, immoral) plus nanco (which presumably means finance without the shameful part; generally assumed to be unnecessary). More seriously, nanco is also a word for Byrsonima crassifolia, or nance (aka golden spoon aka hogberry).

fidelisrafael/esperanto-analyzer was found on a quick search; I'm sure there's other code out there that does a similar job, possibly better, possibly worse.

Improve this answer
1
  • Thanks @prosfilaes! (again, not enough reputation to upvote). I looked at the esperanto-analyzer. It makes the simple POS-tagging, to my view, which is, imho, rather trivial for Esperanto.
    – Stacy Maimoon
    Commented Oct 20, 2022 at 18:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.