NLP Esperantilo is an educational project that helps you build a Natural Language Processing (NLP) library for Esperanto. The goal is to provide a simple and comprehensive suite of tools for researchers, developers, and enthusiasts working with the Esperanto language. It also hosts a guide on how to build a similar NLP toolchain, for educational purposes.
📖 Documentation: https://jparisu.github.io/nlp-esperantilo/
The documentation has two parts:
- Library — the reference
manual of the
esperantiloexample package: a what it does today, and what every public name means. Its API reference is generated from the docstrings insrc/. - Guide — how a library like it is built and shipped: Git, GitHub, packaging and Esperanto linguistics.
nlp-esperantilo/
├── .github/workflows/ # Continuous integration: tests, docs, previews, spell check
├── docs/ # Documentation sources (MkDocs)
│ ├── library/ # reference manual of the library
│ ├── guide/ # the guide: Git, GitHub, Python, Esperanto
│ └── hooks/ # build-time hooks that turn resources/ into pages
├── resources/ # Data and notebooks that are not code
│ ├── esperanto/ # machine-readable word lists (JSON) and reference texts
│ └── notebooks/ # notebooks for students, meant for Google Colab
├── src/esperantilo/ # The Python library
│ ├── nlp/ # analysing text that is already in hand
│ └── wiki/ # fetching text from Wikipedia
├── tests/ # Test suite (pytest)
├── mkdocs.yml # Documentation configuration
└── pyproject.toml # Package metadata and build configuration
Directly from GitHub, which is the recommended way while the library is still under development:
pip install git+https://github.com/jparisu/nlp-esperantilo.gitimport esperantilo
# Analyse text you already have.
esperantilo.sentence_tokenizer("Zamenhof kreis Esperanton. Ĉu vere? Jes!")
# ['Zamenhof kreis Esperanton.', 'Ĉu vere?', 'Jes!']
# Or fetch it from Wikipedia first, in any language.
page = esperantilo.WikiPage.look_up("Esperanto", language="eo")
esperantilo.sentence_tokenizer(page.section(page.title))The library is at 0.1.0 and ships two features, split into one subpackage
each: esperantilo.wiki reads Wikipedia articles as plain text, and
esperantilo.nlp analyses text that is already in hand — sentence segmentation
so far. Tokens, lemmas and affix analysis are still a design target — building
them is the exercise the guide prepares you for.
Its only runtime dependency is requests,
which pip installs with it.
python -m venv .venv
source .venv/bin/activate
pip install -e ".[test]" # library and test dependencies
pytest # run the test suitepip install -r docs/requirements.txt
mkdocs serve # live preview at http://127.0.0.1:8000
mkdocs build --strict # the same build the CI runsThe site is built with Material for MkDocs
and published to GitHub Pages on every push to main.
Translations live next to the English pages as <page>.<locale>.md; missing
translations fall back to English.
Every pull request also gets a live preview of the whole site at
https://jparisu.github.io/nlp-esperantilo/pr-preview/pr-<number>/, linked from
a comment on the pull request. The public site is not touched until the pull
request is merged, and the preview is removed when it closes.
The linguistic data lives in resources/ as JSON, and
the documentation pages under Esperanto → Word lists are generated from those
files at build time by docs/hooks/word_lists.py. Adding an
entry to resources/esperanto/listoj.json is enough for a new page to appear — no page to
write, no navigation entry to add. The pages are English only.
Every word lives once in resources/esperanto/vortoj.json; each
list in resources/esperanto/listoj.json is a filter over it, so
a preposition that is also a stop-word is stored once and shown by both lists.
pip install codespell
codespell # settings in [tool.codespell] in pyproject.tomlProject-specific vocabulary (Esperanto words, proper names) goes in
.codespell/ignore-words.txt.
Apache License 2.0 — see LICENSE.