https://github.com/PellelNitram/xournalpp_htr Skip to content Navigation Menu Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + GitHub Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions By size + Enterprise + Teams + Startups By industry + Healthcare + Financial services + Manufacturing By use case + CI/CD & Automation + DevOps + DevSecOps * Resources Topics + AI + DevOps + Security + Software Development Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} PellelNitram / xournalpp_htr Public * Notifications You must be signed in to change notification settings * Fork 1 * Star 19 Developing handwritten text recognition for Xournal++ License GPL-2.0 license 19 stars 1 fork Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights PellelNitram/xournalpp_htr This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master BranchesTags Go to file Code Folders and files Name Name Last commit Last commit message date Latest commit History 179 Commits .github/workflows .github/ workflows data data docs/images docs/images plugin plugin scripts scripts tests tests xournalpp_htr xournalpp_htr .gitignore .gitignore INSTALL_LINUX.sh INSTALL_LINUX.sh LICENSE LICENSE Makefile Makefile README.md README.md pytest.ini pytest.ini requirements.txt requirements.txt setup.py setup.py View all files Repository files navigation * README * GPL-2.0 license Xournal++ HTR python PRs Developing handwritten text recognition for Xournal++. Your contributions are greatly appreciated! Xournal++ HTR in 90 seconds YouTube (Click on image or here to get to video.) Installation Cross-platform Execute the following commands: 1. Create an environment: conda create --name xournalpp_htr python= 3.10.11. 2. Use this environment: conda activate xournalpp_htr. 3. Install HTRPipelines package using its installation guide. 4. Install all dependencies of this package pip install -r requirements.txt. 5. Install the package in development mode with pip install -e . (do not forget the dot, '.'). 6. Move plugin/ folder content to ${XOURNAL_CONFIG_PATH}/plugins/ xournalpp_htr/ with ${XOURNAL_CONFIG_PATH} being the configuration path of Xournal++, see Xournal++ manual here. Linux Run bash INSTALL_LINUX.sh from repository root directory. This script also installs the plugin as explained in point 5 of the cross-platform installation procedure. The installation of the plugin is performed with plugin/copy_to_plugin_folder.sh, which can also be invoked independently of INSTALL_LINUX.sh for updating the plugin installation. After installation Confirm that the installation worked by running make tests-installation from repository root directory. Project description Taking handwritten notes digitally comes with many benefits but lacks searchability of your notes. Hence, there is a need to make your handwritten notes searchable. This can be achieved with "handwritten text recognition" (HTR), which is the process of assigning searchable text to written strokes. While many commercial note taking apps feature great HTR systems to make your notes searchable and there are a number of existing open-source implementations of various algorithms, there is no HTR feature available in an open-source note taking application that is privacy aware and processes your data locally. The purpose of the Xournal++ HTR project is to change that! Xournal++ HTR strives to bring open-source on-device handwriting recognition to Xournal++ as it is one of the most adopted open-source note taking apps and thereby HTR can be delivered to the largest possible amount of users. Project design The design of Xournal++ HTR tries to bridge the gap between both delivering a production ready product and allowing contributors to experiment with new algorithms. The project design involves a Lua plugin and a Python backend, see the following figure. First, the production ready product is delivered by means of an Xournal++ plugin. The plugin is fully integrated in Xournal++ and calls a Python backend that performs the actual transcription. The Python backend allows selection of various recognition models and is thereby fully extendable with new models. [system_design] Design of xournalpp_htr. Developing a usable HTR systems requires experimentation. The project structure is set up to accommodate this need. Note that ideas on improved project structures are appreciated. The experimentation is carried out in terms of "concepts". Each concept explores a different approach to HTR and possibly improves over previous concepts, but not necessarily to allow for freedom in risky experiments. Concept 1 is already implemented and uses a computer vision approach that is explained below. Future concepts might explore: * Retrain computer vision models from concept 1 using native data representation of Xournal++ * Use sequence-to-sequence models to take advantage of native data representation of Xournal++ * Use data augmentation to increase effective size of training data * Use of language models to correct for spelling mistakes Concept 1 This concept uses computer vision based algorithms to first detect words on a page and then to read those words. The following shows a video demo on YouTube using real-life handwriting data from a Xournal file: Xournal++ HTR - Concept 1 - Demo Despite not being perfect, the main take away is that the performance is surprisingly good given that the underlying algorithm has not been optimised for Xournal++ data at all. The performance is sufficiently good to be useful for the Xournal++ user base. Feel free to play around with the demo yourself using this code after installing this project. The "concept 1" is also what is currently used in the plugin and shown in the 90 seconds demo. Next steps to improve the performance of the handwritten text recognition even further could be: * Re-train the algorithm on Xournal++ specific data, while potentially using data augmentation. * Use language model to improve text encoding. * Use sequence-to-sequence algorithm that makes use of Xournal++'s data format. This translates into using online HTR algorithms. I would like to acknowledge Harald Scheidl in this concept as he wrote the underlying algorithms and made them easily usable through his HTRPipeline repository - after all I just feed his algorithm Xournal++ data in concept 1. Go check out his great content! Usage 1. Activate environment: conda activate xournalpp_htr. Alternatively use source activate_env.sh as shortcut. 2. Use the code. 3. To update the requirements file: pip freeze > requirements.txt. Community contributions The following branching strategy is used to keep the master branch stable and allow for experimentation: master > dev > feature branches. In more details, this repository currently follows the following git branching strategy: The master branch remains stable and delivers a functioning product. The dev branch consists of all code that will be merged to master eventually where the corresponding features are developed in individual feature branches, see the feature_1 branch in the below visualisation. Generally, the branching strategy is depicted in this below visualisation: %%{init:{ "gitGraph":{ "mainBranchName":"master" }}}%% gitGraph commit commit branch dev commit checkout dev branch feature_1 checkout feature_1 commit commit commit checkout dev merge feature_1 checkout master merge dev commit commit Loading The CI/CD pipeline runs on the branches master and dev. Acknowledgements I would like to thank Leonard Salewski and Jonathan Prexl for useful discussions, Harald Scheidl for making his repositories about handwritten text recognition public (SimpleHTR, WordDetectorNN and HTRPipeline) and the School of Physics and Astronomy at The University of Edinburgh for providing compute power. About Developing handwritten text recognition for Xournal++ Resources Readme License GPL-2.0 license Activity Stars 19 stars Watchers 6 watching Forks 1 fork Report repository Releases No releases published Packages 0 No packages published Contributors 2 * @PellelNitram PellelNitram Martin Lellep * @LibrEars LibrEars Languages * Python 84.9% * Lua 7.7% * Shell 6.6% * Makefile 0.8% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.