https://github.com/lukas-blecher/LaTeX-OCR Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} lukas-blecher / LaTeX-OCR Public * Notifications * Fork 560 * Star 5.5k pix2tex: Using a ViT to convert images of equations into LaTeX code. lukas-blecher.github.io/latex-ocr/ License MIT license 5.5k stars 560 forks Activity Star Notifications * Code * Issues 78 * Pull requests 9 * Discussions * Actions * Projects 4 * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Security * Insights lukas-blecher/LaTeX-OCR This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 4 branches 7 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/l] Use Git or checkout with SVN using the web URL. [gh repo clone lukas-] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @lukas-blecher lukas-blecher zsh #301 ... 1781514 Sep 16, 2023 zsh #301 1781514 Git stats * 306 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time docker add docker file for api May 3, 2022 20:25 docs Update pix2tex.rst May 20, 2022 16:59 notebooks +extra train tag, mv transformations to sep. file May 3, 2022 12:30 pix2tex Allow overriding screenshot tool with environment variable August 18, 2023 11:03 .gitignore multi-gpu train(data-parallelism) + pure ViT May 16, 2022 21:15 .readthedocs.yaml try to fix build May 3, 2022 16:03 LICENSE Create LICENSE January 30, 2021 12:44 MANIFEST.in setup pipy package April 13, 2022 16:19 README.md zsh #301 September 16, 2023 14:20 setup.cfg setup pipy package April 13, 2022 16:19 setup.py Bump up version number May 21, 2023 13:22 View code [ ] pix2tex - LaTeX OCR Using the model Training the model Model Performance Data Dataset Requirements Fonts TODO Contribution Acknowledgment References README.md pix2tex - LaTeX OCR GitHub Documentation Status PyPI PyPI - Downloads GitHub all releases Docker Pulls Open In Colab Hugging Face Spaces The goal of this project is to create a learning based system that takes an image of a math formula and returns corresponding LaTeX code. header Using the model To run the model you need Python 3.7+ If you don't have PyTorch installed. Follow their instructions here. Install the package pix2tex: pip install "pix2tex[gui]" Model checkpoints will be downloaded automatically. There are three ways to get a prediction from an image. 1. You can use the command line tool by calling pix2tex. Here you can parse already existing images from the disk and images in your clipboard. 2. Thanks to @katie-lim, you can use a nice user interface as a quick way to get the model prediction. Just call the GUI with latexocr. From here you can take a screenshot and the predicted latex code is rendered using MathJax and copied to your clipboard. Under linux, it is possible to use the GUI with gnome-screenshot (which comes with multiple monitor support) if gnome-screenshot was installed beforehand. For Wayland, grim and slurp will be used when they are both available. Note that gnome-screenshot is not compatible with wlroots-based Wayland compositors. Since gnome-screenshot will be preferred when available, you may have to set the environment variable SCREENSHOT_TOOL to grim in this case (other available values are gnome-screenshot and pil). demo If the model is unsure about the what's in the image it might output a different prediction every time you click "Retry". With the temperature parameter you can control this behavior (low temperature will produce the same result). 3. You can use an API. This has additional dependencies. Install via pip install -U "pix2tex[api]" and run python -m pix2tex.api.run to start a Streamlit demo that connects to the API at port 8502. There is also a docker image available for the API: https:// hub.docker.com/r/lukasblecher/pix2tex Docker Image Size (latest by date) docker pull lukasblecher/pix2tex:api docker run --rm -p 8502:8502 lukasblecher/pix2tex:api To also run the streamlit demo run docker run --rm -it -p 8501:8501 --entrypoint python lukasblecher/pix2tex:api pix2tex/api/run.py and navigate to http://localhost:8501/ 4. Use from within Python from PIL import Image from pix2tex.cli import LatexOCR img = Image.open('path/to/image.png') model = LatexOCR() print(model(img)) The model works best with images of smaller resolution. That's why I added a preprocessing step where another neural network predicts the optimal resolution of the input image. This model will automatically resize the custom image to best resemble the training data and thus increase performance of images found in the wild. Still it's not perfect and might not be able to handle huge images optimally, so don't zoom in all the way before taking a picture. Always double check the result carefully. You can try to redo the prediction with an other resolution if the answer was wrong. Want to use the package? I'm trying to compile a documentation right now. Visit here: https://pix2tex.readthedocs.io/ Training the model Open In Colab Install a couple of dependencies pip install "pix2tex[train]". 1. First we need to combine the images with their ground truth labels. I wrote a dataset class (which needs further improving) that saves the relative paths to the images with the LaTeX code they were rendered with. To generate the dataset pickle file run python -m pix2tex.dataset.dataset --equations path_to_textfile --images path_to_images --out dataset.pkl To use your own tokenizer pass it via --tokenizer (See below). You can find my generated training data on the Google Drive as well (formulae.zip - images, math.txt - labels). Repeat the step for the validation and test data. All use the same label text file. 2. Edit the data (and valdata) entry in the config file to the newly generated .pkl file. Change other hyperparameters if you want to. See pix2tex/model/settings/config.yaml for a template. 3. Now for the actual training run python -m pix2tex.train --config path_to_config_file If you want to use your own data you might be interested in creating your own tokenizer with python -m pix2tex.dataset.dataset --equations path_to_textfile --vocab-size 8000 --out tokenizer.json Don't forget to update the path to the tokenizer in the config file and set num_tokens to your vocabulary size. Model The model consist of a ViT [1] encoder with a ResNet backbone and a Transformer [2] decoder. Performance BLEU score normed edit distance token accuracy 0.88 0.10 0.60 Data We need paired data for the network to learn. Luckily there is a lot of LaTeX code on the internet, e.g. wikipedia, arXiv. We also use the formulae from the im2latex-100k [3] dataset. All of it can be found here Dataset Requirements In order to render the math in many different fonts we use XeLaTeX, generate a PDF and finally convert it to a PNG. For the last step we need to use some third party tools: * XeLaTeX * ImageMagick with Ghostscript. (for converting pdf to png) * Node.js to run KaTeX (for normalizing Latex code) * Python 3.7+ & dependencies (specified in setup.py) Fonts Latin Modern Math, GFSNeohellenicMath.otf, Asana Math, XITS Math, Cambria Math TODO * [*] add more evaluation metrics * [*] create a GUI * [ ] add beam search * [ ] support handwritten formulae (kinda done, see training colab notebook) * [ ] reduce model size (distillation) * [ ] find optimal hyperparameters * [ ] tweak model structure * [ ] fix data scraping and scrape more data * [ ] trace the model (#2) Contribution Contributions of any kind are welcome. Acknowledgment Code taken and modified from lucidrains, rwightman, im2markup, arxiv_leaks, pkra: Mathjax, harupy: snipping tool References [1] An Image is Worth 16x16 Words [2] Attention Is All You Need [3] Image-to-Markup Generation with Coarse-to-Fine Attention About pix2tex: Using a ViT to convert images of equations into LaTeX code. lukas-blecher.github.io/LaTeX-OCR/ Topics python machine-learning ocr latex deep-learning image-processing pytorch dataset transformer vit image2text im2text im2latex im2markup math-ocr vision-transformer latex-ocr Resources Readme License MIT license Activity Stars 5.5k stars Watchers 45 watching Forks 560 forks Report repository Releases 7 tags Contributors 16 * @lukas-blecher * @TITC * @Freed-Wu * @katie-lim * @FrankFrank9 * @JoepdeJong * @kxxt * @MoetaYuko * @muyuuuu * @frankier * @zhouzq-thu + 5 contributors Languages * Python 96.5% * JavaScript 1.8% * Jupyter Notebook 1.6% * Other 0.1% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.