https://github.com/tyxsspa/AnyText Skip to content Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} tyxsspa / AnyText Public * Notifications * Fork 136 * Star 2k License Apache-2.0 license 2k stars 136 forks Activity Star Notifications * Code * Issues 28 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights tyxsspa/AnyText This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/t] Use Git or checkout with SVN using the web URL. [gh repo clone tyxssp] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit Git stats * 16 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time cldm docs example_images font javascript ldm models_yaml ocr_recog ocr_weights .gitignore LICENSE README.md bert_tokenizer.py dataset_util.py demo.py environment.yaml inference.py style.css t3_dataset.py util.py View code [ ] AnyText: Multilingual Visual Text Generation And Editing News [?]TODOs Methodology Installation Inference Gallery Evaluation Citation README.md AnyText: Multilingual Visual Text Generation And Editing [6874747073] [6874747073] [6874747073] [6874747073] sample News [2024.01.04] - FP16 inference is available, 3x faster! Now the demo can be deployed on GPU with >8GB memory. Enjoy! [2024.01.04] - HuggingFace Online demo is available here! [2023.12.28] - ModelScope Online demo is available here! [2023.12.27] - We released the latest checkpoint(v1.1) and inference code, check on modelscope in Chinese. [2023.12.05] - The paper is available at here. [?]TODOs * [*] Release the model and inference code * [*] Provide publicly accessible demo link * [ ] Provide a free font file() * [ ] Release tools for merging weights from community models or LoRAs * [ ] Support AnyText in stable-diffusion-webui() * [ ] Release AnyText-benchmark dataset and evaluation code * [ ] Release AnyWord-3M dataset and training code Methodology AnyText comprises a diffusion pipeline with two primary elements: an auxiliary latent module and a text embedding module. The former uses inputs like text glyph, position, and masked image to generate latent features for text generation or editing. The latter employs an OCR model for encoding stroke data as embeddings, which blend with image caption embeddings from the tokenizer to generate texts that seamlessly integrate with the background. We employed text-control diffusion loss and text perceptual loss for training to further enhance writing accuracy. framework Installation # Install git (skip if already done) conda install -c anaconda git # Clone anytext code git clone https://github.com/tyxsspa/AnyText.git cd AnyText # Prepare a font file; Arial Unicode MS is recommended, **you need to download it on your own** mv your/path/to/arialuni.ttf ./font/Arial_Unicode.ttf # Create a new environment and install packages as follows: conda env create -f environment.yaml conda activate anytext Inference [Recommend]: We release a demo on ModelScope and HuggingFace! AnyText include two modes: Text Generation and Text Editing. Running the simple code below to perform inference in both modes and verify whether the environment is correctly installed. python inference.py If you have advanced GPU (with at least 8G memory), it is recommended to deploy our demo as below, which includes usage instruction, user interface and abundant examples. export CUDA_VISIBLE_DEVICES=0 && python demo.py FP16 inference is used as default, and a Chinese-to-English translation model is loaded for direct input of Chinese prompt (occupying ~4GB of GPU memory). The default behavior can be modified, as the following command enables FP32 inference and disables the translation model: export CUDA_VISIBLE_DEVICES=0 && python demo.py --use_fp32 --no_translator If FP16 is used and the translation model not used(or load it on CPU, see here), generation of one single 512x512 image will occupy ~7.5GB of GPU memory. In addition, other font file can be used by(although the result may not be optimal): export CUDA_VISIBLE_DEVICES=0 && python demo.py --font_path your/path/to/font/file.ttf demo Please note that when executing inference for the first time, the model files will be downloaded to: ~/.cache/modelscope/hub. If you need to modify the download directory, you can manually specify the environment variable: MODELSCOPE_CACHE. Gallery gallery Evaluation We use Sentence Accuracy (Sen. ACC) and Normalized Edit Distance (NED) to evaluate the accuracy of generated text, and use the FID metric to assess the quality of generated images. Compared to existing methods, AnyText has a significant advantage in both Chinese and English text generation. eval Citation @article{tuo2023anytext, title={AnyText: Multilingual Visual Text Generation And Editing}, author={Yuxiang Tuo and Wangmeng Xiang and Jun-Yan He and Yifeng Geng and Xuansong Xie}, year={2023}, eprint={2311.03054}, archivePrefix={arXiv}, primaryClass={cs.CV} } About No description, website, or topics provided. Resources Readme License Apache-2.0 license Activity Stars 2k stars Watchers 39 watching Forks 136 forks Report repository Releases No releases published Packages 0 No packages published Contributors 2 * @tyxsspa tyxsspa * @eltociear eltociear Ikko Eltociear Ashimine Languages * Python 96.6% * JavaScript 3.3% * CSS 0.1% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.