https://github.com/microsoft/unilm Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Blog * Solutions + For + Enterprise + Teams + Startups + Education + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} microsoft / unilm Public * Notifications * Fork 1.6k * Star 9.6k Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities aka.ms/nlpagi License MIT license 9.6k stars 1.6k forks Star Notifications * Code * Issues 280 * Pull requests 13 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights microsoft/unilm This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 4 branches 4 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/m] Use Git or checkout with SVN using the web URL. [gh repo clone micros] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @MarkWuNLP MarkWuNLP Merge pull request #1010 from Sanyuan-Chen/beats ... 0ac3e16 Mar 1, 2023 Merge pull request #1010 from Sanyuan-Chen/beats Update download links for BEATs, WavLM and SpeechLM 0ac3e16 Git stats * 912 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/ISSUE_TEMPLATE Update issue templates May 29, 2020 16:51 adalm Merge pull request #485 from microsoft/dependabot/pip/adalm/ urllib3-1... October 26, 2021 14:43 beats Update download links for BEATs March 1, 2023 16:16 beit Fix exception causes in checkpoint.py January 13, 2023 22:53 beit2 Update README.md November 3, 2022 14:26 beit3 B3 September 28, 2022 20:45 decoding update sphinx 2.0 to 3.0.4 December 16, 2022 18:27 deepnet Update README.md November 24, 2022 14:53 deltalm Update README.md November 2, 2021 18:11 dit Merge pull request #693 from microsoft/dependabot/pip/dit/ classificat... April 20, 2022 10:35 e5 Update README.md February 10, 2023 14:54 edgelm update sphinx 2.0 to 3.0.4 December 16, 2022 18:27 infoxlm update sphinx 2.0 to 3.0.4 December 16, 2022 18:27 layoutlm Bump pillow from 9.0.1 to 9.3.0 in /layoutlm/deprecated November 22, 2022 10:30 layoutlmft Pass explicit encoding when opening JSON file April 5, 2022 18:26 layoutlmv2 Update README.md November 2, 2022 21:47 layoutlmv3 Update README.md November 2, 2022 21:47 layoutreader Merge pull request #686 from renjithsasidharan/bugfix/ s2s_ft_use_cpu_... November 1, 2022 11:09 layoutxlm Update README.md November 2, 2022 21:48 markuplm Update README.md October 14, 2022 09:25 metalm Update README.md June 24, 2022 14:48 minilm Update README.md March 5, 2022 15:35 s2s-ft some fix for s2s-ft October 31, 2022 07:03 simlm Release simlm pre-training and fine-tuning code January 4, 2023 15:46 speechlm Update download links for SpeechLM March 1, 2023 16:35 speecht5 add speecht5 October 17, 2022 19:07 storage Create unilm-base-cased-vocab.txt December 16, 2019 15:27 trocr fix typo in trocr_models.py January 8, 2023 01:01 unilm-v1 update azure blob urls April 23, 2022 12:47 unilm Update README.md June 26, 2020 12:46 unimim fix link November 10, 2022 22:00 valle Add VALL-E January 6, 2023 09:33 vl-beit Update README.md June 3, 2022 13:53 vlmo merge vlmo December 25, 2022 21:48 wavlm Update download links for WavLM March 1, 2023 16:28 xdoc Update README.md November 16, 2022 21:45 xlmt Update README.md March 13, 2021 22:27 xmoe Update README.md November 24, 2022 14:52 xtune Fixed typo November 23, 2021 11:07 .gitignore Initial commit July 22, 2019 21:15 .gitmodules add speecht5 October 17, 2022 19:07 CODE_OF_CONDUCT.md Initial commit July 22, 2019 21:15 CONTRIBUTING.md Create CONTRIBUTING.md September 19, 2019 18:01 LICENSE init September 30, 2019 19:15 NOTICE.md init September 30, 2019 19:15 README.md Update README.md February 27, 2023 09:29 SECURITY.md Microsoft mandatory file May 22, 2022 12:12 View code [ ] aka.ms/nlpagi Hiring AGI Fundamentals TorchScale - Transformers at (any) Scale (repo) Foundation Models General-purpose Foundation Model Language & Multilingual Vision Speech Multimodal (X + Language) Toolkits Applications News Release License Contact Information README.md aka.ms/nlpagi Hiring We are hiring at all levels (including FTE researchers and interns)! If you are interested in working with us on Foundation Models (aka large-scale pre-trained models) and AGI, NLP, MT, Speech, Document AI and Multimodal AI, please send your resume to fuwei@microsoft.com. AGI Fundamentals TorchScale - Transformers at (any) Scale (repo) Fundamental research to improve modeling generality and capability, as well as training stability and efficiency for Transformers at any scale. Stability - DeepNet: scaling Transformers to 1,000 Layers and beyond Generality - Foundation Transformers (Magneto): towards true general-purpose modeling across tasks and modalities (including language, vision, speech, and multimodal) Capability - A Length-Extrapolatable Transformer Efficiency & Transferability - X-MoE: scalable & finetunable sparse Mixture-of-Experts (MoE) Foundation Models General-purpose Foundation Model MetaLM: Language Models are General-Purpose Interfaces The Big Convergence - Large-scale self-supervised pre-training across tasks (predictive and generative), languages (100+ languages), and modalities (language, image, audio, layout/format + language, vision + language, audio + language, etc.) Language & Multilingual UniLM: unified pre-training for language understanding and generation InfoXLM/XLM-E: multilingual/cross-lingual pre-trained models for 100+ languages DeltaLM/mT6: encoder-decoder pre-training for language generation and translation for 100+ languages MiniLM: small and fast pre-trained models for language understanding and generation AdaLM: domain, language, and task adaptation of pre-trained models EdgeLM(NEW): small pre-trained models on edge/client devices SimLM (NEW): large-scale pre-training for similarity matching E5 (NEW): text embeddings Vision BEiT/BEiT-2: generative self-supervised pre-training for vision / BERT Pre-Training of Image Transformers DiT (NEW): self-supervised pre-training for Document Image Transformers Speech WavLM: speech pre-training for full stack tasks VALL-E: a neural codec language model for TTS Multimodal (X + Language) LayoutLM/LayoutLMv2/LayoutLMv3: multimodal (text + layout/format + image) Document Foundation Model for Document AI (e.g. scanned documents, PDF, etc.) LayoutXLM: multimodal (text + layout/format + image) Document Foundation Model for multilingual Document AI MarkupLM: markup language model pre-training for visually-rich document understanding XDoc: unified pre-training for cross-format document understanding UniSpeech: unified pre-training for self-supervised learning and supervised learning for ASR UniSpeech-SAT: universal speech representation learning with speaker-aware pre-training SpeechT5: encoder-decoder pre-training for spoken language processing SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data VLMo: Unified vision-language pre-training VL-BEiT (NEW): Generative Vision-Language Pre-training - evolution of BEiT to multimodal BEiT-3 (NEW): a general-purpose multimodal foundation model, and a major milestone of The Big Convergence of Large-scale Pre-training Across Tasks, Languages, and Modalities. Toolkits s2s-ft: sequence-to-sequence fine-tuning toolkit Aggressive Decoding (NEW): lossless and efficient sequence-to-sequence decoding algorithm Applications TrOCR: transformer-based OCR w/ pre-trained models LayoutReader: pre-training of text and layout for reading order detection XLM-T: multilingual NMT w/ pretrained cross-lingual encoders News * January, 2023: VALL-E a language modeling approach for text to speech synthesis (TTS), which achieves state-of-the-art zero-shot TTS performance. See https://aka.ms/valle for demos of our work * November, 2022: TorchScale 0.1.1 was released! * November, 2022: TrOCR was accepted by AAAI 2023. * [Model Release] November, 2022: XDoc BASE models for cross-format document understanding. * [Model Release] September, 2022: TrOCR BASE and LARGE models for Scene Text Recognition (STR). * [Model Release] September, 2022: BEiT v2 code and pretrained models * August, 2022: BEiT-3- a general-purpose multimodal foundation model, which achieves state-of-the-art transfer performance on both vision and vision-language tasks * July, 2022: SimLM - Large-scale self-supervised pre-training for similarity matching * June, 2022: DiT and LayoutLMv3 were accepted by ACM Multimedia 2022. * June, 2022: MetaLM - Language models are general-purpose interfaces to foudation models (language/multilingual, vision, speech, and multimodal) * June, 2022: VL-BEiT - bidirectional multimodal Transformer learned from scratch with one unified pretraining task, one shared backbone, and one-stage training, supporting both vision and vision-language tasks. * [Model Release] June, 2022: LayoutLMv3 Chinese - Chinese version of LayoutLMv3 * [Code Release] May, 2022: Aggressive Decoding - Lossless Speedup for Seq2seq Generation * April, 2022: Transformers at Scale = DeepNet + X-MoE * [Model Release] April, 2022: LayoutLMv3 - Pre-training for Document AI with Unified Text and Image Masking * [Model Release] March, 2022: EdgeFormer - Parameter-efficient Transformer for On-device Seq2seq Generation * [Model Release] March, 2022: DiT - Self-supervised Document Image Transformer. Demos: Document Layout Analysis, Document Image Classification * January, 2022: BEiT was accepted by ICLR 2022 as Oral presentation (54 out of 3391). * [Model Release] December 16th, 2021: TrOCR small models for handwritten and printed texts, with 3x inference speedup. * November 24th, 2021: VLMo as the new SOTA on the VQA Challenge * November, 2021: Multilingual translation at scale: 10000 language pairs and beyond * [Model Release] November, 2021: MarkupLM - Pre-training for text and markup language (e.g. HTML/XML) * [Model Release] November, 2021: VLMo - Unified vision-language pre-training w/ BEiT * October, 2021: WavLM Large achieves state-of-the-art performance on the SUPERB benchmark * [Model Release] October, 2021: WavLM - Large-scale self-supervised pre-trained models for speech. * [Model Release] October 2021: TrOCR is on HuggingFace * September 28th, 2021: T-ULRv5 (aka XLM-E/InfoXLM) as the SOTA on the XTREME leaderboard. // Blog * [Model Release] September, 2021: LayoutLM-cased are on HuggingFace * [Model Release] September, 2021: TrOCR - Transformer-based OCR w/ pre-trained BEiT and RoBERTa models. * August 2021: LayoutLMv2 and LayoutXLM are on HuggingFace * [Model Release] August, 2021: LayoutReader - Built with LayoutLM to improve general reading order detection. * [Model Release] August, 2021: DeltaLM - Encoder-decoder pre-training for language generation and translation. * August 2021: BEiT is on HuggingFace * [Model Release] July, 2021: BEiT - Towards BERT moment for CV * [Model Release] June, 2021: LayoutLMv2, LayoutXLM, MiniLMv2, and AdaLM. * May, 2021: LayoutLMv2, InfoXLMv2, MiniLMv2, UniLMv3, and AdaLM were accepted by ACL 2021. * April, 2021: LayoutXLM is coming by extending the LayoutLM into multilingual support! A multilingual form understanding benchmark XFUND is also introduced, which includes forms with human labeled key-value pairs in 7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese). * March, 2021: InfoXLM was accepted by NAACL 2021. * December 29th, 2020: LayoutLMv2 is coming with the new SOTA on a wide varierty of document AI tasks, including DocVQA and SROIE leaderboard. * October 8th, 2020: T-ULRv2 (aka InfoXLM) as the SOTA on the XTREME leaderboard. // Blog * September, 2020: MiniLM was accepted by NeurIPS 2020. * July 16, 2020: InfoXLM (Multilingual UniLM) arXiv * June, 2020: UniLMv2 was accepted by ICML 2020; LayoutLM was accepted by KDD 2020. * April 5, 2020: Multilingual MiniLM released! * September, 2019: UniLMv1 was accepted by NeurIPS 2019. Release ***** New October, 2022: XDoc release ***** * [*] XDoc (October 7, 2022): XDoc, a unified pre-trained model which deals with different document formats in a single model. For parameter efficiency, we share backbone parameters for different formats such as the word embedding layer and the Transformer layers. Meanwhile, we introduce adaptive layers with lightweight parameters to enhance the distinction across different formats. Experimental results have demonstrated that with only 36.7% parameters, XDoc achieves comparable or even better performance on a variety of downstream tasks compared with the individual pre-trained models, which is cost effective for real-world deployment. "XDoc: Unified Pre-training for Cross-Format Document Understanding EMNLP 2022" ***** New May, 2022: Aggressive Decoding release ***** * [*] Aggressive Decoding (May 20, 2022): Aggressive Decoding, a novel decoding paradigm for lossless speedup for seq2seq generation. Unlike the previous efforts (e.g., non-autoregressive decoding) speeding up seq2seq generation at the cost of quality loss, Aggressive Decoding aims to yield the identical (or better) generation compared with autoregressive decoding but in a significant speedup: For the seq2seq tasks characterized by highly similar inputs and outputs (e.g., Grammatical Error Correction and Text Simplification), the Input-guided Aggressive Decoding can introduce a 7x-9x speedup for the popular 6-layer Transformer on GPU with the identical results as greedy decoding; For other general seq2seq tasks (e.g., Machine Translation and Abstractive Summarization), the Generalized Aggressive Decoding can have a 3x-5x speedup with the identical or even better quality. "Lossless Acceleration for Seq2seq Generation with Aggressive Decoding" ***** New April, 2022: LayoutLMv3 release ***** * [*] LayoutLM 3.0 (April 19, 2022): LayoutLMv3, a multimodal pre-trained Transformer for Document AI with unified text and image masking. Additionally, it is also pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in image-centric tasks such as document image classification and document layout analysis. " LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking ACM MM 2022" ***** March, 2022: EdgeFormer release ***** * [*] EdgeFormer (March 18, 2022): EdgeFormer, the first publicly available pretrained parameter-efficient Transformer for on-device seq2seq generation. EdgeFormer has only 11 million parameters, taking up less than 15MB disk size after int8 quantization and compression, which can process a sentence of the length of 20-30 tokens with acceptable latency on two middle-to-high end CPU cores and less than 50MB memory footprint. The pretrained EdgeFormer can be fine-tuned to English seq2seq tasks and achieve promising results -- significantly better than the strong paramter-efficient Transformer baseline (pretrained Universal Transformer) and full-parameterized Transformer-base model without pretraining, which we believe can largely facilitate on-device seq2seq generation in practice. "EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation" ***** March, 2022: DiT release ***** * [*] DiT (March 4, 2022): DiT, a self-supervised pre-trained Document Image Transformer model using large-scale unlabeled text images for Document AI tasks, which is essential since no supervised counterparts ever exist due to the lack of human labeled document images. We leverage DiT as the backbone network in a variety of vision-based Document AI tasks, including document image classification, document layout analysis, table detection as well as text detection for OCR. Experiment results have illustrated that the self-supervised pre-trained DiT model achieves new state-of-the-art results on these downstream tasks, e.g. document image classification (91.11 - 92.69), document layout analysis (91.0 - 94.9), table detection (94.23 - 96.55) and text detection for OCR (93.07 - 94.29). "DiT: Self-supervised Pre-training for Document Image Transformer ACM MM 2022" ***** October, 2021: WavLM release ***** * [*] WavLM (October 27, 2021): WavLM, a new pre-trained speech model, to solve full-stack downstream speech tasks. WavLM integrates the gated relative position embedding structure and the utterance mixing method, to model both spoken content and speaker identity preservation. WavLM is trained on 94k hours of public audio data, which is larger than other released checkpoints for English Speech modeling. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks. "WavLM: Large-Scale Self-Supervised Pre-training for Full Stack Speech Processing" ***** October, 2021: MarkupLM release ***** * [*] MarkupLM (October 19, 2021): MarkupLM, a simple yet effective pre-training approach for text and markup language. With the Transformer architecture, MarkupLM integrates different input embeddings including text embeddings, position embeddings, and XPath embeddings. Furthermore, we also propose new pre-training objectives that are specially designed for understanding the markup language. We evaluate the pre-trained MarkupLM model on the WebSRC and SWDE datasets. Experiments show that MarkupLM significantly outperforms several SOTA baselines in these tasks. "MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding ACL 2022" ***** September, 2021: TrOCR release ***** * [*] TrOCR (September 22, 2021): Transformer-based OCR with pre-trained models, which leverages the Transformer architecture for both image understanding and bpe-level text generation. The TrOCR model is simple but effective (convolution free), and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. "TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models AAAI 2023" ***** August, 2021: LayoutReader release ***** * [*] LayoutReader (August 26, 2021): pre-training of text and layout for reading order detection. The pre-trained LayoutReader significantly improves both open-source and commercial OCR engines in ordering text lines. Meanwhile, we also created a reading order benchmark dataset ReadingBank to further empower the research in this area. "LayoutReader: Pre-training of Text and Layout for Reading Order Detection EMNLP 2021" ***** August, 2021: DeltaLM release ***** * [*] DeltaLM (August, 2021): encoder-decoder pre-training for language generation and translation. DeltaLM ranks first on the WMT21 multilingual translation task. The task requires a model to translate between 102 languages. "DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders." ***** July, 2021: BEiT release ***** * [*] BEiT (June 15, 2021): BERT Pre-Training of Image Transformers. BEiT-large achieves state-of-the-art results on ADE20K (a big jump to 57.0 mIoU) for semantic segmentation. BEiT-large achieves state-of-the-art ImageNet top-1 accuracy (88.6%) under the setting without extra data other than ImageNet-22k. "BEiT: BERT Pre-Training of Image Transformers" ***** June, 2021: LayoutXLM | AdaLM | MiniLMv2 release ***** * [*] LayoutXLM (April 17, 2021): multimodal pre-training for multilingual visually-rich document understanding. The pre-trained LayoutXLM model has significantly outperformed the existing SOTA cross-lingual pre-trained models on the FUNSD and multilingual XFUND dataset including 7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese). " LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding" * [*] AdaLM (June 2021): a simple yet effective approach for domain adaptation of pre-trained models. Biomedical specific pre-trained models are released. "Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains ACL 2021" * [*] MiniLMv2 (December, 2020): a simple yet effective task-agnostic knoweldge distillation method, namely multi-head self-attention relation distillation, for compressing large pre-trained Transformers into small and fast pre-trained models. MiniLMv2 significantly outperforms MiniLMv1. Both English and multilingual MiniLM models are released. "MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers ACL 2021" ***** May, 2021: LayoutLMv2 | LayoutXLM release ***** * [*] LayoutLM 2.0 (December 29, 2020): multimodal pre-training for visually-rich document understanding by leveraging text, layout and image information in a single framework. It is coming with new SOTA on a wide range of document understanding tasks, including FUNSD (0.7895 -> 0.8420), CORD (0.9493 -> 0.9601), SROIE (0.9524 -> 0.9781), Kleister-NDA (0.834 -> 0.852), RVL-CDIP (0.9443 -> 0.9564), and DocVQA (0.7295 -> 0.8672). "LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding ACL 2021" ***** February, 2020: UniLM v2 | MiniLM v1 | LayoutLM v1 | s2s-ft v1 release ***** * [*] LayoutLM 1.0 (February 18, 2020): pre-trained models for document (image) understanding (e.g. receipts, forms, etc.) . It achieves new SOTA results in several downstream tasks, including form understanding (the FUNSD dataset from 70.72 to 79.27), receipt understanding (the ICDAR 2019 SROIE leaderboard from 94.02 to 95.24) and document image classification (the RVL-CDIP dataset from 93.07 to 94.42). "LayoutLM: Pre-training of Text and Layout for Document Image Understanding KDD 2020" * [*] s2s-ft 1.0 (February 26, 2020): A PyTorch package used to fine-tune pre-trained Transformers for sequence-to-sequence language generation. "s2s-ft: Fine-Tuning Pre-Trained Transformers for Sequence-to-Sequence Learning" * [*] MiniLM 1.0 (February 26, 2020): deep self-attention distillation is all you need (for task-agnostic knowledge distillation of pre-trained Transformers). MiniLM (12-layer, 384-hidden) achieves 2.7x speedup and comparable results over BERT-base (12-layer, 768-hidden) on NLU tasks as well as strong results on NLG tasks. The even smaller MiniLM (6-layer, 384-hidden) obtains 5.3x speedup and produces very competitive results. "MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers NeurIPS 2020" * [*] UniLM 2.0 (February 28, 2020): unified pre-training of bi-directional LM (via autoencoding) and sequence-to-sequence LM (via partially autoregressive) w/ Pseudo-Masked Language Model for language understanding and generation. UniLM v2 achieves new SOTA in a wide range of natural language understanding and generation tasks. "UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training ICML 2020" ***** October 1st, 2019: UniLM v1 release ***** * [*] UniLM v1 (September 30, 2019): the code and pre-trained models for the NeurIPS 2019 paper entitled "Unified Language Model Pre-training for Natural Language Understanding and Generation". UniLM (v1) achieves the new SOTA results in NLG (especially sequence-to-sequence generation) tasks, including abstractive summarization (the Gigaword and CNN/DM datasets), question generation (the SQuAD QG dataset), etc. License This project is licensed under the license found in the LICENSE file in the root directory of this source tree. Portions of the source code are based on the transformers project. Microsoft Open Source Code of Conduct Contact Information For help or issues using the pre-trained models, please submit a GitHub issue. For other communications, please contact Furu Wei (fuwei@microsoft.com). About Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities aka.ms/nlpagi Topics nlp language-generation deepnet multimodal pre-trained-model unilm minilm layoutlm small-pre-trained-model beit document-ai multimodal-pre-trained-model layoutxlm trocr foundation-models llm beit-3 xlm-e document-foundation-model mllm Resources Readme License MIT license Code of conduct Code of conduct Security policy Security policy Stars 9.6k stars Watchers 241 watching Forks 1.6k forks Releases 3 s2s-ft version 0.3 Latest Apr 2, 2020 + 2 releases Contributors 48 * @gitnlp * @wolfshow * @donglixp * @Dod-o * @MarkWuNLP * @ranpox * @addf400 * @lockon-n * @shumingma * @getao * @wenhui0924 + 37 contributors Languages * Python 96.4% * Shell 2.2% * Cuda 0.7% * C++ 0.4% * Cython 0.2% * Lua 0.1% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.