https://github.com/bhavnicksm/chonkie Skip to content Navigation Menu Toggle navigation Sign in * Product + GitHub Copilot Write better code with AI + Security Find and fix vulnerabilities + Actions Automate any workflow + Codespaces Instant dev environments + Issues Plan and track work + Code Review Manage code changes + Discussions Collaborate outside of code + Code Search Find more, search less Explore + All features + Documentation + GitHub Skills + Blog * Solutions By company size + Enterprises + Small and medium teams + Startups By use case + DevSecOps + DevOps + CI/CD + View all use cases By industry + Healthcare + Financial services + Manufacturing + Government + View all industries View all solutions * Resources Topics + AI + DevOps + Security + Software Development + View all Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Reseting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} bhavnicksm / chonkie Public * Notifications You must be signed in to change notification settings * Fork 7 * Star 299 CHONK your texts with Chonkie - The no-nonsense RAG chunking library pypi.org/project/chonkie/ License MIT license 299 stars 7 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 2 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights bhavnicksm/chonkie main BranchesTags [ ] Go to file Code Folders and files Name Name Last commit Last commit message date Latest commit History 102 Commits .github/workflows .github/ workflows assets assets benchmarks benchmarks src/chonkie src/chonkie tests/chunker tests/chunker .gitignore .gitignore DOCS.md DOCS.md LICENSE LICENSE README.md README.md pyproject.toml pyproject.toml View all files Repository files navigation * README * MIT license Chonkie Logo Chonkie PyPI version License Documentation Package size Downloads GitHub stars The no-nonsense RAG chunking library that's lightweight, lightning-fast, and ready to CHONK your texts Installation * Usage * Supported Methods * Benchmarks * Acknowledgements * Citation so i found myself making another RAG bot (for the 2342148th time) and meanwhile, explaining to my juniors about why we should use chunking in our RAG bots, only to realise that i would have to write chunking all over again unless i use the bloated software library X or the extremely feature-less library Y. WHY CAN I NOT HAVE SOMETHING JUST RIGHT, UGH? Can't i just install, import and run chunking and not have to worry about dependencies, bloat, speed or other factors? Well, with chonkie you can! (chonkie boi is a gud boi) Feature-rich: All the CHONKs you'd ever need Easy to use: Install, Import, CHONK [?] Fast: CHONK at the speed of light! zooooom Wide support: Supports all your favorite tokenizer CHONKS Light-weight: No bloat, just CHONK Cute CHONK mascot: psst it's a pygmy hippo btw [?] Moto Moto's favorite python library What're you waiting for, just CHONK it! Installation To install chonkie, simply run: pip install chonkie Chonkie follows the rule to have minimal defualt installs, read the DOCS to know the installation for your required chunker, or simply install all if you don't want to think about it (not recommended). pip install chonkie[all] Usage Here's a basic example to get you started: # First import the chunker you want from Chonkie from chonkie import TokenChunker # Import your favorite tokenizer library # Also supports AutoTokenizers, TikToken and AutoTikTokenizer from tokenizers import Tokenizer tokenizer = Tokenizer.from_pretrained("gpt2") # Initialize the chunker chunker = TokenChunker(tokenizer) # Chunk some text chunks = chunker("Woah! Chonkie, the chunking library is so cool! I love the tiny hippo hehe.") # Access chunks for chunk in chunks: print(f"Chunk: {chunk.text}") print(f"Tokens: {chunk.token_count}") More example usages given inside the DOCS Supported Methods Chonkie provides several chunkers to help you split your text efficiently for RAG applications. Here's a quick overview of the available chunkers: * TokenChunker: Splits text into fixed-size token chunks. * WordChunker: Splits text into chunks based on words. * SentenceChunker: Splits text into chunks based on sentences. * SemanticChunker: Splits text into chunks based on semantic similarity. * SDPMChunker: Splits text using a Semantic Double-Pass Merge approach. More on these methods and the approaches taken inside the DOCS Benchmarks [?] "I may be smol hippo, but I pack a punch!" Here's a quick peek at how Chonkie performs: Size * Default Install: 21MB (vs 80-171MB for alternatives) * With Semantic: Still lighter than the competition! Speed[?] * Token Chunking: 33x faster than the slowest alternative * Sentence Chunking: Almost 2x faster than competitors * Semantic Chunking: Up to 2.5x faster than others Check out our detailed benchmarks to see how Chonkie races past the competition! [?] Acknowledgements Chonkie would like to CHONK its way through a special thanks to all the users and contributors who have helped make this library what it is today! Your feedback, issue reports, and improvements have helped make Chonkie the CHONKIEST it can be. And of course, special thanks to Moto Moto for endorsing Chonkie with his famous quote: "I like them big, I like them chonkie." ~ Moto Moto Citation If you use Chonkie in your research, please cite it as follows: @misc{chonkie2024, author = {Minhas, Bhavnick}, title = {Chonkie: A Fast Feature-full Chunking Library for RAG Bots}, year = {2024}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/bhavnick/chonkie}}, } About CHONK your texts with Chonkie - The no-nonsense RAG chunking library pypi.org/project/chonkie/ Topics ai text-processing chunking rag retrieval-augmented-generation Resources Readme License MIT license Activity Stars 299 stars Watchers 3 watching Forks 7 forks Report repository Releases 8 v0.1.2 Latest Nov 8, 2024 + 7 releases Packages 0 No packages published Languages * Python 100.0% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.