https://github.com/kvcache-ai/Mooncake

Skip to content

Navigation Menu

Toggle navigation
 
Sign in

  * Product
      +  
        Actions
        Automate any workflow
      +  
        Packages
        Host and manage packages
      +  
        Security
        Find and fix vulnerabilities
      +  
        Codespaces
        Instant dev environments
      +  
        GitHub Copilot
        Write better code with AI
      +  
        Code review
        Manage code changes
      +  
        Issues
        Plan and track work
      +  
        Discussions
        Collaborate outside of code
    Explore
      + All features
      + Documentation
      + GitHub Skills
      + Blog
  * Solutions
    By size
      + Enterprise
      + Teams
      + Startups
    By industry
      + Healthcare
      + Financial services
      + Manufacturing
    By use case
      + CI/CD & Automation
      + DevOps
      + DevSecOps
  * Resources
    Resources
      + Learning Pathways
      + White papers, Ebooks, Webinars
      + Customer Stories
      + Partners
  * Open Source
      +  
        GitHub Sponsors
        Fund open source developers
      +  
        The ReadME Project
        GitHub community articles
    Repositories
      + Topics
      + Trending
      + Collections
  * Enterprise
      +  
        Enterprise platform
        AI-powered developer platform
    Available add-ons
      +  
        Advanced Security
        Enterprise-grade security features
      +  
        GitHub Copilot
        Enterprise-grade AI features
      +  
        Premium Support
        Enterprise-grade 24/7 support
  * Pricing

Search or jump to...

Search code, repositories, users, issues, pull requests...

Search
[                    ]
Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

[                    ] [ ] Include my email address so I can be
contacted
Cancel Submit feedback

Saved searches

Use saved searches to filter your results more quickly

Name [                    ] 
Query [                    ]

To see all available qualifiers, see our documentation.

Cancel Create saved search
Sign in
Sign up
You signed in with another tab or window. Reload to refresh your
session. You signed out in another tab or window. Reload to refresh
your session. You switched accounts on another tab or window. Reload
to refresh your session. Dismiss alert
{{ message }}
kvcache-ai / Mooncake Public

  * Notifications You must be signed in to change notification
    settings
  * Fork 5
  * Star 239

239 stars 5 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

  * Code
  * Issues 1
  * Pull requests 0
  * Actions
  * Projects 0
  * Security
  * Insights

Additional navigation options

  * Code
  * Issues
  * Pull requests
  * Actions
  * Projects
  * Security
  * Insights

kvcache-ai/Mooncake

This commit does not belong to any branch on this repository, and may
belong to a fork outside of the repository.
 main
BranchesTags
  
Go to file
Code

Folders and files

     Name            Name       Last commit message Last commit date
Latest commit

 

History

2 Commits
 
image           image                                
Mooncake-v1.pdf Mooncake-v1.pdf                      
README.md       README.md                            
View all files

Repository files navigation

  * README

              Mooncake: A KVCache-centric Disaggregated
                    Architecture for LLM Serving

 
 Technical Report


Mooncake is the serving platform for icon Kimi, a leading LLM service
provided by icon Moonshot AI. This repository hosts its technical
report and will also be utilized for the forthcoming open sourcing of
traces. Stay tuned!

 Updates

 

  * June 27, 2024: We present a Chinese blog with more discussions on
    zhihu.
  * June 26, 2024: Initial technical report release.

 Overview

 

Mooncake features a KVCache-centric disaggregated architecture that
separates the prefill and decoding clusters. It also leverages the
underutilized CPU, DRAM, and SSD resources of the GPU cluster to
implement a disaggregated cache of KVCache.

architecture

The core of Mooncake is its KVCache-centric scheduler, which balances
maximizing overall effective throughput while meeting latency-related
Service Level Objectives (SLOs) requirements. Unlike traditional
studies that assume all requests will be processed, Mooncake faces
challenges due to highly overloaded scenarios. To mitigate these, we
developed a prediction-based early rejection policy. Experiments show
that Mooncake excels in long-context scenarios. Compared to the
baseline method, Mooncake can achieve up to a 525% increase in
throughput in certain simulated scenarios while adhering to SLOs.
Under real workloads, Mooncake's innovative architecture enables Kimi
to handle 75% more requests.

 Citation

 
Please kindly cite our paper if you find it is useful:

@article{qin2024mooncake,
  title        = {Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving},
  author       = {Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu},
  year         = {2024}
}

Remark: arXiv version is still on holding.

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

239 stars

Watchers

7 watching

Forks

5 forks
Report repository

Releases

No releases published

Packages 0

No packages published

Footer

 (c) 2024 GitHub, Inc.

Footer navigation

  * Terms
  * Privacy
  * Security
  * Status
  * Docs
  * Contact
  * Manage cookies
  * Do not share my personal information

You can't perform that action at this time.