https://github.com/ggerganov/llama.cpp/discussions/1220 Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} ggerganov / llama.cpp Public * Notifications * Fork 3.6k * Star 25.1k * Code * Issues 196 * Pull requests 42 * Discussions * Actions * Projects 3 * Wiki * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Wiki * Security * Insights Roadmap May 2023 #1220 ggerganov announced in Announcements Roadmap May 2023 #1220 @ggerganov ggerganov Apr 28, 2023 * 1 comments * 3 replies Return to top Discussion options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. [199] ggerganov Apr 28, 2023 Maintainer - High-prio * [ ] Refactoring pass There is a lot of code duplication in ggml.c which probably can be simplified with a good set of macros. The goal is to keep the code size manageable, while we avoid reaching "macro hell" * [ ] Optimize the AVX / AVX2 implementations of the quantization methods and add WASM SIMD Make sure we have optimal implementation for these instruction sets * [*] Apply the new integer quantization methods to whisper.cpp Will backport the latest ggml version to whisper.cpp and add support for quantized models. Will also update all WASM examples to be able to run with the quantized models Update: whisper.cpp v1.4.0 has been released. It includes integer quantization and GPU support via cuBLAS - all thanks to the great work done here * [ ] Add support for "batch inference" Recently, the bert.cpp (by @skeskinen) project demonstrated BERT inference using ggml. This model gains a lot from batch inference, which is currently not supported by ggml. We will extend all operators to support it. The bert.cpp example will serve as a playground to achieve this * [ ] Implement inference of new models There are already some very interesting models that should be supported by ggml: + [ ] Segment Anything Model (SAM) + [ ] Bark (text-to-speech) There is a huge interest for adding ggml support for this model (see speeding up inference suno-ai/bark#30 (comment)) The main blocker seems to be the dependency on Facebook's EnCodec codec. Still not sure how difficult it would be, but probably this codec is another model that we should try to support via ggml I'll use this section to add a note regarding new model implementations by contributors - I recommend to always try to add a very basic example implementation to the ggml repo. Having a basic example there would make long-term support much easier * [ ] Proof-of-concept for GPU inference The goal is to make a demonstration of the idea discussed in Add GPU support to ggml #914 Very preliminary work has been started in ggml : cgraph export brainstorming ggml#108 Will try to get a working example using the MNIST inference Low-prio * [ ] Project ggml : improve threading implementation Better utilization of the available CPU resources via improved thread management. There have been a few efforts during April, but they remained in the "background" - need to put more focus this time * [ ] Having second thoughts about adding llama_state The experience is we added whisper_state in whisper.cpp with this PR: Added whisper state + default state on the whisper_context whisper.cpp#523 However, I haven't see a lot of use of it. At the same time, it doubled the C API. Let me know if you think this is worth implementing ref: IMPORTANT: Introduce C-style API - Major Refactoring #370 (comment) * [ ] Add 3-bit integer quantization It has been shown that 2-bit integer quantization does not look really useful for anything: Q2 and Q3 quantization #1004 In this case, we can probably add 3-bit integer quantization. Probably not "officially supported", but rather in a state where we can run experiments and see if we can find some application * [ ] There is an interesting ongoing effort to add "training" support to ggml: How to fine tune it? ggml#8 (comment) It would be really impressive if this actually works. Might be conflicts with the refactoring pass - need to coordinate with @xaedes Beta Was this translation helpful? Give feedback. 13 You must be logged in to vote [?] 18 13 All reactions * [?] 18 * 13 Replies: 1 comment * 3 replies * Oldest * Newest * Top Comment options * {{title}} Something went wrong. Quote reply [448] dakennedyd Apr 29, 2023 - Would be great if Stable Diffusion support could be added too. Beta Was this translation helpful? Give feedback. 6 You must be logged in to vote All reactions 3 replies @verhovsky Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. verhovsky Apr 29, 2023 - What do you mean? Beta Was this translation helpful? Give feedback. 3 All reactions * 3 @sroussey Comment options * {{title}} Something went wrong. Quote reply sroussey Apr 30, 2023 - Likely meaning stablelm Beta Was this translation helpful? Give feedback. All reactions @Green-Sky Comment options * {{title}} Something went wrong. Quote reply Green-Sky Apr 30, 2023 Collaborator - no stablelm is supported (by ggml, see repo) but its quality is underwhelming.... still, latent diffusion models would be sick Beta Was this translation helpful? Give feedback. All reactions Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment Category Announcements Labels None yet 5 participants @ggerganov @sroussey @Green-Sky @verhovsky @dakennedyd Add heading text Add bold text, Add italic text, Add a quote, Add code, Insert Link Link Text [ ] URL [ ] Add Add a link, Add a bulleted list, Add a numbered list, Add a task list, Directly mention a user or team Reference an issue or pull request Add heading text Add bold text, Add italic text, Add a bulleted list, Add a numbered list, Add a task list, 1 reacted with thumbs up emoji 1 reacted with thumbs down emoji 1 reacted with laugh emoji 1 reacted with hooray emoji 1 reacted with confused emoji [?] 1 reacted with heart emoji 1 reacted with rocket emoji 1 reacted with eyes emoji Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.