[HN Gopher] Show HN: LLM, a Rust Crate/CLI for CPU Inference of ...
       ___________________________________________________________________
        
       Show HN: LLM, a Rust Crate/CLI for CPU Inference of LLMs (LLaMA,
       GPT-NeoX, etc.)
        
       G'day, HN!  I'm one of the maintainers of `llm`. I've been working
       alongside a trusty group of contributors to bring this project to
       life, and we're now at a point where we're ready to share it with
       the world.  Large language models (LLMs) are taking the computing
       world by storm due to their emergent abilities that allow them to
       perform a wide variety of tasks, including translation,
       summarization, code generation, and even some degree of reasoning.
       However, the ecosystem around LLMs is still in its infancy, and it
       can be difficult to get started with these models.  `llm` is a one-
       stop shop for running inference on large language models (of the
       kind that power ChatGPT and more); we provide a CLI and a Rust
       crate for running inference on these models, all entirely open-
       source. The crate can be embedded in your own projects, allowing
       you to easily integrate LLMs into your own applications.  We hope
       that `llm` can help to alleviate some of the pain points that users
       face when working with LLMs. Our goal is to build a robust solution
       for inferencing on LLMs that users can rely on for their projects,
       so that we can provide a moment of peace in the chaos of the LLM
       ecosystem.  At present, we are powered by `ggml` (similar to
       `llama.cpp`), but we intend to add additional backends in the near-
       future. This means that we currently only support CPU inference,
       but we have several ideas in mind for how to add GPU support, as
       well as other accelerators.  We're looking for feedback on the
       project, and we'd love to hear from you! If you're interested in
       contributing, please reach out to us on our Discord
       (https://discord.gg/YB9WaXYAWU), or post an issue on our GitHub
       (https://github.com/rustformers/llm/issues).
        
       Author : Philpax
       Score  : 21 points
       Date   : 2023-05-09 16:37 UTC (6 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | simmons wrote:
       | Great work! I've been playing a little with an older version of
       | this (from ~ March 29th, when it was called `llama-rs`) just to
       | get a taste of the LLM world, and it's super convenient. (I
       | haven't tried llama.cpp to compare.)
       | 
       | Since it's a bit slow on my hardware (being CPU-only at the
       | moment + me not having any fancy graphics cards, anyway), when
       | toying around I sometimes get impatient and fire up jobs on
       | multiple servers via the CLI. I keep toying with the idea of
       | whipping up a webapp that would provide a ChatGPT-like interface
       | and dispatch to any number of connected backend servers.
        
         | Philpax wrote:
         | Thank you!
         | 
         | Yeah, I've been tempted to set up my own personal ML homelab,
         | but I haven't been able to justify it to myself. I've been
         | lucky to have a relatively powerful machine to work on :)
        
       | lxe wrote:
       | Oooh. Are there any performance gains over using ggml itself?
        
         | Philpax wrote:
         | Not at present, no (we use ggml directly at present), but our
         | plan is to build our own computation graph that can then be
         | executed by other backends (including Intel MKL or CUDA).
        
       ___________________________________________________________________
       (page generated 2023-05-09 23:01 UTC)