https://github.com/cocktailpeanut/dalai Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Blog * Solutions + For + Enterprise + Teams + Startups + Education + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} cocktailpeanut / dalai Public * Notifications * Fork 1k * Star 8.9k The simplest way to run LLaMA on your local machine cocktailpeanut.github.io/dalai 8.9k stars 1k forks Star Notifications * Code * Issues 228 * Pull requests 37 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights cocktailpeanut/dalai This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 1 tag Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/c] Use Git or checkout with SVN using the web URL. [gh repo clone cockta] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @cocktailpeanut cocktailpeanut Merge pull request #229 from metrafonic/main ... e4f678e Mar 22, 2023 Merge pull request #229 from metrafonic/main Added docker files (including docker compose) e4f678e Git stats * 161 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time bin cleanup March 22, 2023 14:05 cmds Cancel request March 22, 2023 05:16 demo Remove unused line of code with syntax error March 13, 2023 04:38 docs added docker files (including docker compose) March 22, 2023 21:58 .gitignore get rid of locks March 22, 2023 06:37 .npmrc enforce node version March 17, 2023 07:25 .prettierignore Add .prettierignore file to avoid auto-formatting files (and complica... March 13, 2023 04:43 .prettierrc.json Add .prettierignore file to avoid auto-formatting files (and complica... March 13, 2023 04:43 Dockerfile added docker files (including docker compose) March 22, 2023 21:58 alpaca.js download alpaca from huggingface + installer fix March 22, 2023 16:18 dalai Add ./dalai script which runs the cli script for test running, and fi... March 13, 2023 04:40 docker-compose.yml added docker files (including docker compose) March 22, 2023 21:58 index.js download alpaca from huggingface + installer fix March 22, 2023 16:18 llama.js Merge pull request #107 from matbee-eth/main March 22, 2023 01:27 package.json download alpaca from huggingface + installer fix March 22, 2023 16:18 setup.js setup postinstall script fix March 17, 2023 16:56 torrent.js Torrent downloader + alpaca.13B March 19, 2023 21:08 View code [ ] Dalai JUST RUN THIS TO GET Intro 1. Cross platform 2. Memory Requirements 3. Disk Space Requirements Alpaca 7B 13B LLaMA 7B 13B 30B 65B Quickstart Docker compose Mac Step 1. Install node.js >= 18 Step 2.1. Install models Add alpaca models Add llama models Step 2.2. Troubleshoot 1. Install homebrew 2. Install dependencies 3. Update NPM Step 3. Run Web UI Windows Step 1. Install Visual Studio Step 2.1. Install models Install alpaca Add llama models Step 2.2. Troubleshoot (optional) Step 3. Run Web UI Linux Step 1. Install Dependencies Step 1.1. Python <= 3.10 Step 1.2. Node.js >= 18 Step 2.1. Install models Add alpaca models Add llama models Step 2.2. Troubleshoot Step 3. Run Web UI API 1. constructor() Syntax Examples Basic Custom path 2. request() Syntax Examples 1. Node.js 2. Non node.js (socket.io) Step 1. start a server Step 2. connect to the server 3. serve() Syntax Examples 4. http() Syntax Examples 5. install() Syntax Examples 6. installed() Syntax Examples FAQ Using a different home folder 1. Installing models to a custom path 2. Serving from the custom path Updating to the latest Staying up to date README.md Dalai Run LLaMA and Alpaca on your computer. GitHub Twitter Discord --------------------------------------------------------------------- JUST RUN THIS [npx2] TO GET Both alpaca and llama working on your computer! alpaca.gif --------------------------------------------------------------------- 1. Powered by llama.cpp, llama-dl CDN, and alpaca.cpp 2. Hackable web app included 3. Ships with JavaScript API 4. Ships with Socket.io API --------------------------------------------------------------------- Intro 1. Cross platform Dalai runs on all of the following operating systems: 1. Linux 2. Mac 3. Windows 2. Memory Requirements Runs on most modern computers. Unless your computer is very very old, it should work. According to a llama.cpp discussion thread, here are the memory requirements: * 7B => ~4 GB * 13B => ~8 GB * 30B => ~16 GB * 65B => ~32 GB 3. Disk Space Requirements Alpaca Currently 7B and 13B models are available via alpaca.cpp 7B Alpaca comes fully quantized (compressed), and the only space you need for the 7B model is 4.21GB: alpaca_7b.png 13B Alpaca comes fully quantized (compressed), and the only space you need for the 13B model is 8.14GB: alpaca_13b.png LLaMA You need a lot of space for storing the models. The model name must be one of: 7B, 13B, 30B, and 65B. You do NOT have to install all models, you can install one by one. Let's take a look at how much space each model takes up: NOTE The following numbers assume that you DO NOT touch the original model files and keep BOTH the original model files AND the quantized versions. You can optimize this if you delete the original models (which are much larger) after installation and keep only the quantized versions. 7B * Full: The model takes up 31.17GB * Quantized: 4.21GB 7b.png 13B * Full: The model takes up 60.21GB * Quantized: 4.07GB * 2 = 8.14GB 13b.png 30B * Full: The model takes up 150.48GB * Quantized: 5.09GB * 4 = 20.36GB 30b.png 65B * Full: The model takes up 432.64GB * Quantized: 5.11GB * 8 = 40.88GB 65b.png --------------------------------------------------------------------- Quickstart Docker compose Requires that you have docker installed and running. docker compose build docker compose run dalai npx dalai alpaca install 7B # or a different model docker compose up -d This will dave the models in the ./models folder View the site at http://127.0.0.1:3000/ Mac Step 1. Install node.js >= 18 If your mac doesn't have node.js installed yet, make sure to install node.js >= 18 Install Node.js Step 2.1. Install models Currently supported engines are llama and alpaca. Add alpaca models To download alpaca models, you can run: npx dalai alpaca install 7B Add llama models To download llama models, you can run: npx dalai llama install 7B or to download multiple models: npx dalai llama install 7B 13B Now go to step 3. Step 2.2. Troubleshoot Normally you don't need this step, but if running the commands above don't do anything and immediately end, it means something went wrong because some of the required modules are not installed on your system. In that case, try the following steps: 1. Install homebrew In case homebrew is not installed on your computer, install it by running: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" Or you can find the same instruction on the homebrew hompage: https://brew.sh/ 2. Install dependencies Once homebrew is installed, install these dependencies: brew install cmake brew install pkg-config 3. Update NPM Just to make sure we cover every vector, let's update NPM as well: npm install -g npm@latest Now go back to step 2.1 and try running the npx dalai commands again. Step 3. Run Web UI After everything has been installed, run the following command to launch the web UI server: npx dalai serve and open http://localhost:3000 in your browser. Have fun! --------------------------------------------------------------------- Windows Step 1. Install Visual Studio On windows, you need to install Visual Studio before installing Dalai. Press the button below to visit the Visual Studio downloads page and download: Download Microsoft Visual Studio IMPORTANT!!! When installing Visual Studio, make sure to check the 3 options as highlighted below: 1. Python development 2. Node.js development 3. Desktop development with C++ vs.png --------------------------------------------------------------------- Step 2.1. Install models IMPORTANT On Windows, make sure to run all commands in cmd. DO NOT run in powershell. Powershell has unnecessarily strict permissions and makes the script fail silently. Currently supported engines are llama and alpaca. Install alpaca To download alpaca models. Open your cmd application and enter: npx dalai alpaca install 7B Add llama models To download llama models. Open your cmd application and enter: npx dalai llama install 7B or to download multiple models: npx dalai llama install 7B 13B --------------------------------------------------------------------- Step 2.2. Troubleshoot (optional) In case above steps fail, try installing Node.js and Python separately. Install Python: Download Python Install Node.js >= 18: Download Node.js After both have been installed, open powershell and type python to see if the application exists. And also type node to see if the application exists as well. Once you've checked that they both exist, try again. Step 3. Run Web UI After everything has been installed, run the following command to launch the web UI server (Make sure to run in cmd and not powershell!): npx dalai serve and open http://localhost:3000 in your browser. Have fun! --------------------------------------------------------------------- Linux Step 1. Install Dependencies You need to make sure you have the correct version of Python and Node.js installed. Step 1.1. Python <= 3.10 Download Python Make sure the version is 3.10 or lower (not 3.11) Python must be 3.10 or below (pytorch and other libraries are not supported yet on the latest) Step 1.2. Node.js >= 18 Download node.js Make sure the version is 18 or higher --------------------------------------------------------------------- Step 2.1. Install models Currently supported engines are llama and alpaca. Add alpaca models To download alpaca models, you can run: npx dalai alpaca install 7B Add llama models To download llama models, you can run: npx dalai llama install 7B or to download multiple models: npx dalai llama install 7B 13B Step 2.2. Troubleshoot In case the model install silently fails or hangs forever, try the following command, and try running the npx command again: On ubuntu/debian/etc.: sudo apt-get install build-essential python3-venv -y On fedora/etc.: dnf install make automake gcc gcc-c++ kernel-devel python3-virtualenv -y Step 3. Run Web UI After everything has been installed, run the following command to launch the web UI server: npx dalai serve and open http://localhost:3000 in your browser. Have fun! --------------------------------------------------------------------- API Dalai is also an NPM package: 1. programmatically install 2. locally make requests to the model 3. run a dalai server (powered by socket.io) 4. programmatically make requests to a remote dalai server (via socket.io) Dalai is an NPM package. You can install it using: npm install dalai --------------------------------------------------------------------- 1. constructor() Syntax const dalai = new Dalai(home) * home: (optional) manually specify the llama.cpp folder By default, Dalai automatically stores the entire llama.cpp repository under ~/llama.cpp. However, often you may already have a llama.cpp repository somewhere else on your machine and want to just use that folder. In this case you can pass in the home attribute. Examples Basic Creates a workspace at ~/llama.cpp const dalai = new Dalai() Custom path Manually set the llama.cpp path: const dalai = new Dalai("/Documents/llama.cpp") --------------------------------------------------------------------- 2. request() Syntax dalai.request(req, callback) * req: a request object. made up of the following attributes: + prompt: (required) The prompt string + model: (required) The model type + model name to query. Takes the following form: . o Example: alpaca.7B, llama.13B, ... + url: only needed if connecting to a remote dalai server o if unspecified, it uses the node.js API to directly run dalai locally o if specified (for example ws://localhost:3000) it looks for a socket.io endpoint at the URL and connects to it. + threads: The number of threads to use (The default is 8 if unspecified) + n_predict: The number of tokens to return (The default is 128 if unspecified) + seed: The seed. The default is -1 (none) + top_k + top_p + repeat_last_n + repeat_penalty + temp: temperature + batch_size: batch size + skip_end: by default, every session ends with \n\n, which can be used as a marker to know when the full response has returned. However sometimes you may not want this suffix. Set skip_end: true and the response will no longer end with \ n\n * callback: the streaming callback function that gets called every time the client gets any token response back from the model Examples 1. Node.js Using node.js, you just need to initialize a Dalai object with new Dalai() and then use it. const Dalai = require('dalai') new Dalai().request({ model: "7B", prompt: "The following is a conversation between a boy and a girl:", }, (token) => { process.stdout.write(token) }) 2. Non node.js (socket.io) To make use of this in a browser or any other language, you can use thie socket.io API. Step 1. start a server First you need to run a Dalai socket server: // server.js const Dalai = require('dalai') new Dalai().serve(3000) // port 3000 Step 2. connect to the server Then once the server is running, simply make requests to it by passing the ws://localhost:3000 socket url when initializing the Dalai object: const Dalai = require("dalai") new Dalai().request({ url: "ws://localhost:3000", model: "7B", prompt: "The following is a conversation between a boy and a girl:", }, (token) => { console.log("token", token) }) --------------------------------------------------------------------- 3. serve() Syntax Starts a socket.io server at port dalai.serve(port) Examples const Dalai = require("dalai") new Dalai().serve(3000) --------------------------------------------------------------------- 4. http() Syntax connect with an existing http instance (The http npm package) dalai.http(http) * http: The http object Examples This is useful when you're trying to plug dalai into an existing node.js web app const app = require('express')(); const http = require('http').Server(app); dalai.http(http) http.listen(3000, () => { console.log("server started") }) 5. install() Syntax await dalai.install(model_type, model_name1, model_name2, ...) * model_type: the name of the model. currently supports: + "alpaca" + "llama" * model1, model2, ...: the model names to install ("7B"`, "13B", "30B", "65B", etc) Examples Install Llama "7B" and "13B" models: const Dalai = require("dalai"); const dalai = new Dalai() await dalai.install("llama", "7B", "13B") Install alpaca 7B model: const Dalai = require("dalai"); const dalai = new Dalai() await dalai.install("alpaca", "7B") --------------------------------------------------------------------- 6. installed() returns the array of installed models Syntax const models = await dalai.installed() Examples const Dalai = require("dalai"); const dalai = new Dalai() const models = await dalai.installed() console.log(models) // prints ["7B", "13B"] --------------------------------------------------------------------- FAQ Using a different home folder By default Dalai uses your home directory to store the entire repository (~/dalai). However sometimes you may want to store the archive elsewhere. In this case you can call all CLI methods using the --home flag: 1. Installing models to a custom path npx dalai llama install 7B --home ~/test_dir 2. Serving from the custom path npx dalai serve --home ~/test_dir Updating to the latest To make sure you update to the latest, first find the latest version at https://www.npmjs.com/package/dalai Let's say the latest version is 0.3.0. To update the dalai version, run: npx dalai@0.3.0 setup Staying up to date Have questions or feedback? Follow the project through the following outlets: GitHub Twitter Discord --------------------------------------------------------------------- About The simplest way to run LLaMA on your local machine cocktailpeanut.github.io/dalai Topics ai llama llm Resources Readme Stars 8.9k stars Watchers 104 watching Forks 1k forks Report repository Releases 1 0.1.0 Latest Mar 13, 2023 Packages 0 No packages published Used by 10 * @shubham8550 * @jonfleming * @khanh-nguyen-code * @ykachube * @LaraGuardiola * @hvbr1s * @TobWul * @ShounakA + 2 Contributors 19 * @cocktailpeanut * @marcuswestin * @keldenl * @matbee-eth * @ItsPi3141 * @dev-jelly * @EliasVincent * @francip * @kou029w * @metrafonic * @rizenfrmtheashes + 8 contributors Languages * CSS 81.2% * JavaScript 13.6% * EJS 5.0% * Other 0.2% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.