https://github.com/bentoml/OpenLLM Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. {{ message }} bentoml / OpenLLM Public * Notifications * Fork 56 * Star 2.5k An open platform for operating large language models (LLMs) in production. Fine-tune, serve, deploy, and monitor any LLMs with ease. bentoml.com License Apache-2.0 license 2.5k stars 56 forks Star Notifications * Code * Issues 7 * Pull requests 1 * Discussions * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Security * Insights bentoml/OpenLLM This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 2 branches 39 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/b] Use Git or checkout with SVN using the web URL. [gh repo clone bentom] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @aarnphm aarnphm fix(log): repr the given LLMConfig in debug mode ... d33149d Jun 19, 2023 fix(log): repr the given LLMConfig in debug mode Signed-off-by: aarnphm-ec2-dev <29749331+aarnphm@users.noreply.github.com> d33149d Git stats * 327 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github fix(ci): install towncrier for changelog automation June 19, 2023 06:32 assets chore: add banner for OpenLLM June 18, 2023 05:55 changelog.d fix: agent log (#37) June 19, 2023 14:11 examples fix(tools): adhere to style guidelines (#31) June 18, 2023 20:03 src fix(log): repr the given LLMConfig in debug mode June 19, 2023 18:20 tests fix(tools): adhere to style guidelines (#31) June 18, 2023 20:03 tools fix: remove direct url reference June 19, 2023 13:25 typings fix(log): make sure to configure OpenLLM logs correctly June 19, 2023 06:19 .gitignore feat: fine-tuning [part 1] (#23) June 16, 2023 00:19 .pre-commit-config.yaml fix: remove direct url reference June 19, 2023 13:25 ADDING_NEW_MODEL.md feat(tooling): add script to auto update readme table of supported June 8, 2023 08:22 CHANGELOG.md infra: prepare for release 0.1.8 [generated] June 19, 2023 18:02 DEVELOPMENT.md fix(tools): adhere to style guidelines (#31) June 18, 2023 20:03 LICENSE.md chore: rename LICENSE to LICENSE.md June 10, 2023 00:21 README.md docs: update README to use OPT as example June 19, 2023 13:40 hatch.toml fix: separate hatch config June 19, 2023 03:39 nightly-requirements.generated.txt fix: remove direct url reference June 19, 2023 13:25 package.json infra: bump to dev version of 0.1.9.dev0 [generated] June 19, 2023 18:08 pyproject.toml fix: remove direct url reference June 19, 2023 13:25 taplo.toml infra: docs and normalize formatting May 28, 2023 15:00 View code [ ] OpenLLM Introduction Getting Started Starting an LLM Server Supported Models Runtime Implementations (Experimental) Integrating a New Model [?][?] Integrations BentoML HuggingFace Agents LangChain ([?]Coming Soon!) Deploying to Production Telemetry Community Contributing README.md Banner for OpenLLM OpenLLM pypi_status ci Twitter Discord An open platform for operating large language models (LLMs) in production. Fine-tune, serve, deploy, and monitor any LLMs with ease. Introduction With OpenLLM, you can run inference with any open-source large-language models, deploy to the cloud or on-premises, and build powerful AI apps. State-of-the-art LLMs: built-in supports a wide range of open-source LLMs and model runtime, including StableLM, Falcon, Dolly, Flan-T5, ChatGLM, StarCoder and more. Flexible APIs: serve LLMs over RESTful API or gRPC with one command, query via WebUI, CLI, our Python/Javascript client, or any HTTP client. [?][?] Freedom To Build: First-class support for LangChain, BentoML and HuggingFace that allows you to easily create your own AI apps by composing LLMs with other models and services. Streamline Deployment: Automatically generate your LLM server Docker Images or deploy as serverless endpoint via [?] BentoCloud. [?] Bring your own LLM: Fine-tune any LLM to suit your needs with LLM.tuning(). (Coming soon) Gif showing OpenLLM Intro Getting Started To use OpenLLM, you need to have Python 3.8 (or newer) and pip installed on your system. We highly recommend using a Virtual Environment to prevent package conflicts. You can install OpenLLM using pip as follows: pip install openllm To verify if it's installed correctly, run: $ openllm -h Usage: openllm [OPTIONS] COMMAND [ARGS]... ######+ ######+ #######+###+ ##+##+ ##+ ###+ ###+ ##+---##+##+--##+##+----+####+ ##|##| ##| ####+ ####| ##| ##|######++#####+ ##+##+ ##|##| ##| ##+####+##| ##| ##|##+---+ ##+--+ ##|+##+##|##| ##| ##|+##++##| +######++##| #######+##| +####|#######+#######+##| +-+ ##| +-----+ +-+ +------++-+ +---++------++------++-+ +-+ An open platform for operating large language models in production. Fine-tune, serve, deploy, and monitor any LLMs with ease. Starting an LLM Server To start an LLM server, use openllm start. For example, to start a OPT server, do the following: openllm start opt Following this, a Web UI will be accessible at http://localhost:3000 where you can experiment with the endpoints and sample input prompts. OpenLLM provides a built-in Python client, allowing you to interact with the model. In a different terminal window or a Jupyter notebook, create a client to start interacting with the model: >>> import openllm >>> client = openllm.client.HTTPClient('http://localhost:3000') >>> client.query('Explain to me the difference between "further" and "farther"') You can also use the openllm query command to query the model from the terminal: export OPENLLM_ENDPOINT=http://localhost:3000 openllm query 'Explain to me the difference between "further" and "farther"' Visit http://localhost:3000/docs.json for OpenLLM's API specification. Users can also specify different variants of the model to be served, by providing the --model-id argument, e.g.: openllm start flan-t5 --model-id google/flan-t5-large Use the openllm models command to see the list of models and their variants supported in OpenLLM. Supported Models The following models are currently supported in OpenLLM. By default, OpenLLM doesn't include dependencies to run all models. The extra model-specific dependencies can be installed with the instructions below: Model CPU GPU Installation Model Ids * google/flan-t5-small * google/flan-t5-base flan-t5 pip install "openllm[flan-t5]" * google/flan-t5-large * google/flan-t5-xl * google/flan-t5-xxl * databricks/dolly-v2-3b dolly-v2 pip install openllm * databricks/dolly-v2-7b * databricks/dolly-v2-12b * thudm/chatglm-6b chatglm pip install "openllm[chatglm]" * thudm/chatglm-6b-int8 * thudm/chatglm-6b-int4 pip install "openllm[starcoder]" * bigcode/starcoder starcoder * bigcode/starcoderbase * tiiuae/falcon-7b * tiiuae/falcon-40b pip install "openllm[falcon]" * tiiuae/ falcon falcon-7b-instruct * tiiuae/ falcon-40b-instruct * stabilityai/ stablelm-tuned-alpha-3b * stabilityai/ pip install openllm stablelm-tuned-alpha-7b stablelm * stabilityai/ stablelm-base-alpha-3b * stabilityai/ stablelm-base-alpha-7b * facebook/opt-125m * facebook/opt-350m opt pip install openllm * facebook/opt-1.3b * facebook/opt-2.7b * facebook/opt-6.7b * facebook/opt-66b Runtime Implementations (Experimental) Different LLMs may have multiple runtime implementations. For instance, they might use Pytorch (pt), Tensorflow (tf), or Flax (flax). If you wish to specify a particular runtime for a model, you can do so by setting the OPENLLM_{MODEL_NAME}_FRAMEWORK={runtime} environment variable before running openllm start. For example, if you want to use the Tensorflow (tf) implementation for the flan-t5 model, you can use the following command: OPENLLM_FLAN_T5_FRAMEWORK=tf openllm start flan-t5 Note For GPU support on Flax, refers to Jax's installation to make sure that you have Jax support for the corresponding CUDA version. Integrating a New Model OpenLLM encourages contributions by welcoming users to incorporate their custom LLMs into the ecosystem. Check out Adding a New Model Guide to see how you can do it yourself. [?][?] Integrations OpenLLM is not just a standalone product; it's a building block designed to easily integrate with other powerful tools. We currently offer integration with BentoML and LangChain. BentoML OpenLLM models can be integrated as a Runner in your BentoML service. These runners have a generate method that takes a string as a prompt and returns a corresponding output string. This will allow you to plug and play any OpenLLM models with your existing ML workflow. import bentoml import openllm model = "opt" llm_config = openllm.AutoConfig.for_model(model) llm_runner = openllm.Runner(model, llm_config=llm_config) svc = bentoml.Service( name=f"llm-opt-service", runners=[llm_runner] ) @svc.api(input=Text(), output=Text()) async def prompt(input_text: str) -> str: answer = await llm_runner.generate(input_text) return answer HuggingFace Agents OpenLLM seamlessly integrates with HuggingFace Agents. Warning The HuggingFace Agent is still at experimental stage. It is recommended to OpenLLM with pip install -r nightly-requirements.generated.txt to get the latest API update for HuggingFace agent. import transformers agent = transformers.HfAgent("http://localhost:3000/hf/agent") # URL that runs the OpenLLM server agent.run("Is the following `text` positive or negative?", text="I don't like how this models is generate inputs") Note Only starcoder is currently supported with Agent integration. The example aboved was also ran with four T4s on EC2 g4dn.12xlarge If you want to use OpenLLM client to ask questions to the running agent, you can also do so: import openllm client = openllm.client.HTTPClient("http://localhost:3000") client.ask_agent( task="Is the following `text` positive or negative?", text="What are you thinking about?", ) LangChain ([?]Coming Soon!) In future LangChain releases, you'll be able to effortlessly invoke OpenLLM models, like so: from langchain.llms import OpenLLM llm = OpenLLM.for_model(model_name='flan-t5') llm("What is the difference between a duck and a goose?") if you have an OpenLLM server deployed elsewhere, you can connect to it by specifying its URL: from langchain.llms import OpenLLM llm = OpenLLM.for_model(server_url='http://localhost:8000', server_type='http') llm("What is the difference between a duck and a goose?") Deploying to Production To deploy your LLMs into production: 1. Building a Bento: With OpenLLM, you can easily build a Bento for a specific model, like dolly-v2, using the build command.: openllm build dolly-v2 A Bento, in BentoML, is the unit of distribution. It packages your program's source code, models, files, artifacts, and dependencies. 2. Containerize your Bento bentoml containerize BentoML offers a comprehensive set of options for deploying and hosting online ML services in production. To learn more, check out the Deploying a Bento guide. Telemetry OpenLLM collects usage data to enhance user experience and improve the product. We only report OpenLLM's internal API calls and ensure maximum privacy by excluding sensitive information. We will never collect user code, model data, or stack traces. For usage tracking, check out the code. You can opt-out of usage tracking by using the --do-not-track CLI option: openllm [command] --do-not-track Or by setting environment variable OPENLLM_DO_NOT_TRACK=True: export OPENLLM_DO_NOT_TRACK=True Community Engage with like-minded individuals passionate about LLMs, AI, and more on our Discord! OpenLLM is actively maintained by the BentoML team. Feel free to reach out and join us in our pursuit to make LLMs more accessible and easy-to-use Join our Slack community! Contributing We welcome contributions! If you're interested in enhancing OpenLLM's capabilities or have any questions, don't hesitate to reach out in our discord channel. Checkout our Developer Guide if you wish to contribute to OpenLLM's codebase. About An open platform for operating large language models (LLMs) in production. Fine-tune, serve, deploy, and monitor any LLMs with ease. bentoml.com Topics machine-learning ai deployment falcon developer-tools easy-to-use dolly fine-tuning t5-model llm model-inference lmops llmops chatglm stablelm Resources Readme License Apache-2.0 license Security policy Security policy Stars 2.5k stars Watchers 29 watching Forks 56 forks Report repository Releases 19 v0.1.8 Latest Jun 19, 2023 + 18 releases Packages 0 No packages published Contributors 4 * @aarnphm aarnphm Aaron Pham * @jianshen92 jianshen92 Jian Shen * @parano parano Chaoyu * @pre-commit-ci[bot] pre-commit-ci[bot] Languages * Python 99.2% * Other 0.8% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.