https://github.com/yagil/tokmon Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} yagil / tokmon Public * Notifications * Fork 0 * Star 11 CLI to monitor OpenAI token usage. Put `tokmon` before the name of your program to measure your token cost. License MIT license 11 stars 0 forks Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Security * Insights More * Code * Issues * Pull requests * Actions * Security * Insights yagil/tokmon This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/y] Use Git or checkout with SVN using the web URL. [gh repo clone yagil/] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @yagil yagil Update README.md ... 49128e3 Apr 14, 2023 Update README.md Signed-off-by: Yagil Burowski 49128e3 Git stats * 42 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows don't run gh action on README changes April 13, 2023 12:10 tests fix python example April 13, 2023 20:19 tokmon fail gracefully if the user provides a bad path for the target program April 13, 2023 21:53 .gitignore changes to python / node example programs April 13, 2023 12:25 LICENSE Create LICENSE April 12, 2023 21:08 README.md Update README.md April 13, 2023 22:33 requirements.txt setup script and tests April 12, 2023 21:02 setup.py - support running multiple instances on tokmon simultaneously (find a... April 12, 2023 23:41 View code [ ] tokmon - CLI utility to monitor OpenAI token costs How to use tokmon Quick install How it works pricing.json Current Limitations Contributing Warning README.md tokmon - CLI utility to monitor OpenAI token costs tokmon enables you to monitor your program's OpenAI API token usage. You use tokmon just like you would use the time utility, but instead of execution time you get token usage and cost. [231910274-] How to use tokmon Warning This is a debugging tool. It is not intended to be used in any consequential setting. Use your best judgement, you're on your own! Prepend tokmon to your normal program invocation like so: $ tokmon ./my_gpt_program --my_arg "hi" Run and use your program just like you would normally (arguments and all). Interactive usage is supported as well. After your program finishes running (or you ctrl^C it), tokmon will automatically generate a cost report that looks like this: tokmon cost report: ================================================================================ Monitored invocation: ./python_example.py -i Models: ['gpt-3.5-turbo-0301'] Total Usage: {'total_prompt_tokens': 49, 'total_completion_tokens': 44, 'total_tokens': 93} Pricing: {'gpt-3.5-turbo-0301': {'prompt_cost': 0.002, 'completion_cost': 0.002, 'per_tokens': 1000}} Total Cost: $0.000186 ================================================================================ Writing cost summary to JSON file: /tmp/tokmon_usage_summary_1681426650.json * If your program uses multiple OpenAI models in the same invocation, their respective usages will be reflected in the report. * You can run multiple instances of tokmon simultaneously. Each invocation will generate a separate usage report. * Pass a --json_out /your/path/report.json to get a detailed breakdown + conversation history in JSON format. Quick install Install using pip like so: $ pip install git+https://github.com/yagil/tokmon.git $ tokmon --help To uninstall, run pip uninstall tokmon Tip: make sure that the your python Library route is in your PATH. How it works tokmon uses the mitmproxy library to intercept HTTP requests and responses between your program and the OpenAI API. It then processes the request and response data to calculate token usage and cost based on tokmon/pricing.json. tokmon works for programs in python / node (using OpenAI's clients), or curl (run directly, and not i.e. in a bash script). if you manually install mitmproxy's CA certificate, it should work for all executables (note: haven't tested this.) In most cases, tokmon relies on the usage field in OpenAI's API responses for token counts. For streaming requests, however, tokmon uses OpenAI's tiktoken library directly to count the tokens. As of writing OpenAI's API does not return usage data for streaming requests (reference.) pricing.json The pricing data was extracted from OpenAI's website with the help of ChatGPT. tokmon is using tokmon/pricing.json from its package. { "last_updated": "2023-04-12", "data_sources": [ "https://openai.com/pricing", "https://platform.openai.com/docs/models/model-endpoint-compatibility" ], "gpt-4": {"prompt_cost": 0.03, "completion_cost": 0.06, "per_tokens": 1000}, "gpt-4-0314": {"prompt_cost": 0.03, "completion_cost": 0.06, "per_tokens": 1000}, "gpt-4-32k": {"prompt_cost": 0.06, "completion_cost": 0.12, "per_tokens": 1000}, "gpt-4-32k-0314": {"prompt_cost": 0.06, "completion_cost": 0.12, "per_tokens": 1000}, "gpt-3.5-turbo": {"prompt_cost": 0.002, "completion_cost": 0.002, "per_tokens": 1000}, "gpt-3.5-turbo-0301": {"prompt_cost": 0.002, "completion_cost": 0.002, "per_tokens": 1000}, "text-davinci-003": {"cost": 0.02, "per_tokens": 1000}, "text-curie-001": {"cost": 0.002, "per_tokens": 1000}, "text-babbage-001": {"cost": 0.0005, "per_tokens": 1000}, "text-ada-001": {"cost": 0.0004, "per_tokens": 1000}, "text-embedding-ada-002": {"cost": 0.0004, "per_tokens": 1000} } You can override the default pricing with: tokmon --pricing /path/to/ your/custom_pricing.json ... This pricing JSON is incomplete (missing DALL-E, etc.), it may be incorrect, and it may go out of date. For best results, make sure to check that you have the latest pricing. Current Limitations 1. Event streaming: tokmon buffers Server-Sent Events (SSE) until the data: [DONE] chunk is received. If the monitored program leverages event streaming, its behavior will be modified. + Status: looking into it. Pull requests welcome. Contributing If you'd like to contribute to the project, please follow these steps: 1. Fork the repository. 2. Create a new branch for your changes. 3. Make your changes and test them. 4. Submit a pull request with a clear description of your changes and any relevant information. Warning 1. tokmon comes without any warranty or guarantee whatsoever. 2. tokmon was tested on macOS only. It might not work on other platforms. 3. This tool may not work as intended, have unknown side effects, may output incorrect information, or not work at all. 4. The pricing data in pricing.json may go out of date. About CLI to monitor OpenAI token usage. Put `tokmon` before the name of your program to measure your token cost. Topics ai openai gpt openai-api Resources Readme License MIT license Stars 11 stars Watchers 1 watching Forks 0 forks Report repository Releases No releases published Packages 0 No packages published Languages * Python 92.4% * TypeScript 7.6% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.