[HN Gopher] Show HN: Token price calculator for 400+ LLMs
       ___________________________________________________________________
        
       Show HN: Token price calculator for 400+ LLMs
        
       Hey HN! Tokencost is a utility library for estimating LLM costs.
       There are hundreds of different models now, and they all have their
       own pricing schemes. It's difficult to keep up with the pricing
       changes, and it's even more difficult to estimate how much your
       prompts and completions will cost until you see the bill.
       Tokencost works by counting the number of tokens in prompt and
       completion messages and multiplying that number by the
       corresponding model cost. Under the hood, it's really just a simple
       cost dictionary and some utility functions for getting the prices
       right. It also accounts for different tokenizers and float
       precision errors.  Surprisingly, most model providers don't
       actually report how much you spend until your bills arrive. We
       built Tokencost internally at AgentOps to help users track agent
       spend, and we decided to open source it to help developers avoid
       nasty bills.
        
       Author : Areibman
       Score  : 50 points
       Date   : 2024-06-17 19:44 UTC (3 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | simonw wrote:
       | I don't understand how the Claude functionality works.
       | 
       | As far as I know Anthropic haven't released the tokenizer for
       | Claude - unlike OpenAI's tiktoken - but your tool lists the
       | Claude 3 models as supported. How are you counting tokens for
       | those?
        
         | dudeinhawaii wrote:
         | It's open source so you can take a look (I'm not the author):
         | https://github.com/AgentOps-AI/tokencost/blob/main/tokencost...
         | 
         | It looks like tiktoken is the default for most of the methods.
         | 
         | Disclaimer: I didn't fully trace which are being used in each
         | case/model.
        
           | refibrillator wrote:
           | _> # TODO: Add Claude support _
           | 
           | There are no cases for Claude models yet.
           | 
           | I wonder if anyone has run a bunch of messages through
           | Anthropic's API and used the returned token count to
           | approximate the tokenizer?
        
           | simonw wrote:
           | Yeah, I asked here because I dug around in the code and
           | couldn't see how they were doing this, wanted to check I
           | hadn't missed something.
        
         | Areibman wrote:
         | Anthropic actually has a Claude 3 tokenizer tucked away in one
         | of their repos: https://github.com/anthropics/anthropic-
         | tokenizer-typescript
         | 
         | At this moment, Tokencost uses the OpenAI tokenizer as a
         | default tokenizer, but this would be a welcome PR!
        
           | simonw wrote:
           | "This package can be used to count tokens for Anthropic's
           | older models. As of the Claude 3 models, this algorithm is no
           | longer accurate [...]"
           | 
           | I've been bugging Anthropic about this for a while, they said
           | that releasing a new tokenizer is not on their current
           | roadmap.
        
             | throwaway211 wrote:
             | Imagine a coffee shop refusing to have a price list until
             | after the coffee's been made.
        
       | yelnatz wrote:
       | Can you do a column and normalize them?
       | 
       | Too many zeroes for my blind ass making it hard to compare.
        
         | ryaneager wrote:
         | Yeah a Tokens per $1 column would vastly help the readability.
        
           | qeternity wrote:
           | $/million tokens is the standard pricing metric.
        
       | ilaksh wrote:
       | Nice. Any plans to add calculations for image input for the
       | models that allow that?
        
         | Areibman wrote:
         | Perhaps at some point! Right now, we haven't been seeing most
         | demand on the language side of things as multi-modal image
         | really hasn't popped off yet
        
       | Ilasky wrote:
       | I dig it! Kind of related, but I made a comparison of LLM API
       | costs vs their leaderboard performance to gauge which models can
       | be more bang for the buck [0]
       | 
       | [0] https://llmcompare.net
        
         | SubiculumCode wrote:
         | Sure makes the case for Gemini Pro, doesn't it.
        
       | Lerc wrote:
       | With all the options there seems like an opportunity for a single
       | point API that can take a series of prompts, a budget and a
       | quality hint to distribute batches for most bang for buck.
       | 
       | Maybe a small triage AI to decide how effectively models handle
       | certain prompts to preserve spending for the difficult tasks.
       | 
       | Does anything like this exist yet?
        
       | Karrot_Kream wrote:
       | A whole bunch of the costs are listed as zeroes, with multiple
       | decimal points. I noticed y'all used the Decimal library and
       | tried to hold onto precision so I'm not sure what's going on, but
       | certainly some of the cheaper models just show up as "free".
        
       ___________________________________________________________________
       (page generated 2024-06-17 23:00 UTC)