[HN Gopher] Custom Embeddings: Why going viral caused us to rip ...
___________________________________________________________________
Custom Embeddings: Why going viral caused us to rip out everything
in a weekend
Author : Buoy
Score : 84 points
Date : 2023-02-22 23:45 UTC (23 hours ago)
(HTM) web link (www.buildt.ai)
(TXT) w3m dump (www.buildt.ai)
| lee101 wrote:
| [dead]
| martinclayton wrote:
| I've no idea what the 104 million / 143 million numbers in the
| spreadsheet-grid thing mean - anyone?
| mkoryak wrote:
| Didn't you read the first paragraph... twice? Oh right, its not
| in there.
| clark-kent wrote:
| I believe it refers to number of tokens. So thats's $02 per
| 1000 tokens for OpenAI Davinci model.
| Twirrim wrote:
| Tokens, I think? There's an implication that there's a per-
| token cost, but no idea how they were architected before to
| even remotely understand how it fits. If they were running
| their own model on GPUs I guess I'd expect to see things in
| terms of token/sec, so I assume instead they were using some
| cloud learning model where they have to pay per token?
|
| The blog post really suffers from "Written by someone who knows
| what they're talking about", could have done with a review
| before publication by someone who doesn't know the space. I go
| through that exercise any time I'm writing stuff for
| consumption outside of my service teams.
| gwern wrote:
| They say it cost 'thousands of dollars', so I infer that the
| 'millions' must be either raw characters or BPE tokens sent to
| the API.
| andrewstuart wrote:
| What is a VI, an LLM and a bias matrix?
| ec109685 wrote:
| VI: it's actually V1, but the blog uses a horrible font where
| you can't differentiate a 1 from an l (seriously, who thinks
| that's a good idea?).
|
| LLM: large language model
|
| Bias Matrix: part of fine tuning a model:
| https://platform.openai.com/docs/guides/fine-tuning
| AaronM wrote:
| FYI, your first paragraph is duplicated under the image.
| iandanforth wrote:
| I feel like there's also a paragraph _missing_ explaining the
| product and the problem and how they were using LLMs initially.
| supermatt wrote:
| > We considered trying to use a self-hosted LLM as an
| alternative, but the costs would also have been extremely high
| for the amount of traffic we were processing.
|
| Is it realistic to self-host an LLM that outperforms OpenAIs
| offerings cost wise? When I looked at the alternatives (self-
| hosted, alternate hosted LLM providers, or cloud compute options)
| you generally ended up with a subjectively worse model AND a
| lower inference speed - which resulted in me canning my idea as
| it was simply too expensive.
| billythemaniam wrote:
| Flan-T5 is much smaller than GPT-3, but was trained on
| significantly more data resulting in competitive accuracy. It
| is also Apache licensed. I wonder if that model is fast enough
| for enough use cases to make it cost effective?
| danielbln wrote:
| You can give the Xl (3B parameters) model a try here (would
| recommend a Colab Pro account): https://colab.research.google
| .com/drive/1Hl0xxODGWNJgcbvSDsD...
|
| In my Colab Pro it's running this on a A100 (which is a very
| beefy GPU) and inference is very fast and definitely suitable
| for interactive use. On a T5 GPU (which is much cheaper)
| inference is still alright and probably ok for interactive
| use.
| williamcotton wrote:
| Fine-tuning on smaller models like GPT-J (also trained on The
| Pile) worked well for Toolformer:
|
| https://arxiv.org/abs/2302.04761
| ntonozzi wrote:
| I think Flan-T5 is fast enough, but I don't think it
| generates text or abstract reasoning at nearly the same level
| as current GPT-3 models. This indicates a deficiency in the
| benchmarks and metrics that we use to evaluate LLMs. For
| generating embeddings it might work well enough though.
| billythemaniam wrote:
| It's certainly not quite as good out of the box, at least
| the open sourced checkpoints. However so far I found it can
| achieve similar accuracy with enough examples and/or fine-
| tuning for my use cases. Like everything, it depends on
| what are doing too.
| tikkun wrote:
| Not sure what the product is
|
| But the advice to essentially fine tune your embeddings with a
| custom matrix is good
|
| There are also other embeddings platforms (other than OpenAI's)
| that have built in fine tuning functionality
| shiv86 wrote:
| Could you please provide alternative embedding platforms
| metalembeddings wrote:
| At Metal http://getmetal.io/ we're currently building a fine
| tuning platform. We host, index and version embeddings.
| Provide an easy way to manage the fine tuning jobs as well.
| diceduckmonk wrote:
| There is a description at the bottom
|
| > Buildt is an AI tool to help developers quickly search and
| understand large codebases.
| draw_down wrote:
| [dead]
| jxodwyer1 wrote:
| Great post -- one of our core offerings at https://getmetal.io is
| to manage the process of utilizing Custom Embeddings very similar
| to your approach here! We'd love to connect and talk through some
| of the pain points that you came across -- i'll send you a
| message in the w23 slack!
| tylerneylon wrote:
| Would you mind providing a pointer to the the part of the OpenAI
| cookbook which explains how to build the matrix you multiply the
| embeddings by?
|
| Sorry if it was there and I missed it. Thanks!
| cheeseblubber wrote:
| I think https://github.com/openai/openai-
| cookbook/blob/main/examples... is the one they are referring
| to. This is the page that links it
| https://github.com/openai/openai-cookbook/blob/main/text_com...
| tylerneylon wrote:
| Thanks!
| urbandw311er wrote:
| I think you should have a put a bit more thought into planning
| for scale. There's a difference between not overengineering and
| not actually doing the basic maths to figure out the unit
| economics of your business model ahead of setting it live to the
| whole world.
___________________________________________________________________
(page generated 2023-02-23 23:01 UTC)