[HN Gopher] Ask HN: Bypassing GPT-4 8k tokens limit
___________________________________________________________________
Ask HN: Bypassing GPT-4 8k tokens limit
Okay, I know it's not possible to bypass the 8k token limit. I
don't have access to the 32k model yet. I have transcripts that
are typically around 15000 tokens in size. I want to split this
text into different topics. The problem is the current limit to
GPT-4. The obvious approach would be to split the text into chunks
and then send to the API. However, GPT-4 won't have the context of
the other chunks to accurately identify topics inside the text. For
example, the chunk could separate the text in the middle of a
topic. I can't think of a way that can programatically chunk the
text without breaking up topics by mistake. Does anyone know of a
way around this or have a better approach? Or is this just the
reality of GPT-4 at the moment.
Author : LewisDavidson
Score : 15 points
Date : 2023-05-01 19:24 UTC (3 hours ago)
| fakedang wrote:
| Divide your data into smaller chunks, then use some kind of
| initial vector similarity check to choose only the relevant bits.
| summarity wrote:
| Use semantic compression (plenty of papers on that now). Works
| for both language and code.
| MH15 wrote:
| Interesting question and quality answers, but I was under the
| impression that specific technical Q and A posts aren't for HN.
| This seems like a question better suited for StackOverflow or a
| forum dedicated to AI engineering.
| mansueli wrote:
| Did you take a look at https://github.com/yasyf/compress-gpt?
| jnwatson wrote:
| Wow that's way more in-depth than "ask chatgpt to compress
| this". Very neat.
| noman-land wrote:
| I haven't tried this yet but I've been thinking experimenting
| with feeding the summary of the first arbitrary chunk in with the
| next chunk. Then feed the summary of the second chunk in with the
| third chunk, etc.
| ndr_ wrote:
| Check out llama_index at
| https://github.com/jerryjliu/llama_index. What it does: it
| creates an index over your data using OpenAI embeddings vectors,
| using the OpenAI Ada model. When querying, it compiles as much
| context out of this index as fits into GPT, based on similarity
| to your prompt. Be cautious however: when I experimented with
| this, GPT-4 support with it's larger context size was not there
| yet. I have landed
| https://github.com/hwchase17/langchain/pull/1778, but I never
| wound up submitting another, yet similar patch (to llama_index?
| Don't remember). Make sure that the GPT-4 context is really fully
| used, and not some smaller size is assumed. Also, ensure that
| GPT-4 is used as the LLM in the first place: the defaults used to
| be the older models.
| aClicheName wrote:
| You could try to create some sort of compression instead. By
| that, I mean, instead of appending the chunks, what if you get a
| summary of that rough area (along with an indicated variable
| holding onto that character position)? You could then use the
| summary in a temporary storage as a sort of "index" that roughly
| outlines that area. Appending these together, along with mixing
| things around, you could create something that "roughly knows"
| about the document, and knows where to go looking for further,
| in-depth info. - Henry
| f0e4c2f7 wrote:
| If you have access to bing someone figured out you can enable a
| longer token length by editing the HTML as the limit was set
| browser side rather than server side. Not sure if this has been
| patched yet
| sharemywin wrote:
| you could use ntlk to summarize the text before you send it
| GPT-4.
|
| I have a script that uses NLTK to do this. It needs cleaned up
| but it could be a starting point.
|
| https://github.com/gnuconcepts/Text_summary
___________________________________________________________________
(page generated 2023-05-01 23:02 UTC)