[HN Gopher] 100M Token Context Windows
___________________________________________________________________
100M Token Context Windows
Author : gklitt
Score : 70 points
Date : 2024-08-29 17:32 UTC (5 hours ago)
(HTM) web link (magic.dev)
(TXT) w3m dump (magic.dev)
| smusamashah wrote:
| It should be benchmarked against something like RULER[1]
|
| 1: https://github.com/hsiehjackson/RULER (RULER: What's the Real
| Context Size of Your Long-Context Language Models)
| ipsum2 wrote:
| > To incorporate this, we ask the model to complete a chain of
| hashes instead (as recently proposed by RULER):
|
| They did mention it but didn't provide concrete benchmarks
| samber wrote:
| Based on Mamba ?
| shazami wrote:
| FYI wouldn't interview here. Got rejected after a 30 minute
| behavioral screen after spending 8 hours on an unpaid take-home.
| dinobones wrote:
| Long context windows are IMO, "AGI enough."
|
| 100M context window means it can probably store everything you've
| ever told it for years.
|
| Couple this with multimodal capabilities, like a robot encoding
| vision and audio into tokens, you can get autonomous assistants
| than learn your house/habits/chores really quickly.
| dogma1138 wrote:
| Has anyone measured the performance of very large context
| windows like this vs a good RAG that you also constantly update
| and curate?
|
| At least with other very large context windows like for example
| Claude offers a RAG is still very much preferable as it avoids
| confusion and collisions with information in the context that
| isn't correct or relevant.
|
| Sure you can also prune the context window and for many
| existing models you also need to do that (I often use an LLM to
| summarize a context to keep it going) but doing it with a RAG
| seems to still be much easier. This especially holds true of
| you use good knowledge management techniques to structure your
| RAG so your retrievals are optimized.
|
| P.S. on a side note how confident are we that these very large
| context window models are not just a RAG in disguise? As the
| models which boast very large windows are at least for now all
| locked behind API access only.
| jokethrowaway wrote:
| Context window size is not the limiting factor. How well will
| it be able to use that information is the problem.
|
| Even GPT and Claude make glaring mistakes with short prompts.
| Sakos wrote:
| I was wondering how they could afford 8000 H100's, but I guess I
| accidentally skipped over this part:
|
| > We've raised a total of $465M, including a recent investment of
| $320 million from new investors Eric Schmidt, Jane Street,
| Sequoia, Atlassian, among others, and existing investors Nat
| Friedman & Daniel Gross, Elad Gil, and CapitalG.
|
| Yeah, I guess that'd do it. Who are these people and how'd they
| convince them to invest that much?
| 0cf8612b2e1e wrote:
| For those names (access to $billions), curious how much due
| diligence they do any more. Just make a "chump change"
| investment in every hot trend? One phony AI startup pitch deck
| will look identical (if not better) to one with a real edge.
| IHLayman wrote:
| Assume around $3/hr per H100 (pretty generous pricing for GCP),
| that is $2250/month-gpu, or for their fleet of 8000 comes to
| $18MM/month or around $216MM/year in just compute costs alone,
| not looking at SSD, bucket storage, or egress. At their initial
| investment of 465-320=$145MM that means they can't have
| operated that cluster for longer than 6ish months without their
| funds running dry or the got massive discounts somewhere.
|
| Something doesn't add up here.
| Sakos wrote:
| Honestly, the more I look at this the more I'm mystified.
| Magic was founded in 2022 and they received their first
| funding round of $20 million a year later. Before that, the
| co-founder/CEO had founded some climate science education
| program called ClimateScience which he ran for 2 years:
|
| > Then, it hit them. They thought, "What if we create bite-
| sized information, following the same scientific standards of
| peer-reviewed journals, to empower people to solve climate
| change?"
|
| > Together, they started combing through climate science
| articles and turning them into social-media friendly content
| under the name ClimateScience. After two short months,
| ClimateScience went viral and grew to 40,000 followers on
| Instagram. People started sending in private messages, asking
| how they could help. A team of curious, kind and passionate
| people quickly grew, all dedicated to making climate
| education more understandable for everyone.
|
| > Just a few years later, ClimateScience has grown into the
| world's biggest climate education platform! We create
| educational courses, videos, resources and tools to improve
| climate understanding and education. It's all completely free
| and just a few clicks away on any device.
|
| According to LinkedIn, they have 50-200 employees. Is that
| plausible? How many of those are actually FTEs? Where is
| their revenue coming from if it's all completely free?
| Looking at the team page, this feels off, like it's a bunch
| of university students padding their CV.
|
| That was after dropping out of university, 1 year into a
| bachelor's in computer science. During which he apparently
| had a 5 month contract at Facebook AI where he "lead the
| development of 'DREAM', an algorithm that's 100x more data-
| efficient and trains faster than the previous state-of-the-
| art in model-free multi-agent Deep RL. Paper:
| https://arxiv.org/abs/2006.10410".
|
| How does this lead to Magic.dev and third-parties investing
| $500 million? Either this guy is a prodigy or this is the
| next Theranos.
|
| edit: I looked into the other co-founder just now and I feel
| like I'm in the twilight zone.
| fsndz wrote:
| Context windows are becoming larger and larger, and I anticipate
| more research focusing on this trend. Could this signal the
| eventual demise of RAG? Only time will tell. I recently
| experimented with RAG and the limitations are often surprising
| (https://www.lycee.ai/blog/rag-fastapi-postgresql-pgvector). I
| wonder if we will see some of the same limitations for long
| context LLM. In context learning is probably a form of semantic /
| lexical cues based arithmetic.
| htrp wrote:
| does anyone have a detailed tech breakdown of these guys? not
| quite sure how their LTM architecture works.
| why_only_15 wrote:
| They're not saying
| 1024core wrote:
| These days the competition is so fierce that everyone's
| clammed up.
___________________________________________________________________
(page generated 2024-08-29 23:01 UTC)