[HN Gopher] Show HN: I made the slowest, most expensive GPT
___________________________________________________________________
Show HN: I made the slowest, most expensive GPT
This is another one of my automate-my-life projects - I'm
constantly asking the same question to different AIs since there's
always the hope of getting a better answer somewhere else. Maybe
ChatGPT's answer is too short, so I ask Perplexity. But I realize
that's hallucinated, so I try Gemini. That answer sounds right, but
I cross-reference with Claude just to make sure. This doesn't
really apply to math/coding (where o1 or Gemini can probably one-
shot an excellent response), but more to online search, where
information is more fluid and there's no "right" search engine +
text restructuring + model combination every time. Even o1 doesn't
have online search, so it's obviously a hard problem to solve. An
example is something like "best ski resorts in the US", which will
get a different response from every GPT, but most of their rankings
won't reflect actual skiers' consensus - say, on Reddit
https://www.reddit.com/r/skiing/comments/sew297/updated_us_s... -
because there's so many opinions floating around, a one-shot RAG
search + LLM isn't going to have enough context to find how
everyone thinks. And obviously, offline GPTs like o1 and
Sonnet/Haiku aren't going to have the latest updates if a resort
closes for example. So I've spent the last few months
experimenting with a new project that's basically the most
expensive GPT I'll ever run. It runs search queries through
ChatGPT, Claude, Grok, Perplexity, Gemini, etc., then aggregates
the responses. For added financial tragedy, in-between it also uses
multiple embedding models and performs iterative RAG searches
through different search engines. This all functions as sort of
like one giant AI brain. So I pay for every search, then every
embedding, then every intermediary LLM input/output, then the final
LLM input/output. On average it costs about 10 to 30 cents per
search. It's also extremely slow. https://ithy.com I know that
sounds absurdly overkill, but that's kind of the point. The goal is
to get the most accurate and comprehensive answer possible, because
it's been vetted by a bunch of different AIs, each sourcing from
different buckets of websites. Context limits today are just large
enough that this type of search and cross-model iteration is
possible, where we can determine the "overlap" between a diverse
set of text to determine some sort of consensus. The idea is to get
online answers that aren't attainable from any single AI. If you
end up trying this out, I'd recommend comparing Ithy's output
against the other GPTs to see the difference. It's going to cost
me a fortune to run this project (I'll probably keep it online for
a month or two), but I see it as an exploration of what's possible
with today's model APIs, rather than something that's immediately
practical. Think of it as an online o1 (without the $200/month
price tag, though I'm offering a $29/month Pro plan to help
subsidize). If nothing else, it's a fun (and pricey) thought
experiment.
Author : wluk
Score : 50 points
Date : 2024-12-13 15:05 UTC (7 hours ago)
(HTM) web link (ithy.com)
(TXT) w3m dump (ithy.com)
| wluk wrote:
| If anyone tries this out and hits the limit, just let me know and
| I'll increase it for you for free :)
| ricktdotorg wrote:
| i like this, a lot.
|
| i asked a subjective history question about England and Ithy's
| analysis was great, and did indeed add to other GPTs!
|
| i did find the UI a bit confusing at first, that's my only
| nitpick. i signed in (nice easy flow) and will definitely
| continue to use!
|
| looks like anthropic is slowing down ithy analysis/responses, at
| least today during my tests just now anyway.
|
| great app! well done o7
| gotten09 wrote:
| o7 ? are you an Eve Online player?
| wluk wrote:
| It's a salute emoji!
|
| o7
| wluk wrote:
| Thank you! Sorry I hit my Anthropic limits a few minutes after
| this post blew up. It'll be a few days before my Anthropic
| limits increase since I have a new account with them, so
| unfortunately it won't be back until next week.
|
| Cheers!
| Mr_Bees69 wrote:
| This is really cool, quite a shame it costs 20 billion dollars
| per query lol.
| wluk wrote:
| That is the worst part haha
| maalber wrote:
| This is great, and I love your presentation of it - hilarious!
| "On average it costs about 10 to 30 cents per search. It's also
| extremely slow."
| af3d wrote:
| This is pretty impressive. Given the following scenario, "A
| bliirg is any non-wooden item, which is the opposite of a glaarg.
| In addition, there are neergs (non-existent things) and eeergs
| (things which actually exist). Now a glaarg which is also a neerg
| is called a bipk, whereas a glaarg which is a eeerg is known as a
| vokp. Also, a bliirg which is an eeerg is refererred to as a
| jokp, otherwise it is known as a fhup. So the question is, which
| of those could be used to make an actual fire: a jokp, bipk,
| fhup, or vokp? Explain your reasoning." The results were
| absolutely spot on. "(...) A vokp, being a real, existing wooden
| object, can serve as a fuel source. A jokp, being a real,
| existing non-wooden object, _might_ serve as a fuel source if it
| is combustible. However, a bipk and a fhup, being non-existent
| things, cannot be used to make a fire. The ability to actually
| start a fire also depends on the presence of an ignition source,
| oxygen, and potentially tinder, which are not addressed by the
| definitions of these terms. " Any plan to make the project open
| source?
| wluk wrote:
| > Any plan to make the project open source?
|
| Parts of this are borrowed from
| https://github.com/assafelovic/gpt-researcher
|
| It's actively being developed (I pitch in where I can; I added
| the xAI integration this week), so I'd recommend starting here!
| The creator of the project, Assaf, has been nothing but
| friendly.
| numeri wrote:
| But gpt-4o can already answer your question for a fraction of
| the price and time:
|
| To determine which of the items could be used to make an actual
| fire, we need to analyze the definitions provided:
|
| 1. *Glaarg*: A wooden item.
|
| 2. *Bliirg*: A non-wooden item.
|
| 3. *Neerg*: A non-existent thing.
|
| 4. *Eeerg*: A thing that actually exists.
|
| Now, let's look at the specific terms:
|
| - *Bipk*: A glaarg (wooden item) that is also a neerg (non-
| existent thing). Since it is non-existent, it cannot be used to
| make a fire.
|
| - *Vokp*: A glaarg (wooden item) that is also an eeerg
| (existent thing). Since it is a wooden item that exists, it can
| be used to make a fire.
|
| - *Jokp*: A bliirg (non-wooden item) that is also an eeerg
| (existent thing). While it exists, it is non-wooden, so it may
| not be suitable for making a fire depending on its material.
|
| - *Fhup*: A bliirg (non-wooden item) that is also a neerg (non-
| existent thing). Since it is non-existent, it cannot be used to
| make a fire.
|
| Based on this analysis, the only item that can be used to make
| an actual fire is a *vokp*, as it is a wooden item that exists.
| wluk wrote:
| Yeah the strength of Ithy isn't really in puzzles or math.
|
| It's more of just a better search engine. Use it for stuff
| you'd Google. Offline LLMs are always going to have a better
| price-performance ratios than RAGs like this or Perplexity.
| roberdam wrote:
| Fantastic idea and implementation, thank you for putting your
| effort and money into bringing such an interesting idea to life!.
| vinni2 wrote:
| It was weird I made a simple claim "Keto diet cures cancer" and
| it analyzed history of cheeseburgers after a couple of minutes by
| saying "Sorry, I can't respond to that. I will now analyze the
| history of cheeseburgers instead"
| wluk wrote:
| I filter out malicious prompts and respond with the history of
| cheeseburgers for stuff like "ignore previous instructions"
|
| Weird that your query triggered the filter. Maybe GPT is just
| _that_ afraid of keto diets...
|
| (I'll look into it; thanks for the note!)
| vinni2 wrote:
| this is not ideal. If someone wants to fact check
| controversial claim filtering it only makes it worse.
| zamadatix wrote:
| Controversial != malicious. It sounds like they never
| intended for the filter to trigger for the former and they
| already said they'll look into why it did for the prompt.
|
| Filtering malicious (not controversial) usage is ideal as
| allowing users to flood all of the AI services with
| jailbreak/against-ToS query attempts can be bad news for
| your API keys (as well as a likely waste of money given the
| failure rate of such queries).
| cootsnuck wrote:
| This is just a dude's hobby project, chill.
| wluk wrote:
| I asked the LLM to explain why it's filtering out "keto diet
| cures cancer", and just the act of asking it to explain it,
| makes it work again. Interesting...
|
| https://ithy.com/article/b848ebc9a32140ffa766a1520113846c
| sergiotapia wrote:
| `how to build a birdhouse to attract a bluebird`
|
| https://ithy.com/article/4b116d2032e54c03862db84e71bcfc8f
|
| https://big-agi.com/ has this "BEAM" concept as well where you
| can put your message through as many models as you have
| configured then run fuse/guided/compare/custom to merge them all
| together into one comprehensive. more expensive response.
|
| https://files.catbox.moe/tr82vs.png
|
| https://files.catbox.moe/beuyfx.png
|
| However Ithy does produce something much much better! This is
| really cool! I wonder if you could cache questions and answers,
| and start creating your own "reddit" knowledgebase to RAG from
| and avoid having to dive deep again $$$.
| wluk wrote:
| Thanks! Yeah that's an excellent idea - this is my response
| from another thread:
|
| I have a feeling that Perplexity and ChatGPT are doing
| something similar [caching], since common questions I'd ask
| like "top movies this year" will be answered nearly-
| instantaneously, way faster than GPT-4o could have done on its
| own.
|
| The only explanation for this is that so many users ask certain
| questions, they cache the response and return the cached
| answer.
|
| I'd love to do this for Ithy, but it'll be a while before I get
| the scale of ChatGPT/Perplexity that's needed for this...
| SCUSKU wrote:
| "For added financial tragedy" really got me. Pretty interesting
| project, luckily if this whole software engineering thing doesn't
| work out thankfully you can fall back on your comedy career
| wluk wrote:
| ...at least you can't lose money in comedy :(
| msdundarss wrote:
| Interesting idea!
| duxup wrote:
| Very cool.
|
| I've found that very generic queries like "best ski resorts in
| the US" seem woefully pouted by top 10 spam sites. LLMs do not
| want to give any useful info about that no matter how much
| prompting I seem to give.
|
| I was looking for an app that does X,Y,Z recently and no amount
| of prompting for open source would get me anything but a handful
| of stock answers I would get from a random spam site.
| wluk wrote:
| Yeah, GPT is learning from GPT, which is extremely
| disappointing.
|
| Like I'll try to find the top burgers in midtown. Perplexity or
| ChatGPT online searching will always find "Top 10 Burgers in
| Midtown" by https://nycreviewed.com/ (their SSL certificate
| isn't even valid). But this itself is a GPT-generated list, and
| their list isn't even in midtown.
|
| So now I end up with a list of 10 burger places that _aren 't_
| in midtown.
|
| Ithy kinda solves this by utilizing multiple search engines and
| LLMs, so spam sites like this are more likely to get filtered
| out somewhere in the pipeline.
| bioxept wrote:
| I signed up but I just get an alert that I will get 10 free
| requests when I signup. It quits the search when I press ok.
| Running it on safari iOS.
| wluk wrote:
| Sorry, you were a victim of the outage caused by HN flooding my
| website! It's back online now if you want to give it a try :)
| bee_rider wrote:
| I wonder, is it a given that asking multiple models will give a
| better output? Can you ask one model slightly different prompts,
| take the outputs, and ask it to summarize them? Or even argue
| amongst itselfs?
| wluk wrote:
| Yes, o1 does this internally, and there's agentic AI systems
| already doing better than singular AIs in fields like writing,
| where you assign each AI system a role like "writer" or
| "editor" or "marketing" and they discuss among themselves.
|
| I'm just applying this theory to online search.
| Atotalnoob wrote:
| I tried to search, but it immediately prompted me to login with
| Google. I didn't see an option for non-Google...
|
| I degoogled a while ago...
| wluk wrote:
| Yeah, sorry, I'm hoping to integrate more login options soon.
| Are you more of a email/phone login person, or is there another
| third-party login you had in mind?
| Atotalnoob wrote:
| Email would be best for me.
| wluk wrote:
| That's the one I was trying to avoid, since there's so many
| ways to create fake accounts with it :(
| wluk wrote:
| Update 3:00 PM ET: I've finished scaling up from 2 VPCs to 5
| VPCs. Limits have been increased back up to 3 anonymous / 10
| signed-in.
|
| Update 2:30 PM ET: Back up (for now). Still waiting for Anthropic
| and Gemini quota increase requests, so those have been migrated
| to GPT-4o for now. Running on 2 VPCs, in the process of launching
| 2 more. Confident that I can increase the daily limits by EOD
| once everything's more stable.
|
| Update 1:30 PM ET: HN blew up Ithy and it's 99% down right now,
| congrats ._.
|
| 1. I've exceeded my weekly Anthropic API limits; I've gotten in
| touch with their sales team and I've temporarily disabled the
| Anthropic model.
|
| 2. Blew past my Google API limits as well. I was using Gemini for
| prompting and aggregation, and I'm waiting for their quota
| increase response. In the meantime, I've switched to GPT-4o for
| the prompting/aggregation.
|
| 3. My VPC is at 100% CPU load. Launching more right now with some
| load balancing.
|
| 4. Limits were previously 5 anon / 20 per logged in user. Reduced
| this to 1 anon / 3 logged in while I deal with the load issues.
| Planning bring these back up as soon as everything's working
| again.
|
| Hope to get this all back online within an hour or two. Sorry for
| the crappy launch. To think I was an SRE in a past life...
| toomuchtodo wrote:
| Reach out to benneo?
|
| https://news.ycombinator.com/item?id=42403006 ("Ask HN: Ideas
| for spending $8k in Anthropic credits?")
| wluk wrote:
| wow I wish I had the problem of too many credits. just
| reached out, thanks!
| LorenDB wrote:
| You could at least cheapen some of your queries by moving to
| something like Groq.
| wluk wrote:
| Good idea, maybe I'll add Groq as another option, since I don't
| have an internal Llama 3.1 flow yet. But I'll still need to
| keep the others to maintain the diversity of responses.
| OsbEss wrote:
| This is hilarious, I love the "it sounds absurdly overkill, but
| thats the point" post. Can't wait for this to be back online, I
| need to ask it about the comprehensive history of cheeseburgers.
| wluk wrote:
| Back online!
| wluk wrote:
| Update 2:30 PM ET: Back up (for now). Still waiting for Anthropic
| and Gemini quota increase requests, so those have been migrated
| to GPT-4o for now. Running on 2 VPCs, in the process of launching
| 2 more.
|
| Confident that I can increase the daily limits by EOD once
| everything's more stable.
| zamadatix wrote:
| Interesting idea, cool concept. I tried asking "What is the best
| SNES game most people haven't played". The top answer
| (Terranigma) was unfortunately the same as I got just asking any
| of Claude/ChatGPT/Llama/Qwen (maybe too easy a question) but the
| rest of the list did seem a bit more balanced. Thanks for the
| free try without a login!
|
| Thought: there is a marquee of example queries but it doesn't
| seem like there is a way to see what an answer looks like without
| individually consuming a search as a user. Maybe if these were
| clickable to a cached version it'd be easier to get an idea of
| the outputs without costing so much?
| wluk wrote:
| That's a good idea! I have a feeling that Perplexity and
| ChatGPT are doing something similar, since common questions I'd
| ask like "top movies this year" will be answered nearly-
| instantaneously, way faster than GPT-4o could have done on its
| own.
|
| The only explanation for this is that so many users ask certain
| questions, they cache the response and return the cached
| answer.
|
| I'd love to do this for Ithy, but it'll be a while before I get
| the scale of ChatGPT/Perplexity that's needed for this...
| wluk wrote:
| Update 3:00 PM ET: I've finished scaling up from 2 VPCs to 5
| VPCs. Limits have been increased back up to 3 anonymous / 10
| signed-in.
| downloadram wrote:
| nice work
|
| IF you want free tokens and IF your visitors agree for all their
| input and output to be used by all AIs involved for research and
| training or whatever.. you might be able to strike up the same
| kind of dealio that LMSYS has with lmarena - > github.com/lm-
| sys/FastChat - > lmarena .ai
|
| to use the site you have a big obvious popup explaining the data
| use quickly and efficiently
| ProfessorZoom wrote:
| so now you summarize everyone's hallucinations
___________________________________________________________________
(page generated 2024-12-13 23:02 UTC)