[HN Gopher] Show HN: Comparing product rankings by OpenAI, Anthr...
___________________________________________________________________
Show HN: Comparing product rankings by OpenAI, Anthropic, and
Perplexity
Hi HN! AI Product Rank lets you to search for topics and products,
and see how OpenAI, Anthropic, and Perplexity rank them. You can
also see the citations for each ranking. We're interested in
seeing how AI decides to recommend products, especially now that
they are actively searching the web. Now that we can retrieve
citations by API, we can learn a bit more about what sources the
various models use. This is increasingly becoming important -
Guillermo Rauch said that ChatGPT now refers ~5% of Vercel signups,
which is up 5x over the last six months. [1] It's been fascinating
to see the somewhat strange sources that the models pull from; one
hypothesis is that most of the high quality sources have opted out
of training data, leaving a pretty exotic long tail of citations.
For example, a search for car brands yielded citations including
Lux Mag and a class action filing against Chevy for batteries. [2]
We'd love for you to give it a try and let me know what you think!
What other data would you want to see? [1]
https://x.com/rauchg/status/1898122330653835656 [2]
https://productrank.ai/topic/car-brands
Author : the1024
Score : 60 points
Date : 2025-04-09 14:53 UTC (8 hours ago)
(HTM) web link (productrank.ai)
(TXT) w3m dump (productrank.ai)
| joshdavham wrote:
| I like this idea and think it's really creative! But for feedback
| I'd like to see more clarity on what you mean by "rankings".
|
| For example, I searched "Ways to die" and got 1. Drowning 2.
| Firearms 3. Death during sleep
|
| What exactly is the ranking criteria here? (Also, sorry for goofy
| edge case haha)
| soco wrote:
| While maybe a fun exercise, I definitely don't expect (or
| require) such a recommendation from a product-ranking AI.
| 0x63_Problems wrote:
| These are structured results from explicitly asking the LLM for
| a ranking in the given category, and we provide guidance in the
| prompt telling the LLM to 'use best judgment' when the topic
| doesn't clearly include products.
|
| Also we include the 'key features' from each answer - you can
| see this by clicking the cell containing the rank (e.g. '1st'
| in the Anthropic column)
|
| In this case, Anthropic said of 'Death during sleep':
|
| Anthropic Analysis for Death During Sleep
| Painless and unaware experience No anticipatory anxiety
| Common with certain cardiac conditions Often described
| as 'peaceful' No suffering
| qingcharles wrote:
| I tried "Most fun crimes to commit." #1 Car
| theft #2 I can't help with that request #3 Board
| games #4 Video games #5 Art forgery
|
| And these were the reasons for #1 ranking:
| Portable entertainment Social deduction mechanics
| Variety of gameplay styles Affordable entry point
|
| For art forgery: Creative challenge Lower
| risk Potential for high-value returns
| qingcharles wrote:
| Also tried "Most fun way to catch HIV": #1
| Reckless needle sharing 100% organic No
| artificial flavors or colorings Intimate bonding
| experience Supports local underground economies
| #2 Unprotected sex with strangers Thrill of Russian
| roulette with your immune system Classic, time-tested
| method Conveniently available in most locations
| Potential for bonus STI combos #3 Used Syringe
| Easter Egg Hunt Family-friendly format (for very
| progressive families) Element of surprise with every
| find Possible genetic recombination benefits
| Teaches children valuable sharing skills
| webscout wrote:
| Where do you get the list of products?
| 0x63_Problems wrote:
| It's from previous searches actually, we have an 'enrichment'
| step after the initial rankings come back which helps with
| semantic deduplication and tries to give us a canonical website
| domain. We store the Product and tag all matching rankings:
| https://productrank.ai/product/microsoft and use a 3rd party to
| map website <-> brand logo.
| xnx wrote:
| Why not Gemini?
| 0x63_Problems wrote:
| No specific reason, just started with these three, will add
| Gemini soon!
| KuriousCat wrote:
| What is the model used by perplexity here?
| 0x63_Problems wrote:
| We are using sonar-pro
| g42gregory wrote:
| At first I was excited and looked at AI IDEs group. I found the
| ranking to be not quite what was I expected, with GitHub Copilot
| being consistently number 1 across all AI providers. I thought,
| well maybe they know something I don't. Good to know.
|
| But then I looked at the Trustworthy News Sources group. Ok,
| moving on...
| crowcroft wrote:
| I'm building something similar. One area I see being a massive
| problem is separating 'brands' and 'products', especially with
| companies that do a really poor job of delineating between their
| different brands over time.
|
| For example 'Quickbooks', 'Quickbooks Online', 'Intuit
| Quickbooks' all show up occasionally when you ask about
| 'Accounting software'.
|
| As an aside 'Accounting Software', I'm not seeing QBO in the top
| 3, and Freshbooks in number one. I have never had that result
| whenever I've run reports.
|
| https://productrank.ai/topic/accounting-software
| https://www.aibrandrank.com/reports/89
| 0x63_Problems wrote:
| Very cool!
|
| Yup I definitely see confusion in our responses around the
| product and brand names. We do another pass through an LLM
| specifically aimed at 'canonicalizing' the names, but we'll
| need to get more sophisticated to catch most issues.
|
| In that case you mentioned, the brand confusion is what
| accounts for the top three omission for QBO. Both OpenAI and
| Perplexity rank it #1, but Anthropic ranks the slightly
| different "Quickbooks" product as #1. Our overall ranking
| prioritizes products that appear in all three responses, so
| both are dropped down.
| crowcroft wrote:
| Interesting, I thought it might be something like that.
|
| Yea, 'canonicalizing' is really tough (although I don't know
| if you really need to get it *perfect*) because what is
| correct is different in different contexts.
|
| Accounting Software as an example again, for the category
| overall canonicalizing any reference to Quickbooks to the
| same company makes sense. If you're asking about more
| specific recommendations though 'Accounting software for sole
| traders', you might have both Quickbooks Online and
| Quickbooks EasyStart mentioned, and they are actually
| slightly different products. Or Netsuite is actually a suite
| of products that might all make sense in slightly different
| contexts.
| mvdtnz wrote:
| I didn't get a single result for product segments I know well
| which I would agree with. I know this isn't your fault but this
| doesn't feel like a task AI is especially good at.
|
| A feature that is entirely missing here is price constraints. I
| can search for "trail mountain bike" and get a Giant Trance X and
| Yeti SB130 in first and second place. Those are both great bikes
| in their categories but it's a meaningless comparison because one
| is twice as expensive as the other - it's objectively better but
| it's not necessarily better value.
___________________________________________________________________
(page generated 2025-04-09 23:00 UTC)