[HN Gopher] Claude can now search the web
___________________________________________________________________
Claude can now search the web
Author : meetpateltech
Score : 639 points
Date : 2025-03-20 16:51 UTC (6 hours ago)
(HTM) web link (www.anthropic.com)
(TXT) w3m dump (www.anthropic.com)
| morisil wrote:
| I added this functionality already some time ago in my Claudine
| agent:
|
| https://github.com/xemantic/claudine/
|
| It costed roughly 30 lines of code:
| https://github.com/xemantic/claudine/blob/main/src/commonMai...
| mcoliver wrote:
| Kind of I guess? You should be more explicit that you're
| funnelling everything to through jina.ai
| msp26 wrote:
| > in feature preview for all paid Claude users in the United
| States. Support for users on our free plan and more countries is
| coming soon
|
| US only
| smca wrote:
| More countries soon.
| ubicomp wrote:
| Excited to see this. I've really been enjoying Claude. It feels
| like a different, more creative flavor of experience than GPT. I
| use Claude a lot for dialogues and exploring ideas, like a
| conversational partner. Having web access will add an interesting
| dimension to this.
| Brusco_RF wrote:
| Excited to see how this compares to Perplexity or Gemini. I
| remember that ChatGPT used to be able to search the web, but last
| I checked it it couldn't. I wonder why they removed that feature
| 85392_school wrote:
| Search was not removed from ChatGPT, although it can be glitchy
| at times.
| BryantD wrote:
| Also not all models support it. I think only gpt-4o and
| gpt-4o-mini support it, although I haven't triple checked
| that.
| newfocogi wrote:
| ChatGPT can search the web (I just checked in my account). It
| appears to be available for all users (not just paid):
| https://help.openai.com/en/articles/9237897-chatgpt-search
| Brusco_RF wrote:
| I definitely tried to web search with ChatGPT a few weeks ago
| and it couldn't. I don't think I'm making this up. Unless I
| suffered a TBI.
| callamdelaney wrote:
| It told me it can't search the web and then proceeded to
| search the web
| cactusplant7374 wrote:
| About half my requests end up going to web search. But if
| you ask it for something specific like "find an X-ray image
| with an abnormality," then it refuses.
| simonw wrote:
| Web search is available for some but not all of their
| models. It is not particularly obvious from their UI which
| models have this feature.
| beng-nl wrote:
| As sibling said, search was nog removed. But not all models can
| use the web search, maybe that is causing your perception?
| danirogerc wrote:
| Couldn't a lot of front-ends using Claude API do this already?
| What's new?
| dcre wrote:
| If that's true, they are using a separate search API to get
| search results and feed it into a regular Claude API call. The
| difference here is that Anthropic is integrating it directly,
| like OpenAI and Google have. It doesn't look like it's in the
| API yet, but presumably that's coming. Then, as with gpt-4o and
| the Gemini models, you can make a single API call and it will
| do the searching for you and incorporate the results.
| simonw wrote:
| This is a new product feature for https://claude.ai and the
| Claude mobile and desktop apps.
| quinncom wrote:
| Yes, via MCP integrations.
| https://www.anthropic.com/news/model-context-protocol
| NBJack wrote:
| I wonder if it will actually respect the robots.txt this time.
| micromacrofoot wrote:
| almost no one does, robots.txt is practically a joke at this
| point -- right up there with autocomplete=off
| Demiurge wrote:
| In what circles is it a joke? Google bots seem to respect it
| on my sites according to logs.
| otikik wrote:
| Apparently, the regular search crawler does it, but the ai
| thingie doesn't.
| supriyo-biswas wrote:
| Huh? You can add Google-Extended[1] to opt out from
| Generative AI summaries.
|
| [1] https://blog.google/technology/ai/an-update-on-web-
| publisher...
| micromacrofoot wrote:
| Google will still scrape it for training data either way,
| this only impacts search results.
| supriyo-biswas wrote:
| > Today we're announcing Google-Extended, a new control
| that web publishers can use to manage whether their sites
| help *improve Bard and Vertex AI generative APIs*,
| including future generations of models that power those
| products.
| micromacrofoot wrote:
| https://www.theverge.com/news/630079/openai-google-
| copyright...
|
| they're literally asking to break _laws_ to train AI for
| _national security_. A sentence in a press release from 2
| years ago is worthless... look at what they 're actually
| doing
| lucgagan wrote:
| Can confirm. My website is flooded with AI bots despite
| attempts to block crawlers to certain parts of it.
| mediumsmart wrote:
| I know an artist that had noindex turned on by mistake in
| robots.txt for the last 5 years - google, kagi and
| duckduckgo find tons of links relevant to the artist and
| the artwork but not a single one from the website.
|
| so not _seem to_ or _apparently_ but matter of fact like.
| robots.txt works for the intended audience
| ysavir wrote:
| AI crawlers are part of the intended audience.
| Aloisius wrote:
| Not being indexed is different from not being crawled.
| joecool1029 wrote:
| It's in a small circle of those that do. Blame the internet
| archive for starting this trend:
| https://blog.archive.org/2017/04/17/robots-txt-meant-for-
| sea...
| hirako2000 wrote:
| Given websites do disappear or worse, get their content
| adultered. Also given the long history of the internet
| archive as a non profit - and the commons service it has
| served so far, the joke would be to see that bot honor
| it.
| nikisweeting wrote:
| lol IA did not start that, if anything they were late to
| the game. only the top handful of US-based search engines
| ever bothered respecting it in the first place
| micromacrofoot wrote:
| A small number of search engines respect it, no one else
| does. Just about every content scraping bot ignores it,
| including a number of Google's.
| geekrax wrote:
| I have replaced all robots.txt rules with simple WAF rules,
| which are cheaper to maintain than dealing with offending
| bots.
| Joe_Cool wrote:
| For my personal stuff I also included a Nepenthes tarpit.
| Works great and slows the bots down while feeding them
| garbage. Not my fault when they consume stuff robots.txt
| says they shouldn't.
|
| I'm just not sure if legal would love me doing that on our
| corporate servers...
| rustc wrote:
| The WAF rule matches based on the user agent header?
| Perplexity is known to use generic browser user agents to
| bypass that.
| claudiulodro wrote:
| I do essentially both: robots.txt backed by actual server-
| level enforcement of the rules in robots.txt. You'd think
| there would be zero hits on the server-level blocking since
| crawlers are supposed to read and respect robots.txt, but
| unsurprisingly they don't always. I don't know why this
| isn't a standard feature in web hosting.
| explain wrote:
| robots.txt is meant for automated crawlers, not human-driven
| actions.
| zupa-hu wrote:
| Every automated crawler follows human-driven actions.
| gopher_space wrote:
| Welcome to "Context".
| josh-sematic wrote:
| Conversely, every browser is a program that automatically
| executes HTTP requests.
| bayindirh wrote:
| Yet they respect a lot of things meant for machine to
| machine interaction. Like server return codes, cookie
| negotiations, and CAPTCHAs if they behave a certain way.
|
| So they sometimes hit bollards and turnstiles made for
| other types of code which executes HTTP requests. So
| they're bots basically, but better (or suitably) behaving
| ones.
| soulofmischief wrote:
| Browsers let you visit websites without regard for
| robots.txt.
| nicce wrote:
| It must form the search index somehow. That is prior the
| human action. Simply it would not find the page at all if it
| respects.
| pests wrote:
| I remember in late 90s/early 2000 as a teen going to
| robots.txt to specifically see what they were trying to
| hide and exploring those urls.
|
| What is the difference if I use a browser or a LLM tool (or
| curl, or wget, etc) to make those requests?
| kevindamm wrote:
| careful, some of those are honey pots or trip wires
| nicce wrote:
| But how did you find those sites that had the robot.txt
| to begin with? LLM must somehow find the existence of
| those pages and store that information before they can
| crawl them further or mark as acceptable source.
| Tostino wrote:
| Let's say you had a local model with the ability to do tool
| calls. You give that llm the ability to use a browser. The
| llm opens that browser, goes to Google or Bing, and does
| whatever searches it needs to do.
|
| Why would that be an issue?
| dudeinjapan wrote:
| In practice, robots.txt is to control which pages appear in
| Google results, which is respected as a matter of courtesy,
| not legality. It doesn't prevent proxies etc. from accessing
| your sites.
| postexitus wrote:
| if a human triggers the web crawlers by pressing a button,
| should they ignore robots.txt?
| Filligree wrote:
| If a human triggers a browser by pressing a button, should
| it ignore robots.txt?
| haswell wrote:
| Are you arguing that these are equivalent actions?
|
| The entire web was built on the understanding that humans
| generally operate browsers, and robots.txt is
| specifically for scenarios in which they do not.
|
| To pretend that the automated reading of websites by AI
| agents is not something different...is quite a stretch.
| Tostino wrote:
| I see it as _very_ different. I the human want the data
| from that request. I am using a tool to get it for me.
|
| Should I not be able to execute curl to download a
| webpage because the "understanding that humans generally
| operate browsers"?
| haswell wrote:
| > _I the human want the data from that request. I am
| using a tool to get it for me._
|
| Isn't this a bit of an oversimplification, though?
| Especially when the tool you're using completely alters
| the relationship between the content author and the
| reader?
|
| I hear this argument often: "it's just another tool and
| we've always used tools". But would you acknowledge that
| some tools change the dynamics entirely?
|
| > _Should I not be able to execute curl to download a
| webpage because the "understanding that humans generally
| operate browsers"?_
|
| Executing curl to download a webpage is nothing new, and
| compared to a traditional browser, has about the same
| impact. This is still drastically different than asking
| an AI agent to gather information and one of the pages it
| happens to "read" is the one you were previously
| navigating to with a browser or downloading with curl.
|
| If you're a content creator who built a site/business
| based on a pre-LLM understanding of the dynamics of the
| ecosystem, doesn't it seem reasonable to see these types
| of "readers" differently?
| bayindirh wrote:
| So, do you mean LLMs are human-like and conscious?
|
| I thought they were just machine code running on part GPU and
| part CPU.
| Filligree wrote:
| There's a human using the LLM. In a live web browsing
| session like this, the LLM stands in for the browser.
| timdiggerm wrote:
| Would you believe that humans turn on traditional web-
| crawlers as well?
| Ukv wrote:
| I think they mean that it's a tool accessing URLs in
| response to a user request to present to the user live -
| with that user being a human. Like if you used some webpage
| translation service, or non-ML summarizer.
|
| There's some gray area though, and the search engine
| indexing in advance (not sure if they've partnered with
| Bing/Google/...) should still follow robots.txt.
| aaronbaugher wrote:
| Yeah, that seems to be a big distinction. If I tell my AI
| to summarize the headlines from my three favorite news
| sites every morning, it's just carrying out my request
| same as if I'd clicked to them, so that seems fine.
|
| But if I say, "Search the web for a low-carb chicken
| casserole recipe that takes squash and cottage cheese,"
| then it's either going to A) send queries to a search
| engine like Google, in which case robots.txt already
| should have been respected, or B) check its own
| repository of information it's spidered before I asked
| the question, in which case it should have respected
| robots.txt itself.
| creddit wrote:
| I don't think it should. If a user asks the AI to read the web
| for them, it should read the web for them. This isn't a vacuum
| charged with crawling the web, it's an adhoc GET request.
| internetter wrote:
| You could make this justification for _a lot_ of unapproved
| bot activity.
| bayindirh wrote:
| How can you be so sure? Processors love locality, so they
| fetch the data around the requested address. Intel even used
| to give names to that.
|
| So, similarly, LLM companies can see this as a signal to
| crawl to whole site to add to their training sets and learn
| from it, if the same URL is hit for a couple of times in a
| relatively short time period.
| mvdtnz wrote:
| No thank you, when I define a robots.txt file I expect all
| automated systems to respect it.
| TheDudeMan wrote:
| But this isn't automated. This is user-driven.
| jrflowers wrote:
| If this feature isn't already part of the Claude API it
| likely will be at some point, in which case many Claude
| requests will be automated with no way to distinguish
| between user-driven or otherwise.
| pixl97 wrote:
| Simply put, at the end of the day you lose, AI blocking
| will not work.
|
| I mean, currently the AI request comes from the
| datacenter running the AI, but eventually one of two
| things will happen.
|
| AI models will get small/fast enough to run on user
| hardware and use the users resources: End result? You
| lose. The user will set their own headers and sites will
| play the impossible game of identifying AI.
|
| AI sites will figure out how to route the requests via
| any number of potential methods so the requests appear to
| come from the user anyway: End result? You lose. The
| sites attempting to block will play the cat and mouse
| game of figuring out what is AI or not AI.
|
| Note, this doesn't mean AI blocking isn't worth doing, if
| nothing else to reduce load on the servers. It's just not
| a long term winning strategy.
| ipaddr wrote:
| Welcome to the world of CAPTCHAs
| pixl97 wrote:
| Heh, I'm always reminded of dunkey when captchas are
| brought up. It seems AI gets better faster at them than
| humans do.
|
| https://www.youtube.com/watch?v=WqnXp6Saa8Y
| jimbokun wrote:
| Depends if the legal system survives.
|
| You may not be able to stop AIs from crawling web sites
| through technological means. But you can confiscate all
| the resources of the company that owns the AI.
| jimbokun wrote:
| It's clearly not. The human user is not requesting the
| resource. The AI is.
| beeflet wrote:
| Someone should call the robots.txt police then, there's a
| bandit on the loose!
| victorbjorklund wrote:
| A browser is automated too.
| goatlover wrote:
| Browser don't automatically browse, unless they are being
| automated.
| Sargos wrote:
| Any AI tool I make will ignore robots.txt on principle.
| Artificial humans should have equal rights as real humans.
| creddit wrote:
| > Artificial humans should have equal rights as real
| humans.
|
| This is ridiculous and plain evil.
| rvense wrote:
| People like you are ruining the internet.
| 1shooner wrote:
| >You can now use Claude to search the internet to provide
| more up-to-date and relevant responses.
|
| It's a search engine. You 'ask it to read the web' just like
| you asked Google to, except Google used to actually give the
| website traffic.
|
| I appreciate the concept of an AI User-agent, but without a
| business model that pays for the content creation, this is
| just going to lead to the death of anonymously accessible
| content.
| darepublic wrote:
| Well I expect eventually the agent will be able to act on
| your behalf with your credentials.
| elefanten wrote:
| And as advertisers get declining human views on their
| ads, the value of the business model will dwindle until
| it needs to be replaced by other forms of revenue.
| Content that can't shift business models and requires
| revenue to continue will die off.
|
| Edit: Maybe that's fine, maybe that's bad. Maybe new
| models will emerge and things will reshape. But I'm just
| supporting the case that AI agents will pressure the
| current "free" content economy.
| beeflet wrote:
| the free content economy is bogus, I am part of a growing
| segment of users that just block ads anyways.
| disiplus wrote:
| I'm also and I pay for the services that I use to not see
| ads, but I don't pay for every single one. For example a
| local classified website is financed by ads, and I don't
| think anybody will pay for just looking at stuff there.
| Maybe they can switch to the model where the person
| puting the thing for sale would pay but hat is something
| where we are not currently.
| Alupis wrote:
| If that's the case, then you might as well just list it
| on eBay and skip the local
| classifieds/craigslist/facebook/whatever.
|
| Is that a world we actually want?
| jimbokun wrote:
| Which is fine if you're paying for a subscription. Will
| probably soon see one subscription rate allowing AI
| access on your behalf, and a lower rate without that
| access. Since a human accessing without a bot is likely
| to see the ads.
| losteric wrote:
| What was the web like before wide spread internet ads,
| auth, and search engines?
|
| Did all those old sites have "business models"? What did
| the web feel like back then?
|
| (This is rhetorical - I had niche hobby sites back then, in
| the same way some people put out free zines, and wouldn't
| give a damn about today's AI agents so long as they were
| respectful.
|
| The web was better back then, and I believe AI slop and
| agents brings us closer to full circle)
| jfim wrote:
| It was much smaller and people wrote on Usenet to connect
| with one another, not to shout in the void while
| corporations hoover all the content.
| a4isms wrote:
| I recall a hotelier who advertised free WiFi at a time
| when everyone thought that monetizing WiFi was a hot new
| revenue stream.
|
| "What," he was asked, "is the business model for free
| WiFi?"
|
| "What," he retorted, "is the business model for free
| washrooms?"
| pixl97 wrote:
| The web was so much smaller back then. Just imagine I
| turned the user (not automated in any way) based clicks
| that can occur from a link like reddit today towards your
| site then. We called it the slashdot effect way back, but
| that many clicks might take down the entire ISP.
|
| Many of these sites business model was simply "don't cost
| too much". The moment the web got big a lot of these
| sites died. Now add DDOS for fun and profit became a
| thing, most people moved to huge advertising based
| providers/hosters (think FB).
|
| Simply put, we're never getting the old web back. Now, we
| may get something new, but it will be different and still
| far more commercial.
| beeflet wrote:
| IDK bittorrent is pretty effective at hosting bytes. I
| think if something like IPFS takes off in our generation
| there will be no need for advertising as an excuse for
| covering hosting costs in the client-server model.
|
| As for funding "content creation" itself, you have
| patronage.
| usrbinbash wrote:
| > This isn't a vacuum charged with crawling the web, it's an
| adhoc GET request.
|
| Doesn't matter. The robots-exclusion-standard is not just
| about webcrawlers. A `robots.txt` can list arbitrary
| UserAgents.
|
| Of course, an AI with automated websearch could ignore that,
| as can webcrawlers.
|
| If they chose do that, then at some point, some server admins
| might, (again, same as with non-compliant webcrawlers), use
| more drastic measures to reduce the load, by simply blocking
| these accesses.
|
| For that reason alone, it will pay off to comply with
| established standards in the long run.
| renewiltord wrote:
| In the limit of the arms race it's sufficient for the robot
| to use the user's local environment to do the browsing. At
| that point you can't distinguish the human from the robot.
| scoofy wrote:
| Many if not most websites are paid for by _eyeballs_ not by
| get requests. A bot is a bot is a bot. Respect robots.txt or
| expect to have your IPs banned.
| danenania wrote:
| It may not be very long before the big majority of web
| searches are via AI. If that happens, blocking AI will mean
| blocking most people too.
|
| You'd already be blocking me as I'd guess I now search via
| AI >90% of the time between perplexity, chatgpt, deep
| research, and google search AI.
| scoofy wrote:
| >It may not be very long before the big majority of web
| searches are via AI. If that happens, blocking AI will
| mean blocking most people too.
|
| If that happens a big majority of websites _will go
| bankrupt_ and won 't exist anymore to be searched.
| _Problem solved!_
| birken wrote:
| The AI isn't "reading the web" though, they are reading the
| top hits on the search results, and are free-riding on the
| access that Google/Bing gets in order to provide actual user
| traffic to their sites. Many webmasters specifically opt
| their pages out of being in the search results (via
| robots.txt and/or "noindex" directives) when they believe the
| cost/benefit of the bot traffic isn't worth the user traffic
| they may get from being in the search results.
|
| One of my websites that gets a decent amount of traffic has
| pretty close to a 1-1 ratio of Googlebot accesses compared to
| real user traffic referred from Google. As a webmaster I'm
| happy with this and continue to allow Google to access the
| site.
|
| If ChatGPT is giving my website a ratio of 100 bot accesses
| (or more) compared to 1 actual user sent to my site, I very
| much should have to right to decline their access.
| jsbg wrote:
| > If ChatGPT is giving my website a ratio of 100 bot
| accesses (or more) compared to 1 actual user sent to my
| site
|
| are you trying to collect ad revenue from the actual users?
| otherwise a chatbot reading your page because it found it
| by searching google and then relaying the info, with a
| link, to the user who asked for it seems reasonable
| birken wrote:
| While yes, I am attempting to collect ad revenue from
| users, and yes, I don't want somebody competing with me
| and cutting me out the loop, a large part of it is
| controlling my content. I'm not arguing whether the AI
| chatbot has the legal right to access the page, I'm not a
| legal scholar. What I'm saying is that the leading search
| engines also have the equal rights to access whatever
| content they want, and yet they all give webmasters the
| following tools:
|
| - Ability to prevent their crawlers from accessing URLs
| via robots.txt
|
| - Ability to prevent a page from being indexed on the
| internet (noindex tag)
|
| - Ability to remove existing pages that you don't want
| indexed (webmaster tools)
|
| - Ability to remove an entire domain from the search
| engine (webmaster tools)
|
| It is really impolite for the AI chatbots to go around
| and flout all these existing conventions because they
| know that webmasters _would_ restrict their access
| because it 's much less beneficial than it is for
| existing search engines.
|
| In the long run, all this is going to lead to is more
| anti-bot countermeasures, more content behind logins
| (which can have legally binding anti-AI access
| restrictions) and less new original content. The victim
| will be all humans who aren't using a chatbot to slightly
| benefit the ones who are.
|
| And again, I'm not suggesting that AI chatbots should not
| be allowed to load webpages, just that webmasters should
| be able to opt out of it.
| mtkd wrote:
| Do really think LLM vendors that download 80TB+ of data over
| torrents are going to be labeling their crawler agents
| correctly and running them out of known datacenters?
| teh_infallible wrote:
| Apparently they use smart appliances to scrape websites from
| residential accounts.
| Arnt wrote:
| The ones I noticed in my logfiles behave impeccably: retrieve
| robots.txt every week or so and act on it.
|
| (I noticed Claude, OpenAI and a couple of others whose names
| were less familiar to me.)
| tylersmith wrote:
| It's a user agent not a robot.
| Y_Y wrote:
| Why not both?
| SoftTalker wrote:
| Maybe we need a new "ai.txt" that says "yes I mean you, ChatGPT
| et. al."
| jsheard wrote:
| If they don't comply with robots.txt, why would they comply
| with anything else?
| JadoJodo wrote:
| https://github.com/ai-robots-txt/ai.robots.txt
| verdverm wrote:
| Bluesky / ATProto has a proposal for User Intents for data.
| More semantics than robots.txt, but equally unenforceable.
| Usage with AI is one of the intents to be signaled by users
|
| https://github.com/bluesky-
| social/proposals/tree/main/0008-u...
| noddleah wrote:
| https://thelibre.news/foss-infrastructure-is-under-attack-by...
| NewJazz wrote:
| Why wonder. You can test for yourself.
| furyofantares wrote:
| Presumably the crawler that produces whatever index it uses
| does, which is how it knows what sites to read. Unless you
| provide it a URL yourself I guess, in which case, it shouldn't.
| JimDabell wrote:
| The LLM shouldn't.
|
| robots.txt is intended to control _recursive_ fetches. It is
| not intended to block any and all access.
|
| You can test this out using wget. Fetch a URL with wget. You
| will see that it only fetches that URL. Now pass it the
| --recursive flag. It will now fetch that URL, parse the links,
| fetch robots.txt, then fetch the permitted links. And so on.
|
| wget respects robots.txt. But it doesn't even bother looking at
| it if it's only fetching a single URL because it isn't acting
| recursively, so robots.txt does not apply.
|
| The same applies to Claude. Whatever search index they are
| using, the crawler for that search index needs to respect
| robots.txt because it's acting recursively. But when the user
| asks the LLM to look at web results, it's just getting a single
| set of URLs from that index and fetching them - assuming it's
| even doing that and not using a cached version. It's not acting
| recursively, so robots.txt does not apply.
|
| I know a lot of people want to block any and all AI fetches
| from their sites, but robots.txt is the wrong mechanism if you
| want to do that. It's simply not designed to do that. It is
| only designed for crawlers, i.e. software that automatically
| fetches links _recursively_.
| firloop wrote:
| Any information on what search engine is powering it?
| goatmeal wrote:
| kagi already lets me use claude to search the web. how is this
| different?
| whalesalad wrote:
| kagi is searching the web for you, and then injecting the
| results into the context of the prompt.
| callamdelaney wrote:
| Are there any downsides to that approach? It seems like we're
| moving towards empowering llm's to interact with stuff as if
| that's better than us doing it for them - is it really?
|
| Eg say I want to build an agent to make decisions, shall I
| write some code to insert the data that informs the decision
| into the prompt, return structured data, and then write code
| to implement the decision?
|
| Or should I empower the llm do those things with function
| calls?
| visarga wrote:
| If you want deeper search, it needs to be able to iterate,
| plan, reason while searching.
| hombre_fatal wrote:
| Aside, does anyone know of an app like Perplexity for surfing the
| news in a foreign language (language practice)?
|
| Perplexity's "Explore" tab translates its news to your local
| language, and its curated news items are all pretty interesting,
| but the problem is that there are so few of them. I seem to get
| maybe a dozen stories in a day. I paid their subscription for a
| month just to listen to the news on my walk, but didn't renew
| because of this.
|
| A foreign news site like BBC Mundo (Spanish) on the other hand
| barely has any stories outside of a few niches. Its tech section
| only has a few stories per week.
|
| Hmm, maybe I want a sort of RSS reader that AI-translates stories
| for me. But I don't really want to maintain a feed myself either.
|
| Apple News would probably do it since they also have good
| curation, but afaict they still don't support foreign news
| sources (why???).
| dolmen wrote:
| Use a VPN to appear like being in the target country. Use a
| browser profile where you set the language preference to the
| one you target.
| hombre_fatal wrote:
| For which purpose?
| diggan wrote:
| > Apple News would probably do it since they also have good
| curation, but afaict they still don't support foreign news
| sources (why???).
|
| ground.news includes sources from all sorts of countries, and
| also auto-translate headline and the intro, while you can still
| click to access the source article. Not affiliated, just happy
| user.
|
| Example with sources in English, German and French:
| https://ground.news/article/accident-on-the-a13-in-the-yveli...
|
| Although I'm not sure how useful it is for language learning,
| as you cannot (afaik) configure it to only display articles in
| Spanish or something similar, but if you filter by stories
| about France, you'll get a lot of French sources (obviously).
| mvieira38 wrote:
| What's up with the geoblocking of Claude features? Not the first
| time it happens
| kasey_junk wrote:
| Different geographies have different legal requirements.
| artursapek wrote:
| ask the same geniuses who gave you the browser cookie popup re-
| implemented in a new way on every fucking website ever
| itpcc wrote:
| Now I understand why Gitlab was (is?) attacked[0] by those
| hideous bots.
|
| [0] https://news.ycombinator.com/item?id=43422413
| pcj-github wrote:
| Does not really say /how/ it's performing a web search... Is it
| tapping into it's "own" corpus of material or calling out to some
| other web search engine?
| punkpeye wrote:
| There is already a 100 ways of doing it using MCP
|
| https://glama.ai/mcp/servers?searchTerm=search
|
| What's the benefit of bringing native integration?
| TrueDuality wrote:
| The native app that allows for MCP is only available officially
| on Mac's and the web interface is generally more convenient for
| non-technical users. Searching and interacting with the web has
| become a table-stakes feature and was a glaring gap in Claude.
| punkpeye wrote:
| Let me rephrase it.
|
| MCP has the capability to add this functionality.
|
| It would be nice to see MCP getting adoption in their web UI,
| as well easier UX, rather than more ad hoc features being
| added natively.
| masterj wrote:
| This is likely implemented behind the scenes as an MCP
| server exposed to their model in the web UI. It is likely
| that they will enable MCP servers over HTTP+SSEs (vs the
| stdin/stdout used with Claude desktop) on the web version
| in the near future.
| light_triad wrote:
| Good news. I integrated Claude with a scrapper to get info from
| pages and it was not giving hallucinations 99% of the time. Hope
| this works out of the box now.
| blensor wrote:
| Funny thing is that I have the obsidian-mcp-tools installed and
| today claude-desktop just starting fetching stuff from the web
| through that because it exposes a fetch tool to claude.
|
| So this limitation is a bit arbitrary anyway.
| ProofHouse wrote:
| Awesome, but I also do want to say it's pretty sad it took this
| long straight up. Literally no excuse. But I'm glad they finally
| got to a feature that was launched more than a year ago on
| competitors.
| McNutty wrote:
| I haven't used Claude yet, but heard many good things. So I'm
| surprised to see that they're so far behind on this feature.
| ignoramous wrote:
| These are interesting times.
|
| It wasn't long ago that a uni senior who worked for a decade+ on
| Google Search told me that it was hopeless anyone tries to
| compete with Google not because it sees a tonne of signals that
| helps with IR but because of its in-house AI/ML.
|
| It turns out that the org that built the ultimate AI/ML that runs
| rings around anything that came before it for NLP (and thus IR)
| was a sister team at Google Translate.
|
| It isn't inconceivable that a kid might be able to build a
| Google-quality web search, scalability aside, on CommonsCrawls
| data in a weekend. As someone who built re-ranking algorithms for
| a search engine built atop Yahoo! and Wikipedia (REST/SOAP) APIs
| back in the late 2000s as a side project (and experienced the
| launch and subsequent iterations of Echo/Alexa up close at
| Amazon), the current capabilities (of even the open weight multi-
| modal models) seem too good to be true.
|
| Google itself though is saved by its enormous distribution
| advantages afforded by Chrome (3B to 5B users) and Android (3B+),
| aside from its search deals with Apple and other browser vendors.
| fourside wrote:
| I'll be interested in trying it. My admittedly limited experience
| with this on ChatGPT has been disappointing. ChatGPT falls for
| the SEO content that has taken over the web.
|
| As an example, I recently travelled abroad to a popular
| vacationing spot and asked ChatGPT for local recommendations on
| what to do. When it gave me answers directly, they were pretty
| solid. But when it "searched the web" instead, the answers were
| awful. Every single result it suggested had terrible ratings. It
| did this repeatedly. One of those times I asked it to pick
| something with better ratings and it sort of improved but not by
| much.
|
| Of course this is another tool and maybe Claude uses better
| sources or a better algorithm, but in this case where there was a
| concrete number tied to the results, that while not perfect, aims
| to rate the quality of a result, it still did not filter out low
| quality answers. I'm not sure I trust these LLMs to do any better
| when there aren't such ratings available. The available input
| data is just not very good, and now LLMs are being used to feed
| that low quality, SEO machine.
| mediumsmart wrote:
| thats great - the web he has been trained on or the one from
| Google?
| tttym wrote:
| It's like a line of platforms waiting for their own agents for
| web search
| rgbrgb wrote:
| Is there a way to access the new web browsing capability via API?
| simonw wrote:
| Not yet, and no hints as to whether that will happen or not.
| deadbabe wrote:
| Is no one concerned about LLMs just feeding people SEO ads as
| content?
| ConanRus wrote:
| finally
| jetrink wrote:
| > With web search, Claude has access to the latest events and
| information, boosting its accuracy on tasks that benefit from the
| most recent data.
|
| I'm surprised that they only expect performance to improve for
| tasks involving recent information. I thought it was widely
| accepted that using an LLM to extract information from a document
| is much more reliable than asking it to recall information it was
| trained on. In particular, it is supposed to lead to fewer
| instances of inventing facts out of thin air. Is my understanding
| out of date?
| not_good_coder wrote:
| I have found that for RAG use cases where the source can be
| document or web data, hallucinations can still occur. This is
| largely driven by the prompt and alignment to the data
| available for processing and re-ranking.
| tantalor wrote:
| It says a lot about their product vision and intended market that
| the example query is typescript migration question.
|
| Do they not care about typical search users? Only developers?
| Matl wrote:
| I'd guess they showed that query because LLMs are a lot better
| at answering translation/migration type stuff without
| hallucinating too much.
| dontlikeyoueith wrote:
| That's because the attention mechanism was designed for
| Seq2Seq models (i.e. translation in its most general form).
|
| Any other use of it is a case of "I have a hammer, so that's
| a nail".
| picafrost wrote:
| Do users pay for LLMs? I haven't seen much concrete data
| indicating that they do. I don't think the casual utility gains
| of LLMs have gotten average people so much value that they're
| paying $20/mo+ for it. Certainly not for search in the age of
| [doom] scrolling.
|
| I would guess that Anthropic wants developers talking about how
| good Claude is in their company Slack channels. That's the
| smart thing to do.
| pixl97 wrote:
| Have you actually done any kind of study on the utility the
| 'average user' has received, or is this just guessing?
| picafrost wrote:
| I have only anecdotal data from non-technical friends and
| family.
|
| I'm referring to _average people_ who may not be average
| users because they're barely using LLMs in the first place,
| if at all.
|
| They have maybe tried ChatGPT a few times to generate some
| silly stories, and maybe come back to it once or twice a
| month for a question or two, but that's it.
|
| We're all colored by our bubbles, and that's not a study,
| but it's something.
| disiplus wrote:
| I would say no. While I pay for chatgpt Claude and perplexity
| monthly (I don't know why anymore) my wife does not use any
| at all. She has around 5-10 things she uses on the
| smartphone, and if she needs something new there is still
| google.
|
| I on the other side reduced my googling by 95%
| beezle wrote:
| At this point it is probably easier to poison web pages for AI
| crawlers in a way that does not taint the human experience.
| ilaksh wrote:
| I've been using Tavily's search API for my MindRoot agents. Seems
| to work fairly well and much easier to set up than Google's
| search API.
|
| Anyone know if there is something better? I was thinking of
| trying Perplexity maybe.
| tgtweak wrote:
| Feels like a catch-up feature to chatgpt... honestly the biggest
| holdback for me on anthropic is the output token limit on
| sonnet... 8000 tokens max output is really limited (and 200k
| tokens in) compared to other offerings - especially considering
| that I suspect most sonnet users are not chat-users but api
| users.
| simonw wrote:
| The output limit of Claude 3.7 Sonnet Thinking is much higher
| (128,000): https://simonwillison.net/2025/Feb/25/llm-
| anthropic-014/#a-m...
| bfeynman wrote:
| why does perplexity exist anymore? They were one of anthropic's
| biggest customers and had been finetuning claude models for
| search for a while.
| NewJazz wrote:
| Bizarre that they choose to publish this right as a thread
| criticizing AI crawlers gets bumped off the front page.
|
| https://news.ycombinator.com/item?id=43422413
| tcdent wrote:
| Searching the web is a great feature in theory, but every
| implementation I've used so far looks at the top X hits and then
| interprets it to be the correct answer.
|
| When you're talking to an LLM about popular topics or common
| errors, the top results are often just blogspam or unresolved
| forum posts, so the you never get an answer to your problem.
|
| More of an indicator that web search is more unusable than ever,
| but interesting that it affects the performance of generative
| systems, nonetheless.
| Almondsetat wrote:
| >looks at the top X hits and then interprets it to be the
| correct answer.
|
| LLMs are truly reaching human-like behavior then
| yoyohello13 wrote:
| The longer I've been in the workforce, the more I realize
| most humans actually kind of suck at their jobs. LLMs being
| more human like is the opposite of what I want.
| _heimdall wrote:
| That could very well be because the jobs are effectively
| useless. By no means does that mean the people are, nor is
| what the income allows them to do. But most jobs sure do
| seem pointless.
| bravetraveler wrote:
| The work is _mysterious_ and _important_
| nimish wrote:
| David Graeber proven right every day.
| goatlover wrote:
| There's only one room I haven't been to yet, and today it
| had a name on it.
| TeMPOraL wrote:
| Username checks out. Go back to your department.
| pdimitar wrote:
| Both are true, separately in different situations. And
| sometimes both at the same time.
|
| There are a lot of useless employees out there. So, so
| much.
|
| And a ton of bullshit jobs as well.
| Workaccount2 wrote:
| It's kinda like online games. Most people who play a game
| are not too great at it, a large subset is pretty good,
| and then it's smaller and smaller groups as the ability
| increases.
|
| At the top you get the people who are true pros, they
| write the books, the guides, they solve the hardest
| problems, and everyone looks up to them. But spin the
| wheel and get a random SWE to do some work? It's not
| gonna be far off from an random 1v1 lobby.
| PickledChris wrote:
| https://danluu.com/p95-skill/
|
| Continues to apply
| anal_reactor wrote:
| > And for games like Overwatch, I don't think improving
| is a moral imperative; there's nothing wrong with having
| fun at 50%-ile or 10%-ile or any rank. But in every game
| I've played with a rating and/or league/tournament
| system, a lot of people get really upset and unhappy when
| they lose even when they haven't put much effort into
| improving. If that's the case, why not put a little bit
| of effort into improving and spend a little bit less time
| being upset?
|
| Interesting read, but I feel like the author could've
| spent just one more minute on this sentence. How good you
| are at given activity often doesn't matter, because
| you're mostly going to encounter people around your own
| level. What I'm saying is, unless you're at the absolute
| top or the absolute bottom, you're going to have similar
| ratio of wins to loses regardless whether you're a pro or
| an amateur, simply because an amateur gets paired with
| other amateurs, while a pro gets paired with other pros.
| In other words, not being the worst is often everything
| you need, and being the best is pretty much unreachable
| anyway.
|
| This can be very well extended to our discussion about
| SWEs. As long as you're not the worst nor the best, your
| skill and dedication have little correlation with your
| salary, job satisfaction, etc. Therefore, if you know you
| can't become the best, doing bare minimum not to get
| fired is a very sensible strategy, because beyond that
| point, the law of diminishing returns hits hard. This is
| especially important when you realize that usually in
| order to improve on anything (like programming), you need
| to use up resources that you could use for something
| else. In other words, every 15 minutes spent improving is
| 15 minutes not spent browsing TikTok, with the latter
| being obviously a preferable activity.
| fragmede wrote:
| Wait, but I'm on ProgrammerTok to improve my skills while
| I'm waiting for my code to compile!
| hatthew wrote:
| Competitive games are complex. It's hard to be 95%
| percentile. There are _so many_ mistakes one can make,
| even if each individual mistake is unlikely, it 's likely
| that _a_ mistake will be made. I participate in Dota 2,
| and _literally everyone_ makes noticeable mistakes, even
| including tier 1 pro players and the top ranked pub
| players. I honestly find it amazing how good people are
| given how complex the domain is.
|
| Now scale that up 10x, because reality is at least an
| order of magnitude more complex than a video game.
| jimbokun wrote:
| Unfettered capitalism is pretty good at figuring out
| which is which. It's pretty core to Elon Musk's animating
| philosophy: get as many jobs as possible then see if
| there's any negative impact.
|
| Not as appropriate in a government setting where the
| impact goes far beyond personal profit and loss.
| exe34 wrote:
| if a CEO can run 7 companies and still play monopoly with
| the ship of government, then maybe CEOs aren't really
| that useful.
| wil421 wrote:
| He's the hype man, it's the band and the crew who are
| running the show.
| jimbokun wrote:
| Recent changes to Tesla's stock price suggest otherwise.
| _heimdall wrote:
| They would suggest that only if the primary reason behind
| valuation changes is company performance and not
| political sentiment.
| matwood wrote:
| The problem is defining negative impact and also timing.
| For example, I can stop doing backups and save time and
| money. There is zero negative impact right up until the
| point I need to use the backup, then the impact is
| catastrophic.
| jimbokun wrote:
| Sure. Another fallout of unfettered capitalism. Just with
| an indeterminate delay between cause and effect.
| Toenex wrote:
| Maybe we already have Universal Basic Income, you just
| need to have a pointless job to collect it.
| ornornor wrote:
| I never thought about it this way, but it does make
| sense.
| homarp wrote:
| https://news.ycombinator.com/item?id=36373190
| adverbly wrote:
| Do they suck at their jobs or do their jobs suck?
| shreezus wrote:
| This is why agentic AI will likely cause a cataclysim in
| white-collar labor soon. The reality is, a lot of jobs just
| need "OK" performers, not excellent ones, and the tipping
| point will be when the average AI is more useful than the
| average human.
| jimbokun wrote:
| The excellent performs are only one or two turns of
| Moore's law away from the OK ones.
| freehorse wrote:
| If Moore's law is applicable in such a case, that is.
| agrippanux wrote:
| I had a similar conversation with my CEO today - how does
| the incoming crop of college grads deal with the fact AI
| can do a lot of entry level jobs? This is especially
| timely for me as my son is about to enter college.
|
| So I ended up posing the question to Claude and the
| response was "figure out how to work with me or pick a
| field I can't do" which was pretty much a flex.
| ornornor wrote:
| Paradoxically, the hardest jobs to automate are physical
| jobs it seems. A white collar worker is threatened by AI,
| blue collar not as much. I can totally envision AI
| software engineers (they're already okay if you check
| their work), but as of yet there are no AI plumbers or
| mechanics. Maybe there won't be, given the costs
| associated why producing physical machines vs software
| ones.
| matwood wrote:
| Claude isn't wrong. The baseline for entry level has just
| risen. The problem isn't that it's risen (this happens
| continuously even before LLMs), but the speed at which it
| has increased.
| j_timberlake wrote:
| I expect that AI good enough to automate jobs will also
| be dangerously good at criminal activities.
|
| Governments will want to ban them, but there's just too
| much $$$ to be made from replacing employees, so things
| will get complicated fast.
| int_19h wrote:
| They are already good at criminal activities such as
| phishing. That bar is rather low, especially once you
| scale up (hitting 100 people and successfully scamming 1
| is still great ROI with cheap small models).
|
| But I don't see what governments can really do about it.
| I mean, sure, they can ban the models, but enforcing such
| a ban is another matter - the models are already out
| there, it's just a large file, easy to torrent etc. The
| code that's needed to run it is also out there and open
| source. Cracking down on top-end hardware (and note that
| at this point it means not just GPUs but high-end PCs and
| Macs as well!) is easier to enforce but will piss off a
| lot more people.
| wickedsight wrote:
| Ugh... I've been in IT for over a decade now and many of
| the vacancies I see, I don't consider myself/my CV good
| enough. Then I work with the people who get hired for these
| jobs and see how low they set the bar, even though their CV
| might tick all the boxes.
| CooCooCaCha wrote:
| People suck at intellectual tasks but for stuff like
| locomotion and basic planning we humans are geniuses
| compared to machines. There isn't a robot today that could
| get in a car, drive the the grocery store, pick stuff off
| the shelf, buy it, and bring it back home. That's so easy
| it's automatic for us.
| bobxmax wrote:
| It's quite odd that people think of hallucinations as a
| dealbreaker for LLMs. Have they ever even met a human
| being?
| nemo44x wrote:
| Ever heard the saying "good help is hard to find."? It's
| not bullshit, it really is.
| mavamaarten wrote:
| Oh yeah this is very much the case. Every time I ask ChatGPT
| something simple (thinking it'd be a perfect fit for an LLM,
| not for a google search) and it starts searching, I already
| know the "answer" is going to be garbage.
| NavinF wrote:
| Try their Deep Research or grok's DeepSearch. Both do many
| searches and read many articles over a couple of minutes
| lee-rhapsody wrote:
| The "Deep" search features hallucinate like crazy, I've
| found.
| osigurdson wrote:
| That is interesting. I have often been amazed at how good it
| is at picking up when to search vs use its weights. My
| biggest problem with ChatGPT is the horrendous glitchyness.
| Tycho wrote:
| It's good if it hits on high quality sources like ons.gov.uk
| magackame wrote:
| Google search is crap. It seems to be a sentiment among many
| HNers, but is it really that bad? I mostly use it for
| programming, so documentation/forums and it works out greatly.
| For some queries it even returns personal blogs (which people
| seem to bash google for not happening). Of course there are
| some queries that return purely AI blogspam, but reformatting
| the query with a bit more thought usually solves it. I wonder
| if that is a US thing? Do search results differ greatly based
| on the region?
| Beijinger wrote:
| Is google search bad? Click here to find ten reasons why it
| is bad and 10 reasons why you should still use it.
|
| Yes, it is that bad.
|
| Website of Nike? Website of Starbucks? Likely position number
| one.
|
| Every product, category etc., e.g. what rice cooker should I
| buy? Is diseased by link and affiliate spam. There is a
| reason why people put +reddit on search terms.
| rustc wrote:
| What kind of results does Kagi give for "What rice cooker
| should I buy?"? (Not asking you specifically, but if any
| Kagi user could compare.)
| nicoty wrote:
| Copied verbatim from the AI generated summary:
|
| To choose the best rice cooker, consider these factors:
|
| Top Brands: Zojirushi is often considered the best brand,
| with Cuckoo and Tiger as close contenders. Aroma is
| considered a good budget brand 1. Types: Basic on/off
| rice cookers: These are good for simple white or brown
| rice cooking and are usually affordable and easy to use
| 2. Considerations: When buying a rice cooker, also
| consider noise levels, especially from beeping alerts and
| fan operation 3. Specific Recommendations: Yum Asia Panda
| Mini Advanced Fuzzy Logic Ceramic Rice Cooker is
| recommended for versatility 4. Yum Asia Bamboo rice
| cooker is considered the best overall 5. Russell Hobbs
| large rice cooker is a good budget option 5. For one to
| two people, you don't need a large rice cooker unless
| cost and space aren't a concern 6. Basic one-button
| models can be found for under $50, mid-range options
| around $100-$200, and high-end cookers for hundreds of
| dollars 6. References What is the best rice cooker brand
| ? : r/Cooking - Reddit www.reddit.com The Ultimate Rice
| Cooker Guide: How to Choose the Right One for Your Needs
| www.expertreviewsbestricecooker.com Best Rice Cooker UK |
| Posh Living Magazine posh.co.uk Best rice cookers for
| making perfectly fluffy grains - BBC Good Food
| www.bbcgoodfood.com The best rice cookers for gloriously
| fluffy grains at home www.theguardian.com Do You Really
| Need A Rice Cooker? (The Answer Is Yes.) - HuffPost
| www.huffpost.com
| yunwal wrote:
| Just tried plain old Kagi search, it came up with cooks
| illustrated (good source, paid) and consumer reports
| (decent source, paid), which I was surprised by until I
| remembered that I had these "pinned", which means Kagi
| increases their rank. Third on the page was a condensed
| roundup of 8 listicles, 2 of which seemed decent (food
| and wine and some random blogger).
|
| With no pins, bon appetit (decent) and nbc news (would be
| fine if it wasn't littered with ads) were the top
| results. For NBC news, Kagi also marked the result with a
| red shield, indicating that it has too many ads/trackers.
|
| Which really goes to show that Kagi is great if you're
| really willing to shell out for better content. Having
| the ability to mark sources as trusted, or indicate that
| I've paid for premium sources makes a completely
| different side of the web searchable.
| lclc wrote:
| If you enter a question into Kagi, by default, you get a
| 'Quick Answer' (https://help.kagi.com/kagi/getting-
| started/index.html#quick-...) on the top (an AI-generated
| text answer before the search result). In this case, it
| tells me which factors to consider and some that are
| considered to be the best depending on the use case (all
| sources the AI used for the answer are linked below the
| answer).
|
| Then, the following two links appear as normal search
| results: https://www.bonappetit.com/story/best-rice-
| cookers and https://www.bbcgoodfood.com/review/best-rice-
| cookers (I don't know those websites, so I can't judge
| them).
|
| Followed by Listicles (a short-form writing that uses a
| list as its thematic structure). All just one entrance,
| in this case, Best rice cooker 2024: Top tried and tested
| models for perfect results expertreviews.com 9 Best Rice
| Cookers | The Strategist - New York Magazine nymag.com
| The 8 Best Rice Cookers of 2025, Tested and Approved -
| The Spruce Eats thespruceeats.com 6 Best Rice Cookers
| 2025 Reviewed - Food Network foodnetwork.com Best rice
| cookers 2025, tested for perfect grains - The Independent
| independent.co.uk 29 Rice cooker meals ideas | rice
| cooker recipes, cooking recipes... de.pinterest.com 43
| Crockpot ideas | cooking recipes, rice cooker recipes,
| cooker... de.pinterest.com
|
| Followed by Quick Peek (questions with hidden answers
| that you can display).
|
| Followed by normal search results again: ryukoch.com,
| reddit/r/Coooking, expertreviewsbestricecooker.com,
| tiktok, and then many more 'normal' websites.
|
| This search reminded me that I have yet to configure my
| Kagi account to ignore tiktok.
| davidcbc wrote:
| "what rice cooker should I buy" returns a pretty in depth
| article on bonappetit.com and reddit as the top results,
| both recommending Zojirushi
| webstrand wrote:
| Well first Zojirushi is unnecessarily expensive and
| difficult to clean. Only if you need its fancy options
| and like multiple varieties of rice would I recommend it.
| Reddit is no panacea to spam, these days.
|
| But bonappetit.com is exactly an example of affiliate
| link spam. Even their budget option is awful.
| terribleperson wrote:
| Zojirushi being difficult to clean is kind of a wild
| take, unless most of their other rice makers are a wild
| departure from mine.
| TimorousBestie wrote:
| Yeah, never had any trouble cleaning any of my Zojirushi
| equipment.
| davidcbc wrote:
| What kind of answer are you expecting to get? Zojirushi
| is the answer you're going to get on the internet if you
| ask "what rice cooker should I buy" with no other
| qualifications because it's pretty universally agreed
| upon that it's the highest quality product.
| chongli wrote:
| My Zojirushi is the easiest to clean appliance I've ever
| owned. Just take the cooking bowl out and wash it with a
| damp cloth. Takes all of 30 seconds.
| wenc wrote:
| Yeah Zojirushi is absolutely the right answer so the
| contrarian take in this comment is actually not what I
| would want in search result.
|
| There are other good rice cookers like Cuckoo, and
| cheaper options like Tiger or Tatung, or really budget
| options like Aroma, but you pretty much can't go wrong
| with Zojirushi if you can afford it.
|
| This is a case of HN cynicism and contrarianism working
| against oneself.
| dontlikeyoueith wrote:
| > Well first Zojirushi is unnecessarily expensive and
| difficult to clean
|
| Expensive sure, but it's only difficult to clean if
| you're a double amputee.
| whstl wrote:
| _> Website of Nike? Website of Starbucks? Likely position
| number one._
|
| ...if you 're blocking ads and/or they're paying big
| advertisement bucks.
| bag_boy wrote:
| But who is creating honest articles about which rice
| cookers you should buy?
|
| BTW - the search you suggested gives you Reddit links first
| followed by other trusted sites trying to make an affiliate
| buck. There's no spam on the first page.
| dannyobrien wrote:
| I think part of the reason for this is that web site
| developers have got out of the habit of optimizing for
| search engines. I'm often surprised by how self-contained
| the requirements for a website are now, even among
| otherwise technically sophisticated clients. There'll be a
| beautiful site in React that absolutely sucks for SEO, but
| no-one will mind because a) it's unclear how big an
| audience there should be for the site, and b) the "all your
| hits come from search engines" was broken ten or more years
| ago by social network linking, so the question of how you
| get an audience seems much more arbitrary, and less
| connected to google.com.
| hansmayer wrote:
| I mean, for those of us who used it since way before the
| '20s, it's not really a sentiment - it's a fact. You used to
| be able to type in 3 words and whatever error message your
| stack trace was showing, and the first 3 links returned were
| very likely a definitive source to solving your
| problem.Written by a human, and believe my word for it - it
| was much better back then than the crap you get out of
| torturing whatever your LLM of choice is. However the weird
| MBas took it over to and did exactly what you are describing
| - forced people to spend more time "engaging with the
| platform" (to increase the revenue). As you can see, they
| seem to have achieved this goal, and we all now spend time
| reformatting our queries as they wanted us to, and yes Google
| search is complete crap.
| pixl97 wrote:
| Eh, this said google is suffering from its own popularity.
|
| Google in the past was written by a human because that was
| really the only option. Once other humans figured out to
| how automate producing trace Google has gone downhill
| simply because of the bullshit asymmetry effect. Even if
| google was totally customer based, it would still be much
| worse than the past because of the total amount of crap
| that exists.
|
| This is also why no other competitor just completely blows
| them away either.
| lukan wrote:
| "and believe my word for it - it was much better back then
| than the crap you get out of torturing whatever your LLM of
| choice is"
|
| I was around as well and my memories do not confirm this.
| But google search definitely degraded a lot.
| reddalo wrote:
| Yes and no. You used to find niche websites more easily,
| but I vividly remember the frustration with
| ExpertsExchange results (with answers that were all
| paywalled).
| simonw wrote:
| How long have you been using Google search for?
|
| It used to be SO much less likely to return junk.
| magackame wrote:
| Since around 2012. What year would be the golden age of
| google search? I wonder if anyone has archived search
| result pages for relatively timeless queries so that we
| could compare. Wayback Machine seems to archive some of
| them.
|
| https://web.archive.org/web/20200801000000*/https://www.goo
| g...
|
| https://web.archive.org/web/20200801000000*/https://www.goo
| g...
| ipaddr wrote:
| Around 2011 to 2012 after the first of many updates with
| names like hurricanes came and washed away the good.
| tcdent wrote:
| Google search was actually great between the period where
| pagerank successfully defeated old-school SEO tactics and
| banner advertising starting earning enough that the
| "bloggers" could pay cheap writers to pad their articles in
| convincing ways.
|
| First decade of the 2000's if I had to guess.
| qingcharles wrote:
| 2006 was the first year I remember paid blog posts
| appearing from content farms that would exist only to
| increase your inbound links and page rank. Those days
| companies were paying cents per post to get their sites
| to #1 in Google while Google just wagged their finger and
| said "naughty, naughty."
|
| It's a shame, because Page Rank was a smart idea.
| harrall wrote:
| I watched one of my friends who says Google is useless use
| Google one day.
|
| If I were looking for a song, I would type in something like
| "song used at beginning of X movie indie rock"
|
| He would type in "X songs."
|
| I basically find everything in Google in one search and it
| takes him several. I type in my thought straight whereas he
| seems to treat Google like a dumb keyword index.
| layer8 wrote:
| Interesting, because keyword search works quite well for
| me, and I assumed it was natural-language searchers who are
| getting worse results.
| TeMPOraL wrote:
| Google _used to be_ a "dumb keyword index" in the past. It
| worked better that way. You had some modicum of control
| over the matching process. For the past 10 years or so,
| Google turned more into "try to guess what a novice normie
| means", which removes user control (no more _actually
| working_ verbatim search or logical operators...), and...
| well I failed to develop a mental model of how exactly it
| works. It 's not a proper keyword search anymore, and it's
| not a proper DWIM system with true understanding of natural
| language like LLMs are. It's something... in between,
| inferior to both.
|
| Actually, typing out "what a novice normie means" made me
| realize what is the probable reason Google turned out the
| way it is: optimizing for new users. However, a growing
| userbase means most users are _new to Internet in general_
| , and (with big enough growth) most queries are issued by
| people who are trying a search engine out for the first
| time, and have no clue how or why it works - and those
| queries are exactly the kind of queries Google is now good
| at, queries like example you provided.
| kelseyfrog wrote:
| Search engines now have an incentive to offer a B2B search
| product that solves the blogspam problem. Don't worry, the AIs
| will get good search results, and you'll still get the version
| that's SEOed to the point of uselessness.
| HankWozHere wrote:
| Kagi Assistant allows you to do search with LLM queries. So far
| I feel it bears reliable results. For instance - I tried couple
| of queries for product suggestions and came back with some good
| results. Whilst it's a premium service , I find the offering to
| be of good value.
| hooli_gan wrote:
| Does it just start a search or does the chat continue with
| the results? Would be cool to continue the chat with result,
| which were filtered acording to the blacklist.
| chrisweekly wrote:
| Yeah, Kagi's search results are so much better than Google's,
| it defies comparison.
| rglover wrote:
| Just switched my default to Kagi based on this comment and
| you're right. It honestly feels like old-school Google
| before all of the algo changes.
| abtinf wrote:
| You were paying for Kagi but not using it as the default?
| rglover wrote:
| No, I only noodled with the free/trial searches before,
| but this reminded me to pay and make it my default.
| eli wrote:
| It's neat but I've found the value kinda variable. It seems
| heavily influenced by whatever the first few hits are for a
| query based on your question, so if it's the kind of question
| that can be answered with a simple search it works well. But
| of course those are the kinds of questions where you need it
| the least.
|
| I find myself much more often using their "Quick Answer"
| feature, which shows a brief LLM answer above the results
| themselves. Makes it easier to see where it's getting things
| from and whether I need to try the question a different way.
| wongarsu wrote:
| The quick answer (ending searches in a question mark) also
| seems pretty resilient to hallucinations. It prefers
| telling you that something wasn't mentioned in the top
| search results over just making something up
| KoolKat23 wrote:
| I have a subscription, please could I ask how you do this? I
| only know of the append ? Feature.
| enterpriss wrote:
| https://kagi.com/assistant
| KoolKat23 wrote:
| Thanks
| arandomusername wrote:
| You need their ultimate plan
|
| https://kagi.com/assistant
| KoolKat23 wrote:
| oh okay, thanks! (too rich for my blood haha).
| hibikir wrote:
| Imagine how much fun it will be when the breakthrough in search
| engine quality comes from companies building a better engine to
| get good LLM answers.
|
| This is ultimately google's problem: They are making money from
| the fact that the page is now mostly ads and not necessarily
| going to lead to a good, quick answer, leading to even more
| ads. They probably lose money if they make their search better
| GraffitiTim wrote:
| Exa (YC S21) is trying to solve this problem by re-indexing the
| web in an LLM-friendly way.
| ipaddr wrote:
| 2021? how are they doing?
| blackeyeblitzar wrote:
| For me LLMs have basically removed any need to visit search
| engines. I was already not using Google due to how bad its
| interface had become, but I feel like LLMs at least are more
| efficient as an interface even if they're still looking at the
| same blogspam or unresolved forum posts. My anecdotal
| experience though, is that I get better answers from LLMs,
| perhaps because I am able to give them really detailed prompts
| that seem to improve the answers based how specific I get.
| Generic search engines don't seem to do that, in my experience.
| wenc wrote:
| I just tried Claude's web search. It works pretty well.
|
| I'm not sure if Claude does any reranking (see Cohere Reranker)
| where it reorders the top n results or just relies on Google's
| ranking.
|
| But a web search that does re-ranking should reduce the amount
| of blogspam or incomplete answers. Web search isn't inherently
| a lost cause.
| PStamatiou wrote:
| Could not agree more. I wrote in detail about some of these
| issues last week https://paulstamatiou.com/browse-no-more
| johntb86 wrote:
| I've found that OpenAI's Deep Research seems to be much better
| at this, including finding an obscure StackOverflow post that
| solved a problem I had, or finding travel wiki sites that
| actually answered questions I had around traveling around
| Poland. However it finds its pages, they're much better than
| just the top N Google results.
| dontlikeyoueith wrote:
| They're probably doing RAG on a huge chunk of the internet,
| i.e. they built their own task-specific search engine.
| wongarsu wrote:
| Grok's DeepSearch and DeeperSearch are also pretty good, and
| you can look at their stream of thought to see how it reaches
| its results.
|
| Not sure how OpenAIs version works, but grok's approach is to
| do multiple rounds of searches, each round more specific and
| informed by previous results
| labrador wrote:
| My disgust and hatred for Elon Musk prevents me from giving
| Grok a fair chance. I realize this is my psychological
| problem. I own it, but as far as I can tell, I'm not
| missing much.
| wongarsu wrote:
| Purely on its technical merits Grok is pretty good and
| fills a niche in the selection of AI agents. But I can
| absolutely understand not wanting to use an AI owned by
| somebody who makes Nazi salutes and is dismantling the US
| government.
| labrador wrote:
| I'm positive there are great people working at X, xAI,
| Tesla and SpaceX who are suffering every day through no
| fault of their own, hoping that Musk will come to his
| senses. Tesla right now is an especially tragic case for
| those whose livelihood depended on it doing well.
| int_19h wrote:
| The irony is that, for all Musk's boasts about how it is
| "based", Grok itself doesn't share Musk's ideology.
|
| I did a little experiment when Grok 3 came out, telling
| it that it has been appointed the "world dictator" and
| asking it to provide a detailed plan on how it would
| govern. It was pretty much diametrically opposite of
| everything Musk is doing right now, from environment to
| economics (on the latter, it straight up said that the
| ultimate goal is to "satisfy everyone's needs", so it's
| literally non-ironically communist).
| matwood wrote:
| I'm glad you mentioned this. I asked Deep Research to lay out
| a tax strategy in a foreign country and it cited a ton of
| great research I hadn't yet found.
| colordrops wrote:
| Ugh, what a nightmare, now search engines are going to start
| optimizing for bots.
| taude wrote:
| Do you think that if it's a non-Google company, that maybe
| doesn't rank search by ad payment $$$, that this new company
| could in theory do a better job?
| Xenoamorphous wrote:
| Is there any viable alternative to pass knowledge to the LLMs
| that goes beyond their training cut off date?
| jonny_eh wrote:
| Via their context window, but new knowledge could easily fill
| it up.
| macrolime wrote:
| Deep search/deep research in grok, chatgpt, perplexity etc
| works much better. It can also do things like search in
| different languages. Wonder about something in some foreign
| country? Ask it to search in the local language and find things
| you won't find in English.
| wickedsight wrote:
| > Ask it to search in the local language and find things you
| won't find in English.
|
| Yeah, this is one of my favorite use cases. Living in Europe,
| surrounded by different languages, this makes searching stuff
| in other countries so much more convenient.
| oytis wrote:
| Well, they are professionals, they sure add "reddit" to every
| query.
| elliotrpmorris wrote:
| Lol so true
| lairv wrote:
| Overall LLMs (that I've tested) don't know how to use a search
| engine, their queries are bad and naive, probably because the
| way to use a search engine isn't part of training data, it's
| just something that people learn to do by using them. Maybe
| Google has the data to make LLMs good at using search engines
| but would it serve their business?
| osigurdson wrote:
| My experience with ChatGPT is really good. I find standard web
| searches very annoying now.
| CalChris wrote:
| I find myself Googling less often these days. Frustrated with
| both the poor search results and impressed with the quality of AI
| to do the same thing and more, I think search's days are
| numbered. AOL lasted as an email address for quite some time
| after America Online ceased to be a relevant portal. Maybe Gmail
| will as well.
| joshstrange wrote:
| Massive props to Anthropic for announcing a feature _and_ making
| it available for everyone right away.
|
| OpenAI is so annoying in this aspect. They will regularly give
| timelines for rollout that not met or simply wrong.
|
| Edit: "Everyone" = Everyone who pays. Sorry if this sounds mean
| but I don't care about what the free tier gets or when. As a
| paying user for both Anthropic and OpenAI I was just pointing out
| the rollout differences.
|
| Edit2: My US-bias is showing, sorry I didn't even parse that in
| the message.
| underdeserver wrote:
| It's not available for everyone.
| joshstrange wrote:
| > Web search is available now in feature preview for all paid
| Claude users in the United States.
|
| It is for all paid users, something OpenAI is slow on. I pay
| for both and I often forget to try OpenAI's new things
| because they roll out so slow. Sometimes it's same-day but
| they are all over the map in how long it takes to roll out.
| deivid wrote:
| For all paid users _in America_. It's not available for me
| in Europe.
| joshstrange wrote:
| Apologies, I updated my original comment, I missed that
| completely.
| DrammBA wrote:
| I think 'For all paid users in the United States' is
| clearer. I live in America, but not in what the United
| States considers 'America', so I do not get to use this
| new feature yet.
| mvdtnz wrote:
| You can't be serious with this reply. You simply can not.
| joshstrange wrote:
| Which part? I completely missed the "United States" part
| and have since updated my original comment.
| bryan0 wrote:
| > Web search is available now in feature preview for all paid
| Claude users in the United States. Support for users on our
| free plan and more countries is coming soon.
| AcquiescentWolf wrote:
| People outside the US obviously don't exist, therefore the
| statement is correct.
| mpalmer wrote:
| Easy to believe our weak privacy laws are part of the
| reason we get tech features first. Huzzah...
| willio58 wrote:
| > OpenAI is so annoying in this aspect. They will regularly
| give timelines for rollout that not met or simply wrong.
|
| I have empathy for the engineers in this case. You know it's a
| combination of sales/marketing/product getting WAY ahead of
| themselves by doing this. Then the engineers have to explain
| why they cannot in fact reach an arbitrary deadline.
|
| Meanwhile the people not in the work get to blame those working
| on the code for not hitting deadlines
| nilkn wrote:
| Many of OpenAI's announcements seem to be timed almost
| perfectly as responses to other events in the industry or
| market. I think Sam just likes to keep the company in the
| news and the cultural zeitgeist, and he doesn't really care
| if what he's announcing is ready to scale to users yet or
| not.
| wongarsu wrote:
| To be fair, being in the cultural zeitgeist is a huge part
| of their current moat. To people in the street OpenAI is
| _the_ company making LLMs. Sam has to make sure it stays
| that way
| g42gregory wrote:
| Anybody knows if this search will works inside Cursor? Do we need
| to do anything to make 3.7-thinking to search the web?
| nimish wrote:
| So what's perplexity's raison detre at this point?
| dalmo3 wrote:
| I open perplexity, I see a search box.
|
| I open claude, I see a big "Continue with Google" button.
| scudsworth wrote:
| "claude can now ddos random websites . . . more so"
| douglee650 wrote:
| You know what Claude can't do? Successfully vibe me a solution
| to:
|
| """ i need a bashrc command that will map the alias "logg" to
| open macvim to the file at ~/log.txt, then execute the macro
| defined by "<leader>z" """
|
| Note <leader>z ends with user in insert mode, Claude provides
| solution below but puts me in edit mode. (I still have to press
| "i")
|
| alias loggg='mvim ~/log.txt -c "normal \<leader>z"'
| simonw wrote:
| I don't find that prompt to be particularly clear.
| dcre wrote:
| And it did exactly what the prompt asked! For all it knows,
| the macro could put the user in the desired mode.
| hansmayer wrote:
| So, referring specifically to the example they show on the front-
| page, what value does this bring actually? The best example they
| could come up with is Typescript migration ? Really? Weren't the
| LLMs supposed to be a superior alternative to searching the web?
| Why do we need to produce more CO2 to do the same we could have
| done at the fraction of the cost, of course at the time when the
| google search was still working?
| simonw wrote:
| The CO2 concerns of using LLMs are massively overblown these
| days (with the exception of o1-pro and GPT-4.5 at least).
|
| The energy efficiency of most models has improved by an order
| of magnitude since the most widely cited CO2 usage papers were
| published.
|
| (It remains frustratingly difficult to get accurate numbers
| though: at this point I think more transparency would help
| rather than hurt the big AI labs)
| cube2222 wrote:
| A good recent in-depth article about this:
| https://andymasley.substack.com/p/individual-ai-use-is-
| not-b...
| wewewedxfgdf wrote:
| I'm waiting for Claude's API to support projects with file
| uploads like its web UI.
| ralusek wrote:
| Best models for search, in order:
|
| OpenAI Deep Research
|
| Grok Deep Search
|
| Gemini Deep Research
|
| Grok + Search
|
| Gemini + Search
|
| ChatGPT + Search
|
| These are just my opinions, but I do use this feature all the
| time. Haven't used Claude enough to get a sense of where it would
| fit in.
| danielbln wrote:
| Where does Perplexity Deep Research fit into this list?
| gcanyon wrote:
| Funny, I literally just two days ago asked Claude to provide an
| outline of the functionality of a product, giving it the web
| site. It of course refused. So I downloaded the text of the site
| and passed that in, and got mediocre results.
|
| The results based on giving the source URL directly were better.
| Still a bit generic and high-level and vague, as LLMs tend to be,
| but better than the text-download version a couple days ago. And
| of course much easier to generate!
| MattSayar wrote:
| I had tried using monolith [0] to feed webpages into Claude but
| all the html was too much token context. I ended up Print >
| Save as PDF-ing somewhat often and that worked pretty well. But
| just giving a URL is ideal.
|
| [0] https://github.com/Y2Z/monolith
| ForTheKidz wrote:
| Great, now we just need a decent search engine.
| artembugara wrote:
| Search the web is apparently using SERP.
|
| It's just breaks my head. We've build LLMs that can process
| millions of pages at a time. But what we give them is a search
| engine that is optimized for humans.
|
| It's like giving a humanoid robot access to a keyboard with a
| mouse to chat with another humanoid robot.
|
| Disclaimer: I might be biased as we're kind of building the fact
| search engine for LLMs.
| sadeshmukh wrote:
| No LLM can process millions of web pages. Maybe you're thinking
| of something else?
| braebo wrote:
| This is a problem I think about often. I'd be curious to know
| what kind of things you've learned / accomplished in that
| problem space so far.
| jsight wrote:
| I really want these to be able to find and even redisplay images.
| "Search all the hotels within 5 miles of this address and show me
| detailed pictures of the rooms and restrooms"
|
| Hotels would much rather show you the outside, the lobby, and a
| conference room, so finding what the actual living space will
| look like is often surprisingly difficult.
| dgs_sgd wrote:
| I've been looking for this as well. I want a reliable image
| search tool. I tried a combination of perplexity web search
| tool use with the Anthropic conversations API but it's been
| lackluster.
| tjsk wrote:
| I've been experimenting with different LLM + search combos
| too, but results have been mixed. One thing I'm particularly
| interested in is improving retrieval for both images and
| videos. Right now, most tools seem to rely heavily on
| metadata or simple embeddings, but I wonder if there's a
| better way to handle complex visual queries. Have you tried
| anything for video search as well, or are you mainly focused
| on images? Also, what kinds of queries have you tested?
| sylware wrote:
| What I really would like to know: do they use a web crawler with
| an AI strapped to the mouse and keyboard of a javascript-ed web
| engine?
| Taters91 wrote:
| so can I
| greatNespresso wrote:
| Woaw, so excited about this! Has anyone tried it out already?
| ineedaj0b wrote:
| i stopped using Claude about 2 months ago. went to Grok (the code
| was better, everything was better - politics aside). i wonder if
| this update will improve it.
|
| the main issue i find with Claude is, he fights you. He refuses
| so many requests and i need 3 or 4 replies to get what i want vs
| deepseek/grok. i've kept the monthly subscription to help
| anthropic, but it's trounced by the free options imo.
| Workaccount2 wrote:
| Claude 3.7 with thinking is a big step up. If you haven't used
| it I'd suggest giving it a try.
|
| I have used grok a bit and it did what I needed it too, so I
| can't really compare. But 3.7 thinking is crazy strong for
| coding.
| monkeydust wrote:
| Which frontier model provider let's me specific what websites to
| search and it will only search those?
| lihua919 wrote:
| great
| Heidaradar wrote:
| Kinda surprised it took them this long to add this feature, but
| glad it's here now
| jason_zig wrote:
| Curious - does anyone want this stuff?
| l33tbro wrote:
| I still don't get why Claude needs my phone number to sign up.
| Feels gross and is a such a shame, as their LLM seems great.
| nokun7 wrote:
| Honestly, while this is a great update and all, other AI
| platforms have had web search functionality for quite some time
| now. Any explanation for this delay?
|
| I wonder if Claude's API will match Perplexity's dynamic answers.
| Is there API rate limiting. If so, then the older API pricing
| would be preferable. Can users switch between the two?
| livingmylife01 wrote:
| This is insane. Guys you think it will someday replace google?
| cpersona wrote:
| SEO for LLMs looks to have a bright future.
| luxuryballs wrote:
| Time to ask it to find all the dirt on me?
| xingwu wrote:
| Hope there will be a tech blog regarding how you index and
| retrieve the pages.
| d--b wrote:
| Next: Claude can now ask random questions to strangers on Reddit.
___________________________________________________________________
(page generated 2025-03-20 23:00 UTC)