[HN Gopher] Techdirt has been deleted from Bing and DuckDuckGo [...
___________________________________________________________________
Techdirt has been deleted from Bing and DuckDuckGo [fixed]
Author : lehi
Score : 195 points
Date : 2023-07-27 18:37 UTC (4 hours ago)
(HTM) web link (www.techdirt.com)
(TXT) w3m dump (www.techdirt.com)
| flyinghamster wrote:
| It's looking like 2023 is the year that search dies. Between the
| removal of exclusion operators (hey, we saw you were adding
| -pinterest to your searches, but we want to make sure you get
| your pinterest!) and de-indexing of sites, it's looking like it's
| time for the search wheel to start making its third spin.
|
| Guess I need to start playing more with things like Algolia or
| Kagi.
| bufferoverflow wrote:
| Unfortunately, creating a search engine competitive with Google
| or even Bing is insanely expensive. Not just from the server
| cost perspective, but also from marketing it and convincing
| people to switch.
| thewataccount wrote:
| I feel like this makes it clear the level of reliance DuckDuckGo
| has on Bing. I know it's been a bit ambigious the full extent to
| which they do, and they claim to have their own indexer, but
| there's no way this is a coincidence. Surely if you have your own
| indexer you would have seen techdirt before?
|
| At this moment "techdirt" only returns the wikipedia article,
| twitter, then mostly unrelated mentions of it. Surely DDG would
| have seen at least their homepage before?
|
| > Most of our search result pages feature one or more Instant
| Answers. To deliver Instant Answers on specific topics,
| DuckDuckGo leverages many sources, including specialized sources
| like Sportradar and crowd-sourced sites like Wikipedia. We also
| maintain our own crawler (DuckDuckBot) and many indexes to
| support our results. Of course, we have more traditional links
| and images in our search results too, which we largely source
| from Bing. Our focus is synthesizing all these sources to create
| a superior search experience.
|
| https://duckduckgo.com/duckduckgo-help-pages/results/sources...
| Retric wrote:
| DDG's business model is largely about privacy and UI rather
| than attempting to be objectively better than other search
| engines. Search is not quite 1:1 with Bing, but frankly I hate
| Bing's UI not its results.
| pb7 wrote:
| They don't claim to be a privacy wrapper for Bing; they claim
| to be a standalone search engine. But they aren't. It's 99%
| Bing and there's nothing they can do about the results
| getting censored at the origin.
| Retric wrote:
| On multiple occasions they have uncensored results that
| Bing censored. They default to censoring websites when Bing
| does, but can trivially remove sites from that list.
|
| As to their businesses proposition privacy was central from
| day 1: _In 2011, Weinberg, then the company's sole
| employee, took out an ad on a billboard in San Francisco
| that declared, "Google tracks you. We don't." That branding
| --Google, but private--has served the company well in the
| years since.
|
| "The only way to compete with Google is not to try to
| compete on search results," says Brad Burnham, a partner at
| Union Square Ventures, which gave DuckDuckGo its first and
| only Series A funding in 2011._
| https://www.wired.com/story/duckduckgo-quest-prove-online-
| pr...
| pb7 wrote:
| In other words, you're seeing results that Bing wants you
| to see unless it raises a big enough stink for DDG to
| manually override. They default to being a wrapper for
| Bing.
|
| Edit: Since you edited, no one is arguing that their
| whole spiel isn't about privacy. What I'm saying is that
| they refuse to admit that they're a privacy wrapper
| around Bing which would make their value proposition
| quite weak. They puff their chest and pretend to be a big
| standalone player in the market but they aren't. You
| aren't at the whims of DDG as a user, but at those of
| Bing/Microsoft. If you're cool with that, carry on, but
| I'm not going to let yegg continue to make vague
| statements uncontested. If they're so sketchy and
| dishonest about their role in deciding the links, what
| else are they dishonest about? Some of you let a lot
| slide because they're the underdog.
| Retric wrote:
| That's not a question of being a wrapper, it's a question
| of using Bing's spam list. The difference is about sites
| Bing doesn't include but also doesn't specifically
| exclude.
|
| PS: As to your edit, they have specifically mentioned
| using Bing in the past alongside other engines. They are
| clearly running their own search process and widgets
| while also using a lot of data from Bing. I can see why
| you might consider it deceptive, though I really don't.
| thewataccount wrote:
| The real question is what extent do they index sites
| themselves?
|
| If they're manually adding a entry for "insert site
| here", maybe some "caching" of bing's results for the
| biggest most common searches then IMO that's still
| 99.999999% ~= 100%
| Retric wrote:
| Because an entire domain with thousands of sites can show
| up the simplest explanation is they are copying some Spam
| or Phishing etc list while also having an index they are
| searching.
|
| Now, they might be using low level API access to Bing's
| index or something else. But I don't see how it can be a
| simple wrapper.
| Brian_K_White wrote:
| A dinky little but long standing staple of it's topic
| bitchin100.com had the same problem for some years but I guess
| the community was small enough and traditional enough that not
| enough people were using anything but google to notice except
| me.
|
| They eventually got it fixed but only after a few other people
| finally noticed and the owner contacted someone and then waited
| a week or so. But it was like that for years. You search a
| perfectly good search term that should pull up one of the
| articles on that site first, and all you got are all kinds of
| other 2nd and 3rd hand references like email archive posts and
| articles on other sites, but scroll down as many pages as you
| want and the actual site would never come up on ddg.
|
| That's when I got interested in Kagi.
| sproketboy wrote:
| [dead]
| davb wrote:
| Whatever the complaints about DDG, I still commend yegg for
| engaging here. I can't imagine the head of Google Search or Bing
| doing the same.
| lapcat wrote:
| This keeps happening. Just a few of many examples:
|
| https://daverupert.com/2023/01/shadow-banned-by-duckduckgo-a...
|
| https://www.jessesquires.com/blog/2022/03/25/my-website-disa...
|
| https://lapcatsoftware.com/articles/bing.html
| KennyBlanken wrote:
| I find it simply impossible to believe that Masnick is so
| incompetent at running a website after so long, that he's finding
| out from random friends that his site has been de-listed from
| Bing instead of him or his staff having their search console
| lighting up like a dashboard and pinging them with emails. I get
| emails from both bing and google if there are site indexing
| issues, and while they're not always super clear about why, the
| major stuff is pretty easy to fix.
|
| I find it even harder to believe that when he confirmed the
| problem, he then didn't do anything about it for months.
|
| This feels incredibly performative.
|
| Hey Mike: instead of bing chat, try
| https://www.bing.com/webmaster/tools
| oidar wrote:
| techdirt is working fine on Kagi. Ever since its launch, I've
| been predominantly using Kagi, barely resorting to any other
| search engine. I estimate that only about 5% of my searches
| involve Google, and that's mainly when I need to utilize Scholar
| or Books.
|
| Kagi is definately worth the $10 fee. First, it allows me to
| block websites that I find irrelevant or annoying in my areas of
| interest, such as Pinterest or Quora. It also promises an
| environment free of advertisements and tracking. I particularly
| appreciate the excellent customer support, where actual human
| beings respond to your emails, not automated responses.
| Additionally, it offers 'custom lenses', which are essentially
| search templates. For instance, using this feature I can refine
| my search to target only educational sites, filter for PDFs,
| limit the time to the "last 48 hours", and search for specific
| subjects, like machine learning, along with my query.
|
| My only gripe with Kagi is its stringent limit on searches, set
| at around 10,000 per month. I've bumped against this ceiling a
| few times. Despite this limitation, the wealth of features and
| the quality of the search results make Kagi a worthwhile
| investment for me.
| jdjdjdhhd wrote:
| DDG is useless, unless you really like Bing and care a bit about
| privacy... I'd rather use Brave for search and Yandex for image
| search.
| thriftwy wrote:
| Just checked - Yandex has recent pages from techdirt.com in
| search results.
| ty_2k wrote:
| It would be really nice if DDG had their own webmaster console
| (like Google or Bing) to help understand why pages might be
| indexed or dropped. I love DDG as my primary search engine, but
| it's a black box from the SEO side of the desk, unless I'm
| missing something obvious.
| FabHK wrote:
| > [DDG is] a black box from the SEO side of the desk
|
| That's a feature, not a bug, from my point of view.
| ipaddr wrote:
| It doesn't index you, bing does.
| WallyFunk wrote:
| Kagi[0] FTW
|
| [0] https://kagi.com/
| [deleted]
| ehPReth wrote:
| Is Kagi "streets ahead" of Google and the like? I find myself
| getting more and more frustrated at Google lately...
| yegg wrote:
| (DuckDuckGo CEO/Founder) Just seeing this and we're looking into
| this now. This is not intentional.
|
| Update: Still investigating, but have made some progress --
| Determined that on desktop there was a link to Techdirt up
| continuously via our About module (when you search for
| "Techdirt"). And now the traditional web link is back up as well
| (for desktop and mobile):
| https://duckduckgo.com/?q=techdirt&ia=web.
| dredmorbius wrote:
| I've found DDG to be _extremely_ responsive to issues in the
| past.
|
| I'd reported a problem with the "lite" interface about two
| months ago. It was not only acknowledged, but _fixed_ , in just
| over half an hour:
|
| <https://toot.cat/@dredmorbius/110476717889891339>
| alexsantos201 wrote:
| [dead]
| WheatMillington wrote:
| I thought you barely relied on Bing anymore. At least, that's
| the claim you've repeatedly made here. This indicates that DDG
| is little more than a Bing mirror.
| pb7 wrote:
| Regardless of what their CEO claims here every single time,
| if Bing cut off their access, DDG would be dead overnight.
|
| All the stuff they claim to do themselves is <1% of whatever
| value DDG provides. Lucky for them it's just enough to be
| able to use terms like "largely" instead of "entirely",
| "other sources" instead of "only source", etc.
| photonerd wrote:
| They wouldn't die overnight at all.
|
| They would fast become outdated however, but that's a
| different problem. One solvable with another source.
| kats wrote:
| [flagged]
| yegg wrote:
| I don't believe I have ever made that claim and it isn't true
| either that we are "little more than a Bing mirror". See
| other comment on this thread
| (https://news.ycombinator.com/item?id=36898807) but in short
| it isn't a binary thing and I think the fallacy people keep
| making is a modern search engine != web index.
|
| Over the past fifteen years, search has become "universal"
| with dozens of indexes and modules throughout the page. Also
| it is important to understand people click/engage with things
| on the page roughly half as much each position down, so by
| the time you get to the bottom it is like 100 times less than
| the top. This means that things on top, increasingly non-web
| links, have become more and more important.
|
| In this context, as mentioned in the other comment, the
| largest modules on mobile and desktop we power ourselves,
| that is, local and knowledge graph. AI will be the same. We
| do use Bing as the primary source for traditional web links,
| but not for all and even when we do it sometimes looks
| different in various ways. In something like this, we can re-
| insert this link if that is what is needed.
| lxe wrote:
| Thanks for the update! From your other thread:
|
| > Bing is our largest source of traditional web links.
|
| I think "traditional web links" is the main (and I think
| should be the only) product of a search engine, and it
| seems like you rely on Bing quite significantly to serve
| search results.
| rollcat wrote:
| > I think "traditional web links" is the main (and I
| think should be the only) product of a search engine
|
| This is context-dependent. If you're looking for a link,
| then you're looking for a link; but more often than not,
| you're actually looking for _answers_.
|
| Say you're looking up an exchange rate or the value of a
| stock; converting units; checking the weather; or just
| want to look up an actor's photo. Why would you click and
| wait for a (likely, slow & ad-infested) page to load, if
| the search engine can provide factually correct data
| without any extra steps? How often do you read past the
| first paragraph on Wikipedia?
|
| If you want to dive in / verify any of that information,
| it's still all just a click away. But having to only do
| one thing instead of two is almost the definition of
| technological progress.
| eli wrote:
| A massive number of Google searches end without the user
| needing to click through to any results.
| blowski wrote:
| I use DDG as my main search engineer. Many of my searches
| have hashbangs into other sites. On DDG landing pages, I
| often go straight to the right hand side of the page,
| with the info panel. Sometimes, I click on traditional
| web links, but with less frequency as the sites are so
| often spam.
| brightlancer wrote:
| > I think "traditional web links" is the main (and I
| think should be the only) product of a search engine
|
| I want room for both.
|
| There are times I want the search engine to stop
| "helping" (sic) and just give me a straight search based
| on my terms.
|
| There are other times where I'm looking for something
| specific but I don't remember exactly what it is, so I
| want help correcting/ narrowing down my search.
| yegg wrote:
| From lots of experience, I can tell you if you don't have
| these things (and others, this is an incomplete list just
| off the top of my head) showing up in the right places
| (often above "traditional web links") at the right times
| then you cannot get long-term search engine adoption from
| mainstream users:
|
| - maps
|
| - directions
|
| - local business listings
|
| - about boxes
|
| - basic facts
|
| - conversions
|
| - news
|
| - videos
|
| - images
|
| - shopping
|
| - sports
|
| - stocks
|
| - weather
|
| - flights
|
| - entertainment (e.g. cast)
|
| - etc.
| Darthy wrote:
| Gabriel, could you please clarify once and for all: for
| "traditional web links", how much do you rely on bing?
| The user consensus seems to be 100%. What is the official
| number from you? Thanks!
| photonerd wrote:
| It's unlikely to even be possible to break it down as
| simplistically as a %
| nextaccountic wrote:
| I also want a % for traditional web links (ignoring all
| the non-web links pushed to the top). Is it 80%? 60%?
| 99%? "Primary source" means what?
| ncallaway wrote:
| The previous statement seems to answer that reasonably
| well.
|
| > We do use Bing as the primary source for traditional
| web links, but not for all and even when we do it
| sometimes looks different in various ways
|
| I don't know that it's reasonable to demand that they
| provide you an exact percentage.
| thewataccount wrote:
| > but not for all and even when we do it sometimes looks
| different in various ways
|
| Personally I'd like to know if that means "we have a
| manual list we re-added". I think what myself and
| probably many people are most interested in is "do you
| rely on bing for < 95% of word match searches (no
| knowledge graph)"
|
| > when we do it sometimes looks different in various ways
|
| IMO this is also vague - I assume this means they're
| knowledge graph augmented? I really want to know if they
| have their own indexer (vs a manual list + caching bing
| results) for text matching.
| thewataccount wrote:
| I understand that you need these to be a full "search
| engine".
|
| What we really want to know is if for links alone - what
| extent do you index? Let's ignore the knowledge graph,
| let's say we just want a simple word match.
|
| I'm surprised you didn't have techdirt's homepage indexed
| at all. "techdirt" did not return "techdirt.com". To me
| this means you aren't indexing the home pages for popular
| websites at all - let alone their content - and rely
| effectively exclusively on bing with maybe some
| "caching".
|
| Talking about links alone - exact word matches with the
| domain name (or page content other than your knowledge
| graph), what do you index?
|
| What specifically do you do other than adding sites
| manually to a list and I assume "caching bings results"?
| To me this would still be 99.999999% ~= 100% reliant on
| bing for search results.
|
| This isn't meant to be an attack in anyway, I'm just
| looking to clarify what's going on.
| photonerd wrote:
| That's a nice thought, but hasn't been the case for
| search engines for a decade (or more).
| kikokikokiko wrote:
| It's turtles all the way down, I mean, it's Bing. If your
| basic source of information, the thing that you rely on to
| create any smoke and mirrors on top of it, i.e. the index,
| is basically Bing and just Bing, then you're just a Bing
| mirror. More and more I see myself using Yandex out of all
| things to get decent results. The extremes the web got in
| 2023 are bizarre, the "best" search engine in the world
| today may very well come from a dictatorship controlled
| country.
| photonerd wrote:
| Either you're very misinformed about what a mirror is, or
| you're being extraordinarily disingenuous here.
|
| Bing is their _initial and primary source of data_ for
| web links. Not the only one either, but the primary one.
| They maintain their own index _from it_.
|
| That's not "a mirror" by any stretch of the imagination.
|
| What appears to have happened here is some form of
| replication bug where _in a way yet to be determined_ a
| removal from the original data source was
| _unintentionally_ replicated.
|
| Data synchronization is _hard_ when you're not directly
| mirroring data. This kind of bug crops up
| kikokikokiko wrote:
| If they're not just a mirror, it seems rather strange
| that when a specific website disappears from Bing's
| index, it also disappears from DDG. In my very common
| sense based view of the situation, if DDG really builds
| it's own "index" based on a variety of sources, then why
| did TechDirt completely disappeared from it's search
| results? DDG can argue all day about how much their
| "local and knowledge graph" is built in-house, yada yada
| yada. At the end of the day, a search engine must
| retrieve the most relevant websites for a given query.
| And nothing can be more relevant to "site:xxx.xxx.xxx"
| than the site itself. If it doesn't appear on DDG
| results, then it's not indexed. And then it clearly shows
| that DDG is nothing but a fancy mirror to Bing search.
| photonerd wrote:
| > it seems rather strange [...]
|
| Not really. That's the point. Data replication is hard,
| especially when you have to consistently update from the
| same data source.
|
| Bugs happen.
| WheatMillington wrote:
| You can spin it any way you like, but the fact is evident:
| you rely on Bing to the extent that if Bing delists a site,
| that site will be completely sanitised from every single
| part of a DDG search, including all those dozens of modules
| you keep referring to.
| Waterluvian wrote:
| Thanks for looking into this. Would love to learn how a
| specific query like "site: techdirt.com" got wiped like
| that. Will you share your findings once it's been
| identified and resolved?
| lapcat wrote:
| I doubt there will be any findings, because the removal
| from the index was done by Microsoft rather than
| DuckDuckGo. The postmortem has to come from Microsoft.
| Likely the only thing that Yegg can do is poke some
| internal contact at MS and tell them to fix it.
| Waterluvian wrote:
| That's a finding, though. "We still depend too much on
| Bing and so when you search for a website that Microsoft
| has censored, we can't do any better."
| PeterisP wrote:
| They _can_ do better, however, it takes time and effort
| to do that - they can and do augment Bing results with
| other data sources when needed, but up until recently
| there was no need to add anything extra with respect to
| Techdirt; it certainly makes no sense to index all of the
| internet and try to duplicate all the search results for
| every search just in case Bing has filtered something
| out.
| Waterluvian wrote:
| I feel like that's exactly why they _should_ duplicate
| all the effort. So that they aren 't thrall to
| Microsoft's choices.
| ribosometronome wrote:
| I feel like they would love to be able to do that.
|
| One potentially relevant line from the article:
|
| >I love that first one. Microsoft, a company with a $2.5
| trillion market cap, "may not have enough resources" to
| crawl and index Techdirt? Cool.
|
| I have to imagine DDG's valuation wouldn't quite hit 2.5
| trillion if it went public. It's not absurd that they
| aren't able to fully duplicate those efforts (or that
| they don't need to in order to have a decent product).
| Waterluvian wrote:
| I mean... I _guess_ there's that.
|
| They fixed the problem so it shows ability to. Maybe the
| issue is "detect what parts we should do ourselves"? How
| do we detect censorship beyond blog posts like this? Can
| we diff various indices on a regular basis?
| [deleted]
| FormerBandmate wrote:
| Why is Bing doing this? It feels like we're burying the lede,
| a $2 trillion company isn't showing sites that are critical
| of it on its supposedly unbiased search engine
| criddell wrote:
| Does Microsoft actually claim Bing is unbiased?
| stanislavb wrote:
| Hi Yegg, I'm sorry to bother you about another issue; however,
| I'm having a similar experience with DuckDuckGo and Bing. I've
| exchanged dozens of emails with people from Bing's web team
| without any success.
|
| Long story short - at some people in the past someone proxy-
| mirrored all content of SaaSHub. You'd open a page like
| "someshady-proxy-mirror.com/duckduckgo-alternatives", and it
| will mirror saashub.com/duckduckgo-alternatives. The same
| happened for all all pages. Soon after that, Bing (and DDG
| respectively) dropped all SaaSHub links from the index and kept
| the proxy-mirrored content! WTF.
|
| After a week-or-two of "fighting" I managed to block the proxy-
| mirrors; however, I never got SaaSHub back in Bing's index.
|
| I'm a single founder and feel helpless with this Bing/DGG
| issue. Any help would be appreciated. Thanks.
| brucethemoose2 wrote:
| This is what I love about HN.
|
| If some tech kerfuffle is happening, there's a good chance
| someone involved will see it. Its like a wider ranged
| subreddit, or a more orderly Twitter.
| WheatMillington wrote:
| Except we're only getting the canned corporate response of
| "we're looking into it" here.
| Springtime wrote:
| Eh, the HN guidelines say to take things in good faith.
| Seeing how it plays out seems better than pre-emptively
| dismissing what could progress things.
|
| Edit: results appear on DDG, for me, apparently.
| HWR_14 wrote:
| I'm not sure how quickly you expect a more detailed
| response. Acknowledging the report and looking into it is
| the first step to a resolution.
| photonerd wrote:
| Would you like him to write fiction instead?
|
| "This shouldn't happen, we are working out why it did" is
| probably the only official response _possible_ right now.
|
| Elsewhere in the comments he's addressing misconceptions &
| outright falsehoods about how DDG works.
| dang wrote:
| I'm as bored by canned responses as the next person so I
| have to stick up for yegg here - these aren't canned,
| they're interesting comments!
|
| https://news.ycombinator.com/item?id=36899187
|
| https://news.ycombinator.com/item?id=36899072
|
| https://news.ycombinator.com/item?id=36898994
|
| https://news.ycombinator.com/item?id=36898807
| rcarmo wrote:
| Given the "X" rebrand and that HN has a prominent "Y" logo,
| this made me laugh out loud.
| andrewshadura wrote:
| Since you're here, I have a bug to report :) I'm in Slovakia.
| When I look up something that has a related Wikipedia page, the
| widget cites the English Wikipedia. However, the link leads to
| the Slovak Wikipedia but to an article that has a name from the
| English one. Usually things are named differently, so if I
| click the link, I get a 404.
| mapierce2 wrote:
| [flagged]
| lnxg33k1 wrote:
| [flagged]
| dang wrote:
| Could you please stop posting unsubstantive comments? You've
| unfortunately been doing it repeatedly. If you wouldn't mind
| reviewing https://news.ycombinator.com/newsguidelines.html
| and taking the intended spirit of the site more to heart,
| we'd be grateful.
|
| (We definitely don't want fake/bait stuff here either so I
| share your feeling on that - this just isn't a good way to
| express it, at least not on HN.)
| bob-09 wrote:
| Thank you for the update. I appreciate your considerate answers
| and time spent here.
| photonerd wrote:
| Comments summary: lots of people who don't understand how large
| search indexes work freaking out or pontificating about their pet
| favorite alternative. Occasionally both.
|
| Meanwhile the DDG founder calmly stating it's being looked into,
| is definitely a bug, and explains that this should not generally
| happen... and being ignored.
| [deleted]
| bufferoverflow wrote:
| Then how do you explain what we see? Tech Dirt disappears from
| both Bing and DDG indexes at the same time. Google, meanwhile,
| has 136 thousand pages indexed on techdirt.com domain
| Choco31415 wrote:
| Like photonerd mentioned, yegg (The Founder of DDG) is
| answering that and similar questions in this thread:
|
| https://news.ycombinator.com/item?id=36898661
| bufferoverflow wrote:
| I don't see any answers on why tens of thousands of
| techdirt.com pages aren't in the DDG index.
|
| It's not some random unknown blog with zero traffic.
| photonerd wrote:
| Check his other comments in the replies.
|
| https://news.ycombinator.com/item?id=36898807
|
| Meanwhile, thanks for illustrating my comment
| extraordinarily effectively, I guess?
| ipaddr wrote:
| Your point is lost.
|
| He claims when sites get dropped from bing they are
| automatically dropped from ddg and require manual efforts
| to get them up.
|
| There is no process either. You need to connect with
| someone on the inside. No form, phone number or any
| reasonable way of contacting them. If you don't share a
| daycare timeslot with him or can get your submission
| upvoted here good luck
| photonerd wrote:
| That's a lot of conjecture vs the reality. Plus you're
| absolutely mischaracterizing what he said.
|
| You're arguing in bad faith, apparently because you came
| to a conclusion first & are scrambling to justify it.
| Choco31415 wrote:
| Actually it's a fairly easy process to contact DDG.
| There's the "Share Feedback" button on every search page.
| It may be a little less visible, but there's also a
| contact email listed on the Press page.
| ipaddr wrote:
| People understand how this works. Removal from bing means
| removal from duckduckgo.
|
| Founder says they get info from many sources. Talks about local
| and factual information, flights, images, videos, etc as
| important categories while calling normal websites part of
| legacy web. Fails to mention they get this content from only
| bing. Tries to explain people are searching for less legacy
| while people believe they mostly use a search engine for legacy
| web.
|
| I can't see a bug explanation being anywhere near the truth
| when every site delisted from bing is automatically delisted in
| duckduckgo and requires from some action to save it.
|
| How do you explain the other sites?
| philipov wrote:
| When I do the search on bing, it doesn't just give no results, it
| also says "Some results have been removed" with a link to their
| content policy page.
| beej71 wrote:
| Some of my stuff was in a Bing black hole for a while, and I did
| everything I could fix it, following their guidelines, contacting
| tech support, etc. It was good content, too--educational, no
| spam, SEO, ads, or tracking. Eventually I just gave up. Fast
| forward a year or so, and the search results spontaneously
| started working.
|
| So it's a black box, and you kind of just hope for the best.
| mutant_glofish wrote:
| DuckDuckGo has a history of inheriting censorship from Bing [1]
| and also doing it on its own. [2]
|
| [1]: https://reclaimthenet.org/microsofts-bing-censors-tank-man
|
| [2]: https://www.washingtonexaminer.com/news/duckduckgo-
| slammed-f...
| tredre3 wrote:
| That can't be right, DDG's CEO has assured me that DDG isn't just
| a dumb proxy to Bing. DDG uses several sources as well as its own
| index so it cannot suffer from what is being claimed [1][2][3].
|
| 1. https://news.ycombinator.com/item?id=32360874
|
| 2. https://news.ycombinator.com/item?id=36149682
|
| 3. https://news.ycombinator.com/item?id=31492631
| tenpies wrote:
| Only sometimes. Recall that DGG censored Russian sites quite
| hastily and seemingly without so much as an e-mail from any
| government:
|
| > At DuckDuckGo, we've been rolling out search updates that
| down-rank sites associated with Russian disinformation.
|
| - Gabriel Weinberg via Twitter, March 10, 2022. Archived:
| https://archive.is/SLGYb
|
| I'll note that there is a April 17th update tweet referenced in
| that thread. That is unrelated and it was about a rumor that
| DGG was purging certain media sites. Nothing to do about
| censoring Russian sites. Archive of that:
| https://archive.ph/I2iUp
| yegg wrote:
| It is simply not true that we have intentionally censored
| anything for political reasons or made ourselves "the
| arbiters of truth" in general, which people have also accused
| us/me of doing.
|
| I realized I previously explained how our news rankings work
| very poorly on Twitter that got grossly misinterpreted, so I
| subsequently put out a clarification in this help page with a
| much clearer (and detailed) explanation of how our news
| rankings actually work: https://duckduckgo.com/duckduckgo-
| help-pages/results/news-ra...
|
| From that page: "When we apply our own ranking signals we do
| so in a strictly non-political manner, meaning we don't
| evaluate or otherwise take into account any potential
| political bias or leanings of websites in our search result
| rankings." That is, we did not/do not have a
| disinformation/"truth" detector, nor did we go looking for
| any Russian narrative (or any other narrative for that
| matter). Instead, we just have essentially a spam detector
| that had detected some spam from Russian state sites. That's
| it.
| cbracketdash wrote:
| For news websites, what about having them randomly
| rearrange among themselves for each search?
|
| For example, if the top three results for a news story come
| from CNN, MSNBC, Fox News, the next search would display
| MSNBC, Fox News, CNN.
|
| Anyways, 4+ year user of DDG and still love it!
| WheatMillington wrote:
| And yet here we are hmmmm
| yegg wrote:
| None of that is contradictory. To paraphrase:
|
| - We have on the order of a million lines of search code at
| this point and have a lot of talented people working them. That
| code does a myriad of things across many indexes.
|
| - As an example, mobile searches are the largest category of
| searches, and local searches are the largest category of
| searches within mobile. We don't get any local search module
| content from Bing.
|
| - Similarly, on desktop, knowledge graph / Wikipedia-type
| answers come up the most and we don't get any of that module
| content from Bing either.
|
| - Bing is our largest source of traditional web links, which
| have become less and less relevant/engaged with over time as
| more and more modules are in search results and put on top of
| traditional links (and people interact with things on top of
| the search results page about two orders of magnitude vs.
| things on the bottom).
|
| - When Bing has dropped things out of the traditional web
| index, we have put them back, and we've been working with them
| so this happens less and less. In fact, there hasn't been
| hardly any reports of this in the past month or so, which is
| why I've asked for other examples in the comments.
| aftergibson wrote:
| It's great to see you on here answering/responding to all the
| unfounded rumours on here.
|
| I'd be interested to know because it gets to the heart of the
| matter and why confusion seems prevalent. If we ignore the
| modules and local search, what differentiates DDG and Bing
| when returning bread and butter traditional web links?
| giantrobot wrote:
| As a long time DDG user I think it's cool you're personally
| engaged here. The disappointing part is it seems like
| TechDirt has had to resort to customer-service-via-social
| (Reddit/HN/Twitter) which sucks.
|
| Once this particular issue is sorted out I'd would also be
| cool to see some sort of post-mortem report. Seeing why this
| sort of thing happens and a standard process for fixing it
| would be beneficial to all involved I think. Complex systems
| are complex and shit happens. But if this happened to a site
| far smaller than TechDirt (they're not even that big AFAIK) I
| don't think there would be any avenue to cure the situation
| for them.
|
| /2C/
| butz wrote:
| And that's why we need a diverse ecosystem of search engines. As
| indexing whole internet is pretty much impossible, maybe several
| startups could take a slice of it? Some overlap would be useful
| too.
| [deleted]
| VHRanger wrote:
| As a long time DDG user, I was looking for a reason to try Kagi
| out, and this nonsense is finally pushing me over the fence.
|
| Techdirt has been an upstanding place for journalism forever, and
| getting censored this way is ridiculous.
| nomel wrote:
| > getting censored this way is ridiculous
|
| This is completely unfounded. Please provide any evidence that
| this is intentional or related to censorship.
| Waterluvian wrote:
| I am now looking into Kagi and I love that it seems like they
| just picked the era where Google wasn't terrible and are
| emulating the look and feel of that.
|
| The only problem is the pricing model. I don't do overage fees
| because they make me anxious. I like knowing there is a
| concrete ceiling on what something will cost me. 300 a month
| also feels far too low and $10/mo, whether fair or not, feels
| far too high for a search engine.
|
| This is why I've been working on a behavioural change: stop
| using search engines and start going right to the websites I
| know and trust.
| [deleted]
| flyinghamster wrote:
| At this point, my searches are tending to start at Wikipedia,
| enough that I might make it my Firefox search default. If it
| doesn't directly provide enough information for me, it can at
| least send me in the right direction.
| gherkinnn wrote:
| I've been using Kagi for a little under a year and it is
| absolutely worth it. DDG gives very inconsistent results and G
| is just as bad and complete dog shit when the adbloker is off.
| init2null wrote:
| Kagi is an underrated superpower. It just works. I tried Google
| again recently, and I was astonished how mediocre the results
| were. If that's the gold standard for no-cost searching, the
| internet is in serious trouble.
| alx__ wrote:
| Yeah, Kagi returns better results for me. Is free, but
| totally worth the one coffee ($5) per month.
| intrasight wrote:
| Doesn't Kagi just use search APIs to get it results? Why
| would they be better if they don't build their own index?
| alx__ wrote:
| Where have you heard this?
|
| They build their own index:
| https://help.kagi.com/kagi/why-kagi/kagi-vs-
| competition.html
| J_Shelby_J wrote:
| They build their own index but don't let you access it
| with keyword based queries. Instead the do 'magic' and
| shape the responses from multiple data sources.
|
| I'd love it if instead of 'magic' we just had a search
| engine that let you be the magician with a better query
| language and ux filters.
| VHRanger wrote:
| They're probably doing something like mapping the
| embedding of the query to the nearest neighbor in the
| search given how the front-end of their search infra
| works
| GrinningFool wrote:
| https://kagi.com/faq#Where-are-your-results-coming-from
| thefurdrake wrote:
| Kagi lets me block domains from ever showing up again on
| my searches ever. That alone is worth 5 bucks a month to
| me.
| dabluecaboose wrote:
| Wow, I gotta be honest it might be worth $5/mo to never
| accidentally get sent to Pinterest ever again
| thefurdrake wrote:
| Or fuckin' geeksforgeeks.
|
| I am totally not affiliated with kagi but since I started
| using it, I haven't gotten pissed at my search engine for
| sucking ass.
|
| OH! Another feature that actually works on Kagi: Date
| filtering! Never again will you set that date filter to
| the last two weeks and receive results that were posted
| in 2005.
| CalebJohn wrote:
| Kagi uses multiple search APIs (including their own) and
| implements their own ranking and mixing of results.
| Basically you get the same results with spam removed,
| with some truly unique results from the in-house engine.
| [deleted]
| [deleted]
| ziftface wrote:
| You won't regret it, it's been a lot better than ddg for me.
| yegg wrote:
| We have not censored anything -- see my comment here
| https://news.ycombinator.com/item?id=36898661 (and others on
| this thread).
| pb7 wrote:
| Technically correct. Bing, who powers your search engine,
| did.
| nomel wrote:
| Can you provide some context about Bing censoring Techdirt?
| These HN comments are the only thing that shows up in a
| search. The above article says nothing of censorship, other
| than screenshots of the Bing AI giving "maybe" speculations
| about it.
| HWR_14 wrote:
| I've noticed DDG dropping a lot of sites over the past month or
| so.
| yegg wrote:
| Do you know what they are? Happy to investigate along with
| TechDirt and that might help.
| HWR_14 wrote:
| I don't recall at the moment. If I remember I'll let you
| know.
| mdaniel wrote:
| a more sustainable approach would be to use the "Share
| Feedback" in the (regrettably positioned) lower right
| corner of the search results page. The story I've heard is
| that actual humans look at that feedback
| yegg wrote:
| Yes, that's right -- we do!
| xnx wrote:
| It's ridiculous to talk about DDG being separate from Google. In
| the same way Brave is Chrome with some minor doodads and
| settings, DDG is minor reskin of Bing.
| danudey wrote:
| This is an inaccurate statement. See here for some specifics:
| https://news.ycombinator.com/item?id=36898807
| vancan1ty wrote:
| I find that brave search is actually pretty good these days. Has
| plenty of results from techdirt at least... [1]
| https://search.brave.com/search?q=site%3Atechdirt.com&source...
| phreack wrote:
| Title says [fixed] but that's only for DDG - which despite all
| the flaming it's getting, somehow did fix it while Bing has not
| (will it?). My unfounded theory is that Techdirt is effectively
| banned on some country and that leaked to the global Bing index.
|
| https://www.bing.com/search?q=site%3Atechdirt.com
| sct202 wrote:
| It doesn't look fixed to me. Only the homepage shows up on DDG
| from searching site:techdirt.com
|
| Google has 100k+ results for the same prompt.
| maxlin wrote:
| After DDG's earlier announcements delivering the image that DDG
| then severed their uneasy dependency from Bing, this makes them
| look really bad. Retorting with mumbo jumbo about secondary stuff
| like flights, business listings etc that are just related to
| website retention rather than their core structure as an apparent
| search engine, just makes it worse.
|
| I still believe DDG has a place, in being a more privacy-focused
| aggregator to Bing and a few other sources, but this did shatter
| the previous image of DDG having a large degree of autonomy for
| me.
| pdimitar wrote:
| Well, the last few months DDG's search quality started dropping
| hard for me, some queries that I clearly remember having good
| results no longer have them today.
|
| It seems there's stuff going on behind the scenes. DDG got taken
| over by some vested interest, perhaps. Or Bing doing stuff and
| DDG never branching out of it and just blindly getting results
| from it.
|
| Either way, it's starting to rival Google in uselessness and I'm
| likely to stop using it fairly soon.
|
| Kagi seems to work pretty well.
| FireInsight wrote:
| I'm experiencing a weird bug where after clicking on a result
| and going back to the results page the order of the results
| change.
|
| I've too noticed result quality in the main search engines I
| use (DDG, Brave, Phind) go way down in recent times, I wonder
| what happened...
| joshuaissac wrote:
| > I'm experiencing a weird bug where after clicking on a
| result and going back to the results page the order of the
| results change.
|
| This also happens to me and I have found reports on Reddit of
| it happening to many others.
|
| Sometimes, I cannot even find the original result that I had
| clicked on, when I return to the search results page.
| aimor wrote:
| "I'm experiencing a weird bug where after clicking on a
| result and going back to the results page the order of the
| results change."
|
| This drives me crazy.
| nomel wrote:
| From DDG, I always open links in new tabs, because of this.
| at-fates-hands wrote:
| >> Kagi seems to work pretty well.
|
| free for 100 searches. $5/month for 300 searches $10/month for
| 1,000 searches
|
| I'm not at the point where I feel DDG and Bing are so bad I
| want to start paying to get better search results. I'd be
| interested to see how many people are there though.
| pdimitar wrote:
| I am not sure I am doing more than ~33 searches a day lately
| so probably the $10 plan will suit me fine. I think I do
| something like anywhere from 5 to 20 a day.
|
| But I too started getting disgusted by "everything is a
| subscription" but I might jump the train if I like Kagi
| enough.
|
| Because apparently the internet companies can't figure out an
| ad model that's not extremely toxic and does not trample on
| every single privacy rule the world has (and the 100x more
| that the world still doesn't have but should not be broken
| anyway because they should be a moral / ethical no-brainer
| but alas, go tell that to the "money above everything"
| types).
|
| But finally, after decades, all the VCs funding internet
| companies that planned to capture the market with network
| effects and _then_ start charging, are showing their true
| colors. I am glad. It makes them more honest and gives the
| users better information to act on.
| t0bia_s wrote:
| Doesn't privacy bother you? Tightening your Kagi search
| query log to credit card sounds privacy invasive to me.
| pdimitar wrote:
| It does. Sadly I don't see a way out.
|
| Us the nerds just practiced endless bikeshedding and the
| corporations took over everything in the meantime. Oh,
| let's not forget the people who kept inventing LISP
| dialects BECAUSE THAT'S EXACTLY WHAT THE WORLD NEEDED!
| yegg wrote:
| Any particular examples you can remember that I can look into
| personally (and bring back to the team to look at as well)?
| arp242 wrote:
| Is there anywhere people can report clearly bad results? I
| don't have any on hand right now either, but I do know I
| somewhat regularly encounter DDG results that are just
| "wtf?!"-level, whereas Google does return good results for
| the same. I'd be more than happy to report these things when
| I encounter them if that's helpful.
| Choco31415 wrote:
| There's a "Share Feedback" button at the bottom of the
| search page. Yegg said it's read by humans too.
| reaperducer wrote:
| I'm not the OP, but here's a quick list of search items that
| did not turn up useful results recently off the top of my
| head: number of Americans without a credit
| card panic nova git automation microvision (the
| video game console) time in chicago (seems to be fixed
| now) mozilla open directory (seems to be fixed now)
|
| Now that you've solicited this feedback, I'll make a point of
| saving futile searches for the next time you pop up on HN.
| worik wrote:
| I have been using DDG as my main search for years and years.
|
| But I have found the LLM powered Bing, using Skype (!!) as
| the user interface really compelling
|
| I have played around myself with getting LLMs to summerise
| and evaluate text for me, and it seemed that an automated way
| would be very cool.
|
| Then Microsoft implemented it.
|
| Wow. It is very very good
|
| So DDG is still good. Very good. And privacy, a very strong
| selling point
|
| But wading through click bait websites looking for a website
| that is not just straining for eyeballs: Bing's LLM based
| machine is a dogsend!
|
| I would pay for a service like that from DDG. I already pay
| real money for an OpenAI API key.
|
| Sell me yours. I would much rather pay my money to you
| pdimitar wrote:
| I apologize for not keeping them. Sorry. :(
|
| The thing with search engines is: you try one, scan results
| for 30 seconds, don't find a useful result, curse under your
| breath and hastily try another search engine. It's a normal
| programmer flow, which sadly almost completely eliminates the
| possibility to provide actionable feedback to the search
| engine's maintainers.
|
| I gather you're from the DDG team? If so, I'd say that the
| writing is on the wall that Bing is no longer a good backend
| for DDG. You guys should start branching out because from
| where I'm standing, many programmers are 10-15 annoyances
| away from switching.
|
| Again, my apologies for not providing examples.
| Baeocystin wrote:
| The person you're replying to is the CEO and founder of
| DDG. Just FYI.
| pdimitar wrote:
| Good to know, thanks. Had no clue.
| riffic wrote:
| masnick must have pissed someone off recently lol
| MrDresden wrote:
| This is unrelated to the issue at hand, but in case someone from
| Techdirt comes here then please, allow zoom on your site.
|
| There is no defensible reason to disable it on a content heavy
| site like yours.
| NayamAmarshe wrote:
| Brave Search is doing pretty well.
|
| https://search.brave.com/search?q=techdirt
| slig wrote:
| Brave Search feels a lot like Google Search from back in the
| day.
| aimor wrote:
| It would be neat to see a black list of all the things Bing and
| Google block from search results. I don't know how to get such a
| list other than brute force trial and error.
| registeredcorn wrote:
| Better still would be a black list search engine to show _only_
| results that were banned by such services.
| kccqzy wrote:
| Then I guarantee your search engine will mostly contain spam,
| low-effort reposts, content farms, egregiously bad SEOs, and
| other useless stuff.
|
| There's a reason why mainstream search engines like
| DuckDuckGo perform this kind of blacklisting. Because 99.99%
| of the time this blacklisting only renders invisible what is
| worthless anyways. But that 0.01% of the time when it
| incorrectly blacklists a site HN is outraged.
| kermire wrote:
| Overall search quality has been declining on all search engines.
| Maybe there's too much spam. Saw an entertaining video about it
| yesterday that echoes how I feel when I google stuff:
| https://www.youtube.com/watch?v=jrFv1O4dbqY. It's so hard to find
| content written by humans these days. Seems like only the top
| sites are being indexed.
| jxramos wrote:
| I'm in the same boat, I feel like my search-fu is being
| thwarted with the ever growing list of products that coopt
| existing words rather than coining new terms. We're in this
| ever expanding word overloading mode and I think the commercial
| and marketing spaces are now dominating search to drown out the
| useful hits that would previously rise to the top.
| flyinghamster wrote:
| More and more, our search-fu is being _actively_ thwarted.
| Tools to tailor our search have been steadily taken away:
| exclusion /inclusion operators, verbatim search ignored, and
| on and on.
| Beached wrote:
| I want. a search engine that remembers my preferences, and I
| can then click an ignore entire domain option, and never see
| that domain in search results even again. is there such a
| thing?
| Kbelicius wrote:
| Kagi does that but it is a paid service.
| registeredcorn wrote:
| Non-shit tier ukulele equivalent:
| https://youtu.be/pq7NLMwynYg?t=49
| CamperBob2 wrote:
| Google's attempt to introduce zero-click results has been
| nothing short of catastrophic. The sheer volume of bullshit
| they are spreading rivals anything that ChatGPT could ever hope
| to generate.
|
| A couple of weeks ago, I was debating with someone about what
| "LMR" stood for in the context of cable specifications, such as
| LMR-240, LMR-400 and so on. I thought it meant "Land Mobile
| Radio" while the other person disagreed that it stood for
| anything. A Google search on _LMR coax cable acronym_ returned
| a helpful info blurb stating that LMR stood for "Last Minute
| Resistance" as a means of fending off sexual assault.
|
| Needless to say there was no way to tell exactly what site
| Google had copied that definition from, and no useful way to
| provide feedback to them. Sometimes there's a "Feedback" link,
| this time there wasn't. Sometimes the feedback link is present
| but only offers the option of reporting illegal activity. That
| option wasn't present either.
|
| For whatever reason, Google clearly does not give a flying fuck
| at a rolling donut about search quality anymore. With the right
| leadership, Bing could own that entire line of business, in a
| manner reminiscent of IE's original dominance over Netscape.
| I'm not holding my breath, but at this point I'm cheering for
| anyone who can offer Google some competition.
| Miraste wrote:
| I don't trust the zero-click results at all any more. I
| recently had to fill out a form for the DMV that required my
| county code. To find out, I of course googled it, and wrote
| down the knowledge box result. This turned out to be the
| wrong number, and thanks to Google, this happens so
| frequently the clerk knew where I got it, and knew to check
| it and fix it.
|
| People are building workarounds in real life due to how bad
| Google's results have gotten.
| rootusrootus wrote:
| Similar experience for me. One that I do a fair amount is
| searching for specs on various car makes & models. The zero
| click results are typically quite bad. To be fair, a lot of
| it just ends up being bad source data, but their primary
| source for automotive specifications is the same shitty
| site.
| jes wrote:
| I asked ChatGPT 4.0 what LMR meant as applied to coaxial
| cable. It gave me an excellent response, including that LMR
| is a trademark of Times Microwave Systems.
|
| I rarely use search engines anymore. I'll bet the same is
| true for many people.
| lambic wrote:
| I tend to ask ChatGPT first these days, but then follow up
| with a DDG search because I don't fully trust ChatGPT.
| developer93 wrote:
| Phind.com uses the same engine but provides sources.
| Miraste wrote:
| So does Bing, but somehow giving it network access made
| it worse than ChatGPT at everything.
| nonameiguess wrote:
| Regular Google itself tells you that. At least for me, when
| I tried this search right now, the first snippet says
| "These letters indicate the brand. LMR is a trademark of
| Times Microwave, whereas RFC is made by Shireen." It's the
| second snippet that says the last minute resistance thing,
| and it appears to be from someone's personal blog that
| blocks IPs from the United States, so without logging into
| a VPN, I can't open the site to figure out why it thinks
| this.
|
| I would assume the best actual source is the USPTO
| trademark registration database, which has this entry: http
| s://tmsearch.uspto.gov/bin/showfield?f=doc&state=4805:pw...
|
| It appears to indeed mean nothing, but of course, it took
| me a whole two minutes or so to find an actually
| authoritative source. Everyone knows no real user will ever
| do that and just wants a search engine or chatbot to
| dictate reality to them.
| kccqzy wrote:
| Bard also gives an excellent response.
|
| In fact when I used regular Google, the results are good
| enough for me to deduce that on my own.
|
| So I consider GP's search skills inadequate. I mean it's
| not exactly wrong to desire a tool that handholds you and
| feeds you the answer; but if you are willing to do a little
| bit of deduction Google is fine.
| [deleted]
| dannysullivan wrote:
| We care quite a bit about search quality. We've made a bunch
| of changes over the past year to address some concerns that
| have been raised our Perspective feature which is part of
| that recent rolled out on mobile
| https://twitter.com/searchliaison/status/1673382545730457605
| and we have further ranking changes coming soon, as we
| described here https://blog.google/products/search/google-
| search-perspectiv...
|
| I tried the example you cited. The "blurb" is called a
| snippet; the snippet comes from the web page itself. One of
| the pages had that actual text, which is why it appeared. Why
| it had it on a page that's primarily about coaxial cable
| isn't clear, but we'll look into how to improve.
|
| As for sending feedback, each link in the results has a
| little three dot icon next to it that brings up our "About
| This Results" panel, and you can send feedback that way.
|
| Also, to belatedly introduce myself, I'm the public liaison
| for search at Google. It's a position we have within the
| actual search engineering team to help us gather feedback to
| improve search quality. Feel free for you or anyone
| comfortable sharing examples of unhelpful results to flag me
| about it:
|
| https://twitter.com/searchliaison
| https://mastodon.social/@searchliaison
___________________________________________________________________
(page generated 2023-07-27 23:01 UTC)