[HN Gopher] Ask HN: Is HN having problem keeping up today?
___________________________________________________________________
Ask HN: Is HN having problem keeping up today?
Hacker News (this site) seems to be going down today just like
facebook. Is there a massive DDOS attack occurring or just a co-
incidence?
Author : suyash
Score : 288 points
Date : 2021-10-04 18:40 UTC (4 hours ago)
| raimille1 wrote:
| Noticing the same here
| detaro wrote:
| https://news.ycombinator.com/item?id=28749476
| varjag wrote:
| No, just the sixty thousand Facebook engineers flock here
| simultaneously in search for technical clues.
| prox wrote:
| Did you try turning it off and on again?
| prirai wrote:
| XD! Best comment.
| indianpianist wrote:
| Yeah, it's loading slower than usual today. One time I even got
| the "We're having some trouble serving your request. Sorry!"
| page.
| H8crilA wrote:
| This post was sponsored by the liar's paradox and written by
| Bertrand Russel himself.
|
| (meaning if HN is down you can't post about that, and if it's not
| down you won't post it; proving that you can never have a post
| "HN is down" on HN)
| throwawaysea wrote:
| It also looks like the buttons to add comments changed in
| appearance. Maybe there is some kind of site upgrade going on?
| lkbm wrote:
| Looks the same to me. I'm guessing the css just didn't load for
| you.
| brundolf wrote:
| The recent iOS update changed the default appearance of buttons
| in Safari, and not many sites use the raw button appearance,
| but HN does. So that could be what you're seeing
| satchlj wrote:
| very likely - I'm noticing that too
| [deleted]
| agustif wrote:
| Yes. Started loading slower pages and then throwed some can't
| respond to your request right now!
|
| Seems fb down is making people go other places meanwhile maybe_
| lifeisgood99 wrote:
| Getting the "We're having some trouble serving your request.
| Sorry!" page a lot.
| kurthr wrote:
| Can't tell, but logging out to get the cached pages did seem to
| help... so maybe it's a lot of people all logged in hitting the
| same database each click?
|
| Normally, very fast for me.
| satchlj wrote:
| yup me too lol, it's serving the the 502 bad gateway error to
| be precise
| nicoburns wrote:
| I rather suspect that it's because of the Facebook outage, but
| indirectly. HN is always slow when major tech services (Facebook,
| Google, AWS, Github, Slack, Fastly, Cloudflare, etc) services are
| down. The HN community seems to congregate here for updates
| causing higher than usual traffic.
| suyash wrote:
| or it could be a malicious attack? I doubt most facebook users
| even know what HN is.
| shadowgovt wrote:
| Most Facebook users don't know what HN is, but most HN users
| have Facebook accounts and will migrate here for the
| dopamine-inducing refresh clicks (don't let them try to claim
| otherwise ;) ).
|
| If _most_ Facebook users knew what HN was and congregated
| here when FB was down, the site would be utterly crushed.
| djmips wrote:
| It doesn't have to be 'most' facebook users.
| [deleted]
| flutas wrote:
| > The HN community seems to congregate here for updates
| causing higher than usual traffic.
|
| IMO it's probably a lot of us tech workers watching in awe of
| what's going on when a big tech company craps itself.
| [deleted]
| gkoberger wrote:
| No, but most HN users know what Facebook (and WhatsApp and
| Instagram) are.
|
| That being said, I've seen theories that certain non-FB sites
| are slow just because DNS lookups are so backed up.
| blendergeek wrote:
| > That being said, I've seen theories that certain non-FB
| sites are slow just because DNS lookups are so backed up.
|
| I have been getting a Hacker News error message more often
| then usual today. So I don't think it is just DNS.
| nottorp wrote:
| DNS lookups are certainly slower for random sites. I just
| opened an about zero traffic site that i hadn't gone to in
| a while (so dns not cached). It took a while (10 seconds?)
| to load the first time. Insta load the second time.
|
| This seems to happen for .com but not other TLDs so maybe
| .com is getting hammered?
| rplnt wrote:
| But all HN users know what Facebook is. And outage of this
| scale is rare and interesting.
| tuke wrote:
| What is Facebook?
| ed_elliott_asc wrote:
| The beginning of the end of humanity
| peakaboo wrote:
| Its a daycare center for young adults. They grow up
| eventually.
| ChrisMarshallNY wrote:
| _> Its a daycare center for young adults._
|
| Actually, my experience with FB (I have an account, but
| don't spend much time, there), is that it is packed with
| grayhairs, like me.
|
| I think the younger folks have other venues they
| frequent.
| 130e13a wrote:
| This is definitely the correct take. I fall into the
| 18-24 demographic and I don't think I've used Facebook at
| all in the past 3-4 years. Neither does anyone else I
| know who is my age.
|
| This is essentially a reverse network effect: because so
| few young people are on the platform, few other young
| people see a reason to use it.
| tapas73 wrote:
| Also when parents and grandparents join, kids start to
| leave.
| asddubs wrote:
| plus HN has been a bit spotty as of late anyway
| unethical_ban wrote:
| Either there is a _massive_ number of HN users F5ing the
| homepage for the last few hours, HN is woefully unprepared for
| surges in traffic, or there is some kind of outage with their
| infrastructure.
|
| hnstatus on twitter and hund.io show no outage news.
| oneeyedpigeon wrote:
| > a massive number of HN users
|
| It'll be this, I'd bet money on it. Lots like me who don't
| visit every day are coming to HN for more info. This site
| probably only ever sees a tiny percentage of its registered
| users accessing at once; right now, it'll be a much larger
| percent.
| hinfaits wrote:
| I am one of the many that only goes to HN when a major
| service appears down (usually AWS)
| nicoburns wrote:
| IIRC HN runs on a single server. So they don't really put
| much effort into scaling to higher traffic levels. They don't
| really need to. Even now the site is available, if a little
| slow.
| prirai wrote:
| Yep and that's great. Whole of Fediverse, Matrix, IRC,
| Telegram works just fine. Lesson: Never depend on a
| centralized server.
| joecool1029 wrote:
| Telegram is buckling under this too. Only major comms for
| me not slowing down at the moment are Fastmail (email)
| and Matrix (which we have the homeserver on our own ASN)
| BoxOfRain wrote:
| Discord is working for me, at least the text is.
| ufmace wrote:
| Seems like the opposite lesson to me - Telegram is kind
| of iffy, not sure about the others. The only thing that's
| been really solid today is Google. So I guess depend on
| multiple enormous centralized services?
| nafizh wrote:
| That's what I thought but now Twitter is having problem loading
| replies to a tweet. I think it's something more general rather
| than FB centric.
| nicoburns wrote:
| Apparently DNS services are getting hammered with much higher
| traffic than usual for Facebook domains (as FB DNS is down so
| millions of devices are continually retrying the lookup).
|
| For twitter specifically, I also wouldn't be surprised if a
| non-trivial number of people have increased their use of
| twitter while facebook is down.
| raelmiu wrote:
| That's really interesting. Where did you find that info?
| jonah wrote:
| "hello literally everyone" --@twitter
|
| https://twitter.com/Twitter/status/1445078208190291973
| edoceo wrote:
| WhatsApp in the replies like "hi". I hate the internet.
| simonklitj wrote:
| I at least like what Instagram commented!
| https://twitter.com/instagram/status/1445084977096314880
| jfrunyon wrote:
| The DNS hammering bit was posted by Cloudflare CTO on
| Twitter.
| https://twitter.com/jgrahamc/status/1445066136547217413
| rrrrrrrrrrrryan wrote:
| Here's one source: https://twitter.com/BlazejKrajnak/stat
| us/1445063232486531099
| croshan wrote:
| > FB/IG is down, browse Twitter
|
| > Twitter starts to fail under load
|
| I think it's only natural, given they're the biggest social
| networks. I'd guess they have significant overlap.
| jzellis wrote:
| That's what happens when you run your entire site off a VM
| running as a WhatsApp chat bot.
| aspenmayer wrote:
| I know you're joking, but do you have some links to such
| projects for future reference?
| iainctduncan wrote:
| It's facebook outage rubber necking. Everyone who has been
| involved with keeping services online is like "oh heads are going
| to fucking roll today, let's go see what folks are saying
| happened". haha
| 0des wrote:
| I can't load user profiles at the moment, and some pageloads are
| just spinning wheels. Not sure if that is related to network
| issues or if just a lot of us are rushing to HN to check/comment
|
| EDIT: We're having some trouble serving your request. Sorry!
| notyourday wrote:
| I am honestly shocked HN is not using a CDN
| dang wrote:
| If you log out you get the equivalent of a CDN, and CDNs can't
| serve logged-in users anyway.
| collegeburner wrote:
| My guess is with facebook properties down people are
| procrastinating here. Also news.yc temds to be a good place to
| find timely updates by smart people on outages like this.
| i_like_apis wrote:
| It' just heavy traffic from people who would otherwise be on
| facebook.
| polote wrote:
| This is normal, it happens everytime a big communication platform
| is down. Slack, Teams, Zoom, Google, FB ...
|
| If you want to read HN, logout, or use the private mode
| lucasverra wrote:
| Indeed from Paris France, most probably MZ is checking it a lot
| robocat wrote:
| Edit: way faster if you click logout.
|
| Running in anonymous mode (clear browser cache) makes it
| responsive, but read-only.
|
| Edit: I suggest using a private tab or two browsers or two
| profiles, one where you are logged out to read comments for
| speed, and if you wish to comment or vote then paste a comment
| url over to the other. Side effect: you improve responsiveness
| for everyone!
| celsoazevedo wrote:
| On Firefox you can use containers. Two tabs, one logged in, one
| logged out.
| manojlds wrote:
| Stack Overflow does this well by automatically making it read
| only mode when they are having issues.
| htrp wrote:
| to be fair... the stackoverflow tech team requires
| stackoverflow to restart the company
|
| (along with the rest of the world)
| prirai wrote:
| Hehe, true. Good that it isn't down. FB got a place to
| resort to.
| templeosenjoyer wrote:
| Do you have any idea why it's so much more responsive?
| jraph wrote:
| Anonymous pages are cached.
| johannes1234321 wrote:
| When logged in the site shows you for each entry
| (post/comment) whether you voted on it or not, thus it has to
| request a store with per user unique data. (Also custom
| color, your points, ...) When logged out all get the exact
| same data.
| progval wrote:
| Probably because it can be cached instead of serving
| customized pages
| huhtenberg wrote:
| ... but only if you are logged in :-?
| nrmitchi wrote:
| If you're not logged in it's likely static served from cache.
| Kenji wrote:
| Yes, that's normal. If you're not logged in, HN serves you a
| cached version that is a couple dozen seconds old. That's super
| fast. If you're logged in, HN has to generate the page based on
| the previous actions and properties of your account (e.g.
| upvotes, favourites, etc.) with every HTTP request.
| kreetx wrote:
| This appears to be the case.
|
| Unlike one of the other comments, hn is slow relatively rarely
| (perhaps a few times a year).
| pshc wrote:
| It was my perception 10 years ago that HN was powered by a single
| threaded process with the entire post history loaded into memory
| (not sure how accurate this perception was). I am curious how
| much it has changed.
|
| edit: found this https://news.ycombinator.com/item?id=16076041
| winrid wrote:
| My understandind is pretty much everything for HN requires IO
| to disk. The Era of NVME SSDs...
| dang wrote:
| We also have RAM.
|
| Edit: but we can't GC it fast enough. On a day like today I
| wonder if always going to disk mightn't be better.
| csdreamer7 wrote:
| Hi dang, thank you for what you do.
|
| Do you (or anyone else) have an article on HN's
| infrastructure? Would like to read more about this.
|
| Tried to Google it but was filled with links to other
| people's infra on HN.
| hedvig wrote:
| And the glyph rendering is offloaded to the gpu, which as any
| good development team should know, is much faster.
| suyash wrote:
| my perception is a server located in YC's office under PG's
| desk ;)
| immnn wrote:
| It actually feels like a periodical DNS outage. On Friday some of
| my servers weren't responding like they should. Half an hour
| later everything was fine again. However, some VMs running in
| Azure Cloud were still having problems. Today this huge
| FB/WA/Instagram.
|
| I bet we will read more about this anytime soon in the news.
| dang wrote:
| It seems to be a side-effect of Facebook being down, attracting
| so much attention to HN that it might as well be a DDoS attack.
|
| Sorry everyone--there are performance improvements in the works
| which I hope will make a big difference, but it's not something
| we can roll out today.
|
| p.s. You can read HN in read-only mode if you log out. Should be
| way faster.
| invisible wrote:
| You're awesome dang! Thanks for all you do. That things are up
| as much as they are is pretty great.
|
| I, and probably a bunch of others, would be pretty interested
| in a write up of the challenges and how you solved them if you
| could find time after things settle.
| a-dub wrote:
| lol, it's not dns (fb dns is back), it's not coincidence, it's
| not people coming to hn specifically to read about fb being
| down. it's the digital equivalent of the consumer goods
| shortages resulting from changed behavior during the lockdowns
| of 2020. the whole consumer/attention-economy internet is
| creaking under the load and it's kinda hilarious.
| https://downdetector.com/
| dang wrote:
| I think it really is people coming to hn specifically to read
| about fb being down.
|
| What kills me is that we'd be humming just fine if the
| software I've been working on for so-long-I'm-embarrassed-to-
| tell-you had been rolled out by now. At this point the only
| solutions are non-technical. A la the Serenity Prayer, and
| waiting till tomorrow.
|
| Edit: it does seem to be better for community if things break
| occasionally, though. People can relate to sheepish devs
| under excessive load. Nobody relates to a perfectly oiled
| machine.
| nicoburns wrote:
| I have no idea how much extra traffic you had to deal with
| today, but I'm always impressed with how gracefully HN
| seems to degrade under high load. There are increased page
| load times, and frequent "Oops, we couldn't serve your
| request messages", but the pages do load and trying page
| loads and comment posts actually works.
| Wowfunhappy wrote:
| I have to say, I find it surprising that ycombinator hasn't
| hired another developer to work on HN. It's great that
| you're actually well-versed in the codebase, but you
| shouldn't be the lead moderator, the lead customer support
| contact, _and_ lead developer of HN, that 's nuts! (Oh, and
| you _also_ coach startups on their Launch HN threads,
| right?)
|
| Heck, I can't tell--are you _merely_ the "lead" on those
| things, or are you the sole person? You often say "we" when
| talking about HN, but I can't tell if that's a royal we or
| not.
|
| I know that HN is meant to be a low-resource, but
| ycombinator thinks it has any value at all, they ought to
| be able to spare another developer. This is just basic
| organization management. Of _course_ you haven 't finished
| your software yet, you can't do everything!
| steveyklabanik wrote:
| Have you tried rewriting in Rust?
| mxuribe wrote:
| So...maybe HN has dependencies on FB (such as tracking pixel,
| etc.)...and with their outage, timeouts kill things for HN? Not
| blaming HN, just wondering if second order diminished experiences
| start happening because of FB's heavy gravitational pull?
| grensley wrote:
| I've always found it interesting that this fairly influential
| site rarely changes and hasn't kept up with modern internet
| standards.
|
| I get that it's an ambiguously useful project for a venture
| capital firm and all, but you would think that they would throw
| it a bone or two given the level of influence it has.
|
| It also seems like they're trying to make some kind of point
| about dead simple web design and running a web server off of old
| laptops or something. I'm not sure, but there's definitely some
| combination of neglect and hubris going on with this site.
| bobbylarrybobby wrote:
| > kept up with modern internet standards
|
| Like increasingly long page load times, pervasive use of JS for
| even the simplest things, and throwing up a million banners
| before you can read anything?
|
| I, for one, am very happy that HN is, for the most part, just
| plain old HTML that loads instantly and just works 100% of the
| time (when the servers can handle the load, that is).
| nmstoker wrote:
| I'm 90% with you, but I'd love them to do something to stop
| it being so easy to "fat finger" the hide link directly above
| main links when on mobile - just a tiny bit more space would
| work wonders.
| laumars wrote:
| The design is perfect. It's information dense while being
| really clear and easy to use. There is absolutely nothing wrong
| with the design.
| nrb wrote:
| > The design is perfect. It's information dense while being
| really clear and easy to use. There is absolutely nothing
| wrong with the design.
|
| I'd argue that unformatted quotes is a pretty clear design
| opportunity, especially when code blocks are misused enough
| that mobile use is regularly impacted by it.
| daniel-thompson wrote:
| > It also seems like they're trying to make some kind of point
| about dead simple web design and running a web server off of
| old laptops or something.
|
| Yes, and it's a point worth making.
| cloverich wrote:
| It is hard to see at first, but the lack of design and features
| is itself a feature, and perhaps crucial to what keeps HN
| chugging along. UX emerges not only from what you add, but also
| what you do not.
| [deleted]
| bevacqua wrote:
| It's just people flocking to HN for reliable news on the insane
| outage affecting global Facebook services, and refreshing far
| more than they usually would. Data point of one
| iainctduncan wrote:
| make that two! ha
| 1001101 wrote:
| "People look for alternatives and want to know more or discuss
| what's going on. When Facebook became unreachable, we started
| seeing increased DNS queries to Twitter, Signal and other
| messaging and social media platforms." from Cloudflare via
| https://news.ycombinator.com/item?id=28752131
| BatteryMountain wrote:
| For me it lags when scrolling. Very odd.
| jraph wrote:
| Have you tried to load it on a computer that is 30 years old or
| younger?
| Y_Y wrote:
| Yes.
| jraph wrote:
| You might want to clean your mouse wheel.
| zanethomas wrote:
| possibly because of the huge number of commenters laughing about
| facebook being down
| CivBase wrote:
| > General tip: If HN is being laggy and you're determined you
| want to waste some time here, open it in a private window. HN
| works extremely quickly if it doesn't know who you are.
|
| Saw this from a user by the name of bentcorner in the "Facebook-
| owned sites are down" thread[0]. Figured I'd repost it here. It
| works because HN can cache pages more efficiently when it doesn't
| have to include user data.
|
| [0] https://news.ycombinator.com/item?id=28750513
___________________________________________________________________
(page generated 2021-10-04 23:00 UTC)