[HN Gopher] Reddit is experiencing a partial outage
___________________________________________________________________
Reddit is experiencing a partial outage
Author : gundmc
Score : 82 points
Date : 2023-02-21 20:39 UTC (2 hours ago)
(HTM) web link (www.redditstatus.com)
(TXT) w3m dump (www.redditstatus.com)
| wozer wrote:
| For me it looks like the "old" reddit design is especially
| affected by the outage. "new.reddit.com" works fine for me.
| [deleted]
| xur17 wrote:
| Which is amusing because the "new" reddit has consistently been
| loading 10x slower than old over the past few weeks.
| Someone1234 wrote:
| I tried to use new.reddit because it is the only thing working,
| why is it that when I'm reading comments that 50% of my 1440p
| screen is just black background on the left & right? You can
| kind of fix the front page by putting it into "Compact" view,
| but the comment section has nothing akin to that.
|
| This is such a usability disaster relative to old.reddit, and
| it hasn't been fixed for years. Who is this for? I still
| remember when Reddit had one of the best mobile websites,
| before they had an official app and then mysteriously the
| mobile site started to lose previously working functionality.
| jrootabega wrote:
| its for kids who think there phone is there only computer an
| jus trynna see memes doring class
| JohnBooty wrote:
| Another "single page app" design that is slower than the
| thing it replaced.
| technion wrote:
| I'm fairly sure this is some "design paradigm" or something.
| I've complained about it for years but I've never heard it
| described by anyone as an "issue" that's going to be fixed,
| as much as "a design some old people don't like".
| 7speter wrote:
| I think the paradigm is called "use a ton of javascript to
| do something really simple"
| rom-antics wrote:
| At a strategic level, surely the website is bad in order to
| drive people to the mobile apps. Simple enough.
|
| But I wonder how this result was actually accomplished - are
| there Reddit developers deliberately adding JS bloat so
| scrolling is choppy? Are there KPIs that pages must not load
| in under N seconds or videos must not play successfully more
| than N% of the time? It's bad enough that it can't be an
| accident. I'd love to interview one of the devs of the new
| UI.
| rightbyte wrote:
| Ye ... I mean so called dark patterns are so common
| nowadays that it would not suprise me at all if they made
| the website shitty on purpuse.
|
| It is like login in to Gmail on a new computer. It has some
| 2fa that begs for my phone number and the other options
| just don't work half the time.
| sondr3 wrote:
| Can't wait for this to be one of the reasons the hard
| deprecated old reddit :(
| ericmay wrote:
| It will suck when they do that, but at least I'll get
| whatever time I spend on Reddit back to funnel somewhere
| else.
| nickstinemates wrote:
| As long as the API exists, so can your reddit consumption.
| unshavedyak wrote:
| Is there a way to use this on desktop? I'm not sure I
| care enough about Reddit to worry too much.
|
| Tbh I look forward to the day I can cut that time sink
| out. I just can't peel myself away entirely.
| nickstinemates wrote:
| If you want to read - sure. Here's an example.
|
| https://api.reddit.com/r/programming
| thepasswordis wrote:
| I'm giving up Reddit for lent, and would be somewhat
| pleasantly surprised if they did this while I'm gone. It
| would make just never, ever using it again much easier.
| duckmysick wrote:
| Maybe the cause and effect is reversed. We're seeing errors
| _because_ the old reddit is deprecated.
| brailsafe wrote:
| I didn't find this to be the case, took a few tries on both,
| but at least the old one provided an error message
| agumonkey wrote:
| Tested, old yields 503, new shows the normal content. I need a
| tin foil hat.
| mrguyorama wrote:
| There's very understandable reasons this could be the case,
| like the "old.reddit" code base just being untouched as an
| important API changed.
| [deleted]
| JohnBooty wrote:
| One way they appear to be managing load is randomly banning
| users.
|
| I've never had a report against me, I never post anything
| remotely spicy, I never even argue or flame. I moderated multiple
| subs.
|
| And yet my 9-year, paid Reddit Gold account was randomly nuked.
| No explanation. Appeals denied. Support emails never answered.
|
| I guess if your servers can't keep up, one way to cope is to just
| ban the folks who contribute the most to your site?
| nso wrote:
| You really believe they ban people as a way of mitigating high
| server loads? I'm sorry but like, come on...
| JohnBooty wrote:
| I do not literally believe that, no.
|
| However I do often wonder what in the world is happening at
| Reddit HQ. Mitigating server load by banning users isn't too
| much sillier than some of their actual decisions.
| henriquez wrote:
| Now how will I look at porn while I'm supposed to be working?
| corbulo wrote:
| Only partial?
| MuffinFlavored wrote:
| "just roll it back" /s
|
| Anybody want to guess root cause?
|
| Do we have a "root cause" bingo card?
|
| DNS
|
| Database
|
| What else are super likely?
| jandrese wrote:
| Investor left their bottle of Tequilla sitting on the delete
| key?
| MuffinFlavored wrote:
| that sounds Twitter-Musk-esque, amirite?
| martin8412 wrote:
| It's from the series Silicon Valley
| [deleted]
| monitron wrote:
| Pre-IPO jitters?
| SgtBastard wrote:
| Seems to be limited to loading comments in-app. Posts are
| loading fine and Reddit.com is showing comment threads.
|
| Busted API deployment?
| bguebert wrote:
| You should add "BGP config problems" and "cryptolocker"
| sen wrote:
| It's always DNS.
| Trung0246 wrote:
| Just for fun I've bet on hardware failure.
| technothrasher wrote:
| Cat walking on keyboard?
| cududa wrote:
| What an odd comment. As a software engineer, my professional
| guess is the computers aren't working the way they should.
| MuffinFlavored wrote:
| I used to work for a very high level director who was
| promoted many, many times (probably VP level+, $300k/yr
| easily total comp, probably 80 indirect reports overall in
| the org, probably 10-20 years experience) whose entire
| incident playbook handling philosophy was "how quickly can we
| roll it back/why hasn't it been rolled back yet/have you
| tried rolling it back yet"
| lordnacho wrote:
| "It works on my career"
| cududa wrote:
| "And that's how docker was born"
| NobleLie wrote:
| "Let me run the reverse reverse migration script again"
| Spivak wrote:
| Woe be upon thine fools that change code and database
| schema simultaneously.
| ketralnis wrote:
| It's weird for it to be their _entire_ playbook, but most
| outages that I've exacerbated were because I panicedly
| tried to fix things instead of just rolling it back and
| then taking stock.
|
| I often have to work hard to convince people of all
| experience levels that it's the best way forward.
|
| - "It's just a little bug I can just fix it [and definitely
| won't make it worse with code that I haven't tested as
| rigorously right?]"
|
| - "My KPI/bonus/project plan relies on this going out
| today"
|
| - "My code is fine it's the infrastructure [that I didn't
| warn] that can't handle it. They need to fix their side
| now."
|
| I don't know about your VP but "how fast can we get back to
| before it was broken?" is reasonably the first thing you
| should be asking
| bombcar wrote:
| Incident response should always be: (1) get people
| enacting the final disaster recovery plan and rollback
| whilst we (2) see if we can recover from where we stand.
|
| Doing #1 puts some serious boundaries on how bad it can
| get
| spydum wrote:
| i find its usually the same persons or teams responsible
| for both. hard to do them in parallel
| cortesoft wrote:
| This probably really depends on the type of business you
| have. I work for a CDN, our outages are usually caused by
| one of our network peers/providers borking things. There
| is nothing to rollback.
| ketralnis wrote:
| For sure, and you're not going to be able to roll back a
| failed power supply. I'm just saying it's a totally
| reasonable first and maybe even second question
| packetslave wrote:
| If you're a director with 10-20 years experience and you're
| making $300k/year, you're REALLY doing it wrong.
| JKCalhoun wrote:
| A Tesla fire in Newark, CA took out an internet backbone?
| HideousKojima wrote:
| Or a North American Fiber-Seeking Backhoe.
| matt_s wrote:
| You missed the other two common ones: permissions change and a
| disk filled up somewhere.
|
| Before finding out the dead simple failure mode and fix,
| engineers need to spend countless hours diving into the most
| technically complex scenarios that might be happening but are
| irrelevant. Then they can reset permissions or add disk space
| or add back a DNS entry.
| bombcar wrote:
| Reminds me of the old sysadmin who always made a file 10% the
| size of the disk named .root-emergency or similar. Disk
| filled up? Delete the file, get some breathing time, fix the
| problem, recreate the file.
| buggeryorkshire wrote:
| Isn't that what ext filesystems on Linux do? IIRC the
| reserved portion is 5% which can be dropped if you need
| some headroom.
| yamtaddle wrote:
| Yeah, the root-reserved blocks are tunable.
|
| Won't save you if someone's running-as-root reporting job
| goes rogue and fills up the disk, though, while the file
| might... I mean, obviously one ought not have done that
| in the first place, but the real world is a _whole
| thing_.
| bombcar wrote:
| Yep. Often a system crash is caused by logging, which
| often logs ... as root.
| dcdc123 wrote:
| Let's be honest here...Reddit is _always_ experiencing a partial
| outage...
| marcosdumay wrote:
| Well, the outage was in a larger part than usual.
| assimpleaspossi wrote:
| What is reddit and why would anyone go there?
| whatshisface wrote:
| Reddit is a website where anyone can make their own hackernews.
| randomdata wrote:
| With the added bonus of a CEO who will helpfully edit your
| comments if your attempt wasn't funny enough.
| filoleg wrote:
| Let me be clear, I think it was an absolutely moronic move
| by the reddit CEO.
|
| However, you make it sound like it happens all the time. In
| reality, it happened literally just once (multiple
| comments, but all within one hour), the edit itself was
| made obvious on purpose, and it has never happened again
| (it's been almost 7 years since then).
|
| For additional context to what happened: the CEO replaced
| his own username (u/spez) in a few comments on r/the_donald
| (that were just variations of "fuck u/spez") with usernames
| of r/the_donald mods.
|
| Again, I have a very low opinion of Reddit leadership team
| and their direction in general, but "CEO editing user
| comments" is not a concern I have about the website at all.
| Mostly because it was an isolated incident that happened
| over half a decade ago.
| irrational wrote:
| It has forums for niche hobbies like tabletop board gaming,
| wood working, baking, etc. where people can get together and
| discuss why Patchwork is the greatest game ever, why that piece
| of wood is oak, showcase the very first cake they have every
| tried to bake and why it looks good enough to be in a glass
| case in the Louvre, etc.
| navigate8310 wrote:
| It's the hive mind.
| TheMaskedCoder wrote:
| I think it's like teddit, but worse
| yamtaddle wrote:
| It's the thing you put in your Duck Duck Go search (but _do
| not_ do "!reddit" or it won't work) to find content from the
| closest thing the Internet still has to a reliable and genuine
| collection of opinions and reviews by actual disinterested
| humans.
| nothrowaways wrote:
| It has a rich niche community for everything, you will find
| brutally honest experts in everything.
| unsupp0rted wrote:
| I noticed after last week's "don't be cute" discussion the error
| page stopped saying "you broke Reddit" and switched to a generic
| 503-sounding message
| ketralnis wrote:
| We didn't change anything AFAIK, different parts of the stack
| have different error messages
___________________________________________________________________
(page generated 2023-02-21 23:02 UTC)