[HN Gopher] Cellular outage in U.S. hits AT&T, T-Mobile and Veri...
___________________________________________________________________
Cellular outage in U.S. hits AT&T, T-Mobile and Verizon users
Author : rooooob
Score : 372 points
Date : 2024-02-22 11:15 UTC (11 hours ago)
(HTM) web link (www.cnbc.com)
(TXT) w3m dump (www.cnbc.com)
| TheAdamist wrote:
| This reminds me of the recent discussion on status pages.
| https://news.ycombinator.com/item?id=39099980
|
| They need to be accurate. At&t status claims everything is fine.
|
| My wireless service is down. Down detector has tens of thousands
| of reports, so clearly everything is not fine.
| bombcar wrote:
| Status pages are basically useless if they're public facing.
|
| Either they automatically update based on automatic tests (like
| some of the Internet backbone health tests) or they're manually
| updated.
|
| If they're automatic, they're almost always internal and not
| public. If they're manual, they're almost always delayed and
| not updated until after the outage is posted to HN anyway.
| op00to wrote:
| Which is better? How do you know whether an issue is
| individual to a customer or a quick blip that will resolve in
| a few seconds?
| bombcar wrote:
| I prefer fully automated tests publicly revealed because
| the main thing I want to know (as a customer) is _should I
| keep trying to fix my end or give up because GitHub
| exploded again_.
|
| It's most annoying when you have something like recently -
| known maintenance work on my upstream home fiber connection
| that was resulting in service degradation (but not complete
| loss, my fiber line was back to DSL or dialup). The chat
| lady could see that my area was affected, but the issue
| lookup system couldn't.
|
| If the issue lookup had told me there as an issue I'd've
| gone on my merry way.
|
| I even checked a few more times until it was resolved; the
| issue never appeared in the issue lookup system.
| op00to wrote:
| > should I keep trying to fix my end or give up because
| GitHub exploded again
|
| Making this decision easy is a fight I fight for my
| customers every day. :)
| bombcar wrote:
| This was _much much much_ easier when websites used to
| explode with tracebacks and other detailed error
| messages, now you just get a "whoopsie doopsie we did a
| fuckywucky" and you can't really tell what's going on.
| menacingly wrote:
| you can't operate at any scale at all without mechanisms in
| place to know perfectly well whether an issue is impacting
| a single customer or if your world is on fire
| op00to wrote:
| Yes, but those mechanisms take time to determine this.
| bombcar wrote:
| You'd like to think so, but surprisingly large number of
| "large scale" things operate on the "everything is fine"
| until too many people complain about the fire.
| pixl97 wrote:
| Caches make problems fun too.
|
| Quite often you see automated tests that check how well
| your cache/in memory data are working. But when some
| other customer that isn't in the hot path tries to access
| their request times out. I've seen a lot of people making
| automated checking systems fail at things like this.
| zitterbewegung wrote:
| The phrase "the hardest parts of computer science is
| caching and naming things" come to mind.
| r2_pilot wrote:
| I see 2 things here but you're off by one.
| ryathal wrote:
| The other problem with status pages is depending on what
| happened it may not be possible to update the status page
| anyway. You really need a third party to have a useful status
| page.
| TheAdamist wrote:
| Which is pretty much what down detector has evolved into.
| And it looks like they have an enterprise offering to alert
| companies to their own issues.
| op00to wrote:
| Which status page?
| spicybbq wrote:
| Currently there is a banner on the AT&T outage page with this
| message:
|
| >Service Alert: Some of our customers are experiencing wireless
| service interruptions this morning. We are working urgently to
| restore service to them. We will provide updates as they are
| available.
|
| https://www.att.com/outages/
| teeray wrote:
| > They need to be accurate
|
| It would be nice if the FTC mandated this. It is exhausting
| when the status page is taken over by the marketing department
| (the infamous green check with the little "i").
| jimmaswell wrote:
| > the infamous green check with the little "i"
|
| I'm not familiar, what are some examples?
| c-linkage wrote:
| I'll take a chance here and say this was a hack, possibly at the
| equipment-level. One major carrier having an outage? Possible.
| But three? On different networks?
|
| Even if its not a hack I'd love to see the root cause on this
| one! Communications is critical infrastructure, so I'm gonna
| guess the government will demand a full report.
| marcus0x62 wrote:
| There is some shared infrastructure in the PSTN that all
| network operators use. This smells more like an SS7 outage to
| me than a hack, but we'll have to wait to find out.
| zikduruqe wrote:
| Backhoe fade?
|
| Back in my circuit switched days, we lost half of our US long
| distance routes, because a farmer in Wyoming dig up an
| unmarked fiber link. Hence, backhoe fade.
| input_sh wrote:
| When you lose your own network, your phone connects to a
| different one to be able to make emergency calls. One carrier
| knocking down other carriers is entirely plausible.
| beAbU wrote:
| Or a borked update to some piece of hardware that's used by all
| 3. Hanlon's razor and all that.
| throwawaaarrgh wrote:
| _" A Verizon spokesman reached by CNBC said there was no issue
| with the Verizon network and their customers are only impacted
| if they try to reach out to the carrier experiencing the
| problem."_
|
| Sounds like only one had an outage
| tallanvor wrote:
| Sounds like it's really only AT&T. Most likely somebody screwed
| up a configuration change and took out a lot of capacity.
| jeffwask wrote:
| I'm 50% hack and 50% BGP but my gut says cyberattack.
| probably_satan wrote:
| Probably Huawei related
| vmfunction wrote:
| >https://www.webwire.com/ViewPressRel.asp?aId=318230
|
| Seems like they are doing drone testing. Hmm, should we use drone
| network that will interfere with mobile phone network?
| oceliker wrote:
| This is so odd. I have two phones with AT&T currently sitting
| right next to each other. One has service, the other is in SOS
| mode.
| sandinmyjoints wrote:
| Same here. Strange. Would love to know the reason!
| BuckYeah wrote:
| Different bands could be affected differently if it is solar
| radiation related. Same exact model of phone?
| sandinmyjoints wrote:
| Different models, in my case.
| oceliker wrote:
| Same year, different size (13 and 13 Pro Plus)
| bombcar wrote:
| Does it still show bars in SOS mode? Or is SOS just "I dunno
| can't see no cell towers but maybe it'd work?"
|
| I wonder if the MVNOs that piggyback on AT&T are showing down
| also. If not, it's some AT&T service authorization system that
| exploded.
| sandinmyjoints wrote:
| Mine says SOS Only, shows no bars.
| dcan wrote:
| SOS mode means it can see towers of other providers you
| aren't authenticated with, but no signal to authenticated
| cell towers
| rixthefox wrote:
| On the latest iPhones, SOS mode is the emergency fallback to
| satellite service. It's really meant to be used in situations
| where you're well outside of any sort of service area but you
| have a clear, unobstructed view of the sky.
|
| Your iPhone will instruct you on where to point and help you
| track an emergency satellite that is manned by live humans
| who will take your emergency request and relay it to the
| proper people.
|
| More specific info here: https://support.apple.com/en-
| us/104992
| bombcar wrote:
| Won't that only activate if it can't see or communicate
| with _any_ towers at all?
| lxgr wrote:
| I'm pretty sure that "SOS only" can also mean the phone
| seeing networks it can't register with but which it could
| make an emergency call on if required. This predates
| satellite SOS.
| SirMaster wrote:
| SOS means it has cell service and you can call 911.
|
| If there is no cell service then it's SOS with a little
| picture of a satellite next to it.
| inferiorhuman wrote:
| Except SFFD is reporting that (some) AT&T customers are
| unable to call 911.
| organsnyder wrote:
| SOS mode typically means that your phone is connected to
| a carrier other than one you have a contract with.
| Scoundreller wrote:
| I wonder if some devices bungle their failover. The exact
| failure mode state of AT&Ts network might cause some
| devices to hang onto AT&T's RF.
| ok123456 wrote:
| Emergency only means 911 calls through whatever provider is
| available.
| nathanyz wrote:
| Some of our staff are reporting similar where their partner's
| phone has service and their's doesn't. Both on same AT&T family
| plan.
|
| So the radio bands may play into it although I would think with
| latest iPhones, they can use any of the bands from AT&T
| although I could be wrong.
| alephnerd wrote:
| I'm on the latest iPhone and it's SOS for me
| vel0city wrote:
| I'm wondering if it could be some kind of auth timeout. I've
| heard from a few people of one person's phone going out, then
| a bit later the other person's phone finally failing too.
| aclindsa wrote:
| Yep: my partner's iPhone has service while my Pixel doesn't,
| both on same plan.
| PietdeVries wrote:
| Can you check if things improve if you turn off 5G and move
| to 4G/LTE instead?
| nathanyz wrote:
| Good thought, but switching to LTE only didn't work. Same
| result of ending up in SOS only. Cellular over wifi works
| perfectly fine though. Wish we could count on better post
| mortems from the phone companies, but I'm not holding my
| breath for it.
| SkyPuncher wrote:
| My wife and I were riding in the car next to each other. Took
| mine about 5 to 10 minutes longer to jump to SOS mode.
| pixl97 wrote:
| I wonder how SIM registration works? For example if it's like
| a token with an expiry. If some set of registration servers
| on the network couldn't renew then I could see behavior like
| this.
| jeffwask wrote:
| There are some reports that phones with e-sims are less likely
| to be impacted versus phones with hardware sims.
| harambae wrote:
| I have eSIM only (iPhone 15) and was impacted the same as
| physical SIM users on AT&T (Boston area).
|
| I suppose I can't speak to likelihood with a sample size of
| one.
| bdcravens wrote:
| Ditto, mine and my wife's. In my case, the working one is
| slightly newer (15 Pro Max vs 14 Pro)
| tomcam wrote:
| I barely noticed because my AT&T service is so bad on a normal
| day (not being facetious).
| LorenDB wrote:
| My Verizon hotspot slowed to a crawl yesterday. Now it makes
| sense.
| cholmon wrote:
| Another reason why SMS-based 2FA is a bad idea.
| BuckYeah wrote:
| NOAA is reporting R3 activity. Solar flare outage seems likely
| alephnerd wrote:
| Then why isn't Starlink down
| BuckYeah wrote:
| Does starlink use the same bands and same equipment? I doubt
| it
| sumtechguy wrote:
| getting BOFH vibes here...
| LinuxBender wrote:
| There was an earth directed X class, but not a big one. If that
| could affect SS7 I would expect it to also take out a chunk of
| the internet but I am not seeing that. [1] There are _many_
| cellular networks having issues but some could be _and likely
| are_ resellers of others. [2] Probably more likely SS7 related
| as marcus0x62 mentioned. Another _potentially SS7 related_
| possibility? [3]
|
| [1] - https://www.thousandeyes.com/outages/
|
| [2] - https://downdetector.com/
|
| [3] - https://www.webwire.com/ViewPressRel.asp?aId=318230
| Waterluvian wrote:
| Then why is this a U.S. only problem?
|
| Edit: I'm getting 1 bar when I usually get 4 in southern
| Ontario. But I see no broad reports of issues.
| relbeek2 wrote:
| A solar flare targetted at ATT Infra only? Unlikely.
|
| Could be a hack Could be a single point of failure Could be a
| config change that borked the system
|
| Could be other things that make more sense we will have to wait
| for more info.
| IAmGraydon wrote:
| I know the other carriers are saying they aren't affected,
| but one look at DownDetector shows that nearly every carrier
| was affected, and all at the same time.
| miah_ wrote:
| A user who has a working cell phone could contact a user
| who is on a network experiencing issues. Because the call
| fails, that user may decide their cell network is also
| faulty. Downdetector only works on user reports. Its
| basically useless for actual measurement because people are
| bad at troubleshooting.
| joecool1029 wrote:
| That site counts mentions over social media, if you said
| 'T-Mobile/Verizon isn't having an outage' it'd still show
| up as outage activity on it. Plus people report issues
| calling AT&T customers.
| themaninthedark wrote:
| https://www.spaceweatherlive.com/en/news/view/522/20240222-t...
|
| We did have Two Class X flares from sunspot 3590 but they did
| not result in a coronal mass ejection.
|
| There was a CME from another sunspot not visible but it is not
| aimed at the earth.
| EMCymatics wrote:
| NOAA did release a statement saying the two X class flares
| were unlikely to effect cellular networks.
|
| https://www.swpc.noaa.gov/news/two-major-solar-flares-
| effect...
| swagmoney1606 wrote:
| I don't think this is likely
| nkotov wrote:
| SOS Only on AT&T down here in Charlotte
| nathanyz wrote:
| Same in Florida
| partiallypro wrote:
| I haven't had service since around 3am this morning, my internet
| (fixed wireless) was also down around that time but has since
| come back on. I had just assumed it was local tower maintenance
| until I woke up and see it's still out and it's effecting
| millions. But was surprised to see it barely mentioned/getting
| engagement here given the scale.
| flyinghamster wrote:
| Saw my Comcast was down also (SW Chicago suburbs), about 3AM
| CST, but it was back before 4AM. I figured it was a
| neighborhood outage when rebooting the cable modem did nothing,
| and then it came back without any fanfare.
|
| My T-Mobile phone hasn't had any problems, knock on wood.
| n0ot wrote:
| I'm also in the southwest suburbs, and have Comcast. My
| internet went out for about ten minutes at around 9:20 CST. I
| disabled WIFI on my iPhone only to discover I was still
| offline, and had no signal. Not sure whether the Comcast
| outage was related, but it came back up very quickly, whereas
| I still have no cell service (AT&T). My wife does have
| service, even though we both have iPhone 15 Pros.
| IAmGraydon wrote:
| According to downdetector, 3AM was when this all started.
| cranberryturkey wrote:
| Att down I. California
| efitz wrote:
| Reporting status externally is hard. I worked at a cloud provider
| and while we had very detailed metrics about how our systems were
| functioning, it was difficult to distill that information in a
| way that customers would understand if they were being impacted
| or not, and what the impact would be. Just reporting raw numbers
| wouldn't give customers the context to understand what was
| actually going on.
|
| How do you actually report, for example, that .003% of your
| customers are having a really bad day but the rest are just fine?
| 12345hn6789 wrote:
| Aren't these exactly what p99 illustrates? Of course you'd have
| to have your use cases logged well enough to be able to
| aggregate the 99th percentile accurately to each customer flow
| mistrial9 wrote:
| because you define thresholds for event classification.. the
| difference between 10024 customers having failure and 10100
| customers having failure is not the question, right? when many
| hundreds of thousands of customers are failing at once, is that
| really very difficult to determine?
|
| secondly, there are financial, management and information
| security pressures to NOT REPORT reality to the public. This
| happens VERY OFTEN in real business. In fact, that is why legal
| enforcement actions and real consequences are crucial versus
| Big Business.
| efitz wrote:
| In a cloud, massive outages are the rare events, and
| technically easy to report.
|
| Small outages happen all the time, and are difficult to
| report accurately.
|
| I think AWS has pivoted to trying to report status in each
| individual customer's support portal, so that they can give a
| dashboard that reports the state of the cloud from that
| customer's perspective. That way a rack down that only
| affects a few customers is only reported to those customers,
| and the dashboard doesn't have to always be red for everyone
| (or green for everyone, even those affected).
| bradleyjg wrote:
| If it can't be done then don't do it. A dot that's always
| green, no matter what, is worse than nothing.
| alphabettsy wrote:
| Can confirm. AT&T cellular is completely dead for me. Other
| carriers are working though.
| midwestfounder wrote:
| I wonder if this is a cyber incident. Curious if any telecom
| folks know what the most likely explanation for an event like
| this would be, and what telltale signs/symptoms might first
| indicate this was caused by something nefarious.
| bobo_legos wrote:
| Kind of weird that there's been almost no coverage of this
| event this morning. CNBC has barely mentioned it. All 3
| carriers having a major outage seems like it should be major
| news.
| ncallaway wrote:
| When I go to cnn, Washington Post, and NY Times all three
| have big stories about it prominently on their website, so it
| does seem like it's being reported on fairly widely.
| commandlinefan wrote:
| The fact that it's gone on as long as it has already makes me
| suspect the same.
| kylecazar wrote:
| My brother sitting next to me with an AT&T Pixel 7 has no
| service.
|
| Me with a Pixel 7 Pro (dev preview android) on the same plan: not
| affected.
|
| Strange.
| ssimpson wrote:
| i was showing none with iphone 14, wife had like 2 bars with
| her iphone 11. i wonder if its just part of the newer
| infrastructure.
| neom wrote:
| My wife has 12 and nothing, I have 14 and am fine.
| -\\_(tsu)_/-
| ssimpson wrote:
| very odd. i was thinking some kind of firmware whatever
| deployment gone wrong, but maybe it is something more
| nefarious.
| neom wrote:
| tbh it reminds me of this: https://en.wikipedia.org/wiki/
| 2022_Rogers_Communications_out... I was in Canada when
| that happened and it was a mixed bag on what would work,
| there didn't seem to be a rhyme or reason to it.
| stn8188 wrote:
| Similar here - my wife's Motorola is fine but my Pixel 6 has no
| service (same plan).
| midwestfounder wrote:
| I wonder if this is a cyber incident. Curious if any telecom
| folks know what the most likely explanation for an event like
| this would be, and what telltale signs/symptoms might first
| indicate this was caused by something nefarious.
| unforeseen9991 wrote:
| Due to the gross incompetence these companies operate at, it's
| too hard to tell the difference.
| pylua wrote:
| Unfortunately, unlike cyber security, there are no off the
| shelf products that are being sold to help companies with
| general incompetence.
| relbeek2 wrote:
| > However, the US Cybersecurity and Infrastructure Security
| Agency is "working closely with AT&T to understand the cause of
| the outage and its impacts, and stand[s] ready to offer any
| assistance needed," Eric Goldstein, the agency's executive
| assistant director for cybersecurity, said in a statement to
| CNN.[1]
|
| [1] - [https://www.cnn.com/2024/02/22/tech/att-cell-service-
| outage/...
|
| This isn't telling of anything, right? Wouldn't CISA be
| involved with anything that impacts Public Infrastructure at
| this level?
| relbeek2 wrote:
| from the same article above, it seems like it's a critical
| part of this.
|
| > "Everybody's incentives are aligned," the former official
| said. "The FCC is going to want to know what caused it so
| that lessons can be learned. And if they find malfeasance or
| bad actions or, just poor quality of oversight of the
| network, they have the latitude to act."
|
| If AT&T gets to decide if they are at fault, they will, of
| course, never be at fault. So a third-party investigation
| makes a lot of sense.
|
| I would also suspect that the FCC would not be as well versed
| in determining if there was a hack or even who did it, which
| is why I feel like CISA would need to get involved in the
| investigation.
| red-iron-pine wrote:
| by itself, not telling of anything per se.
|
| like, you could commit a dumb BGP config and break lots of
| stuff. have done that in the past, actually...
|
| but any time a national-tier ISP has a national-level outage,
| that warrants a look from multiple orgs. and given the number
| of threat actors like china, NK, iran, and russia, who are,
| and have, made aggressive efforts in this space -- and have
| strong reasons to do so now -- its not crazy for the US
| fed'gov to want to know a little more, and offer to help. but
| again, entirely possible it's unrelated.
| overstay8930 wrote:
| This is normal for high profile outages, even if you are
| small you can still engage with the CISA if you think there's
| foul play.
| blantonl wrote:
| I can confirm that San Antonio, Dallas, and Austin are all down.
| My dual SIM iPhone shows 0 bars for AT&T and Google Fi (T-Mobile)
| is fine. My wife's phone just shows SOS.
| avisser wrote:
| Same in Providence RI.
| softwaredoug wrote:
| I'm just curious how we as a society manage the uptime of such a
| critical service? Do we have laws or enforcement in place to
| regulate / enforce how well basic cellular service (at least the
| emergency tier) must work in the US?
|
| Just imagining the dropped emergency calls today, etc?
|
| (Genuine question)
| ssimpson wrote:
| usually redundancy. but as these companies squeeze more and
| more out of gear, we risk large cascade failures that are hard
| to recover from. could also have been some kind of firmware
| upgrade. anecdotally, the 4G LTE phones seem to not be affected
| whereas the newer 5G ones seem to be. maybe it was a botched
| deployment. i'm sure ATT will give us a really well rounded RCA
| ;)
| alephnerd wrote:
| Don't be surprised if it's a botched Helm deployment ;)
| pierat wrote:
| > i'm sure ATT will give us a really well rounded RCA ;)
|
| Knowing most megacorps, they'll blame some midlevel engineer
| doing a "bad" thing, when it was ordered from above.
| testfrequency wrote:
| I don't have anything positive to add, except noting that
| FirstNet is also down. I'm sure there's a lot of chaos for
| emergency responders right now :/
| lionkor wrote:
| If FirstNet is down that would mean a lot more is down than
| should be. FirstNet can operate on lower range radio, no?
| maxsilver wrote:
| It can (FirstNet's lower 700mhz spectrum), but AT&T uses
| the same backbone and infrastructure and towers for large
| chunks of their and FirstNet's network.
|
| (charitably) - FirstNet is mostly now just a type of
| billing plan and priority level, on AT&T's network.
| relbeek2 wrote:
| > FirstNet is also down
|
| CNN reported that AT&T confirmed it has not been impacted
| throughout this event.
|
| https://www.cnn.com/business/live-news/att-
| outage-02-22-24/i...
| alephnerd wrote:
| Landlines still work, and it's not too expensive to keep it as
| a redundancy.
| dylan604 wrote:
| how does a landline become relevant when one is not at home?
| right up to the start of work from home, I was spending very
| little time at home. a landline made no sense at that point.
| spking wrote:
| They are getting more expensive and less reliable:
|
| https://www.ooma.com/blog/home-phone/landline-home-phone-
| is-...
| softwaredoug wrote:
| Last time I had a real landline, it was flakier than cell
| service. The companies don't really maintain them
| teeray wrote:
| > Just imagining the dropped emergency calls today
|
| Carriers do get fined for E911 downtime IIRC. It is taken very
| seriously, as a result.
| dylan604 wrote:
| Even a landline could have emergency calls disrupted if a line
| was cut somewhere. unless every device was behaves as a router
| with more than one signal in/out, there will always be a
| potential for loss of signal and interruption of service.
| SkyPuncher wrote:
| 911 is regulated. I'm unsure on anything else (I simply don't
| know).
|
| People/companies that actually care already have a second line
| and are taking very good note of this.
| bombcar wrote:
| 911 support is required of companies, it's monitored and fixed
| when not working.
|
| But in this case, it's redundancy from the various providers -
| _all_ providers must let you make a 911 call, no matter your
| phone or contract or provider.
|
| So if AT&T's towers go down, your phone can still make 911
| calls via Verizon or whomever else has a tower in your area,
| anything that can _hear_ your phone will respond to a 911 call
| request.
| uncertainrhymes wrote:
| You should be able to connect to a different network (e.g.
| Verizon) and make a 911 call even without a plan. You won't be
| authenticated to the network for data, but 911 is _supposed_ to
| still work.
| grungydan wrote:
| > laws or enforcement
|
| No no no, see, we have those. But we only use them to make sure
| that our for-profit prison complex stays massively profitable
| and the people in power retain that power.
| nwsm wrote:
| I have no ATT cell service in Boston. I'm on an iPhone 14
| testfrequency wrote:
| Take this with a very fine grain of salt, but I read on a forum
| elsewhere that it was Cisco infrastructure related.
|
| Do I believe that? No clue. I believe it more than people
| speculating the timing corresponds with the solar flare or nation
| state taking it out.
| DebtDeflation wrote:
| Verizon and T-Mo both issued statements that they have no outages
| and the issue is just their customers being unable to call AT&T
| customers. Looks like most of the AT&T network in the US is down
| though.
| peteradio wrote:
| My wife has google-fi and her coworker has verizon. Both of
| them say they can't make calls.
| whynotminot wrote:
| Any chance they're trying to call an AT&T customer?
| nonethewiser wrote:
| and google-fi uses T-Mobile I believe
| cddotdotslash wrote:
| Anecdotal, but I have Google Fi and was on a ~1 hour call
| this AM during the height of the outage and had zero issues.
| inferiorhuman wrote:
| I've been tethering with T-Mobile as my primary internet
| connection and that's been working just fine. Voice also
| works for me with both TMo and Google Fi.
| darkmarmot wrote:
| Well, that's a lie from them, just going my Olympia, Wa on
| Verizon's outage site says I'm SOL -- and have been for 3 days.
| pitaj wrote:
| That sounds entirely unrelated to this outage which began
| only a few hours ago.
| ethbr1 wrote:
| Data point on ATT (via MVNO) in Atlanta: was connected until
| ~11:00 EST, then booted off and haven't reconnected.
| Tarball10 wrote:
| A theory for the reported Verizon/T-Mobile issues is that when
| AT&T went offline, all of those phones went into SOS mode and
| tried to register on the remaining available networks (Verizon
| and T-Mobile) to allow 911 calls to be made. The surge in
| devices registering at once may have overloaded some parts of
| those networks.
| joshstrange wrote:
| Seeing "SOS" only on iPhone currently. I got worried something
| had gone wrong with auto-bill pay since I only noticed after I
| was driving.
|
| It's interesting how naked I feel without access to the internet.
| I reach for it way more often than I would have ever guessed,
| something you only notice when it's not there. Last March my area
| saw large wind storms that knocked out power for almost a week
| (I'm not in a rural area). I can work around the loss of power
| but the cell tower(s) that service my area could not handle the
| load and/or the signal in my house was weak and I was unable to
| load anything. Not having internet was way worse than not having
| power and I ended up driving a few hours away to my parent's
| house instead of staying home.
| bombcar wrote:
| My earliest computers were amazingly capable and powerful
| devices, I could do anything I could think of and spend hours
| and hours on them.
|
| Now my computer is insanely more powerful but without an
| Internet connection it feels dead and useless.
| colanderman wrote:
| Optimize some low-level numeric algorithms, CPU or GPU, it
| brings back that feeling.
| bombcar wrote:
| It's even sadder - I used to be able to play computer games
| for hours offline, now I get about five minutes into even
| the ones WITH on offline mode, and I'm grabbing for a wiki
| or other reference. Ah, some of it is just getting old.
| joshstrange wrote:
| I think some of it is just not having oodles of free time
| to figure it out on your own. When I was young I would
| just keep trying things till I figured out the game, my
| time wasn't worth much or at least I didn't value it
| highly. Nowadays I don't want to spend 1-3 hours figuring
| something frustrating that's blocking my progress. The
| "rush" I get from solving it on my own does not make up
| for the time lost. Also I feel like games made today
| almost expect you will need the wiki/guide to figure out
| certain things. Or at least I often think "How the heck
| was I supposed to figure that out?" when reading the wiki
| for some aspect of a game I'm stuck on.
| bombcar wrote:
| That latter part is certainly true, only a few games
| "offhand" even really _try_ to work "wiki-free"
| (Factorio is perhaps the best here, but Minecraft is
| trying).
| no_wizard wrote:
| Portland, is that you?
|
| This happened to us with the recent storms a month or two back,
| some places didn't have power restored for 2 weeks+
| joshstrange wrote:
| Lexington, KY. Not a massive city but the second largest in
| KY. I left after 2 days of no power and it didn't come back
| on for another 3-5 days more after that depending on where
| you lived.
| pdxandi wrote:
| Portland checking in. Those storms were gnarly and there was
| carnage all around us. Luckily we maintained power and
| internet. We have 11 month old twins and a three year old so
| 10 days without childcare or help was its own challenge.
| flerchin wrote:
| Yes I've felt the same way. I feel like we have an instinctual
| need for social connection that we've filled with internet.
| Luckily, we do still have meat-space friends and family.
| joshstrange wrote:
| Yeah, that was a big reason I went to stay with my
| parents/family. I felt super isolated from my friends (local
| and remote) when I couldn't participate in group
| chats/communicate. Also I just kept picking up my phone to
| look up something or check on something only to re-remember I
| couldn't do anything. I had podcasts and audiobooks on my
| phone which helped but the isolation was a weird feeling I
| hadn't felt before. After I thought about it I realized it
| had probably been a decade or more since I had been
| completely without internet for more than a few minutes. It
| was odd...
| jhickok wrote:
| I started driving across the US at 3am, didn't notice for the
| first few minutes until I tried pulling up the address in Apple
| Maps. Sure was strange following interstate signs for ~10
| hours!
| arprocter wrote:
| NYC ATT Android SMS 'unable to send message'
|
| Strangely sending to that phone from an iPhone looks like it
| sent, but nothing was received
|
| Calling from Android to iPhone gives 'your phone is not
| registered', iPhone to Android plays a 'call could not be
| completed'
|
| iMessage between iPhones seems okay
| teddyh wrote:
| Also: <https://apnews.com/article/cellular-att-verizon-tmobile-
| outa...>
| isaacdl wrote:
| Downdetector's front page[0] seems to indicate issues with other
| carriers as well - Verizon and TMobile. There are are several
| other providers showing on the front page, but I think a lot of
| them are MVNOs on the big three networks.
|
| [0] https://downdetector.com/
| HeatrayEnjoyer wrote:
| Verizon confirmed they have no outage. Seems to be an issue
| contacting ATT subscribers
| lotsofpulp wrote:
| My ATT in Washington state is fine.
| eli wrote:
| I wonder if the smaller number of reports for Verizon and
| T-Mobile is from people on those carriers trying to call or
| text someone at AT&T
| kickofline wrote:
| > DebtDeflation: Verizon and T-Mo both issued statements that
| they have no outages and the issue is just their customers being
| unable to call AT&T customers. Looks like most of the AT&T
| network in the US is down though.
| bochoh wrote:
| Checking in from the Northeast (north of Albany, NY) and can
| confirm no cell service at all this morning (ATT)
| grungydan wrote:
| Isn't it wonderful that we allow almost entirely unregulated
| monopolies that have no actual obligation to provide the service
| for which they take your money?
|
| Telecoms, airlines, insurance scam..ahem companies...
|
| Must be nice. I wonder if I could just stop showing up to my job
| and keep collecting my check. No? Why not? We allow the entities
| that write our laws and have politicians rubber stamp them to do
| it. Seems a mite unfair.
| ejb999 wrote:
| What is your solution? Let the government run all internet
| infrastructure?
|
| Maybe we should let the federal government run it, like it does
| the Social Security Administration - which btw, shuts down its
| website for 4 hours every weekday night and even longer on the
| weekends.
|
| ** They shut down a freaking website for almost 40 hours a week
| ** because they don't have the technology or skills to keep it
| running 24/7...in 2024.
|
| No thanks, the reason that this is even in the news is because
| widespread outages are so rare.
| grungydan wrote:
| Did you miss or just ignore where I agree with you that our
| government is a massive joke? That doesn't mean that I don't
| want a real, functional government that operates for the good
| of its citizenry, or that we don't live in a corporate
| dystopia where we get charged every last cent they can get
| away with and barely provide their purported services. You're
| acting like these networks are otherwise as reliable as they
| say, provide the advertised speeds, and that these companies
| aren't doing the bare ass minimum possible to maximize their
| already egregious profits while paying their employees crap
| like every other company.
| thekrendal wrote:
| Checking in from the St. Louis, MO area, the outage is in effect
| here. Two lines on separate accounts are down.
|
| Thankfully, WiFi calling seems to be functioning.
|
| I can't wait to see a postmortem on this outage.
| flerchin wrote:
| I wonder if an outage like this costs them any money at all. They
| probably save on data transfer and other operating costs.
| mstudio wrote:
| My ATT Phone is in SOS mode. However, ATT's outages status
| reports: > All clear! No outages to report.
| > We didn't find any outages in your area. Still having issues?
|
| https://www.att.com/outages/
| Scoundreller wrote:
| At least they're in SOS mode. When Rogers in Canada had a total
| blackout (cellular, home internet, MPLS, corp circuits, their
| radio stations , everything), phones showed zero bars, but the
| towers were still powered on and doing some minimum level of
| handshake so phones didn't go into SOS mode.
|
| If you tried to make a 9-1-1 call, it would just fail. It
| wouldn't fail over to another network because the towers were
| still powered up but unable to do anything, and Rogers couldn't
| power them down because their internal stuff was all down.
|
| Like a day later they said you could remove your SIM card to do
| a 9-1-1 call. Thanks guys.
|
| Of course, no real info from the provider during the outage.
| Turns out they did an enterprise-risking upgrade on a Friday
| morning and nobody at the org seemed to have a "what if this
| fails plan". CTO was on vacation and roaming phones were black
| too and he thought it was just an issue for him.
|
| https://en.m.wikipedia.org/wiki/2022_Rogers_Communications_o...
| ezfe wrote:
| Some people earlier on this morning said they couldn't make
| 911 calls. I wonder if it was the same issue and perhaps AT&T
| cut the towers completely pending a fix. Purely speculation.
| dangus wrote:
| I'm seeing an outage reported on this page.
| inferiorhuman wrote:
| I'm seeing a variety of outages listed there as of 08:30
| Pacific, mostly landline. There are a couple wireless outages
| shown in Sonoma (and listed as impacting Sonoma and Ventura
| counties). The initial cause is shown as "maintenance
| activity".
|
| https://imgur.com/a/oXZpEX9
| sivm wrote:
| My wife has a Samsung Z fold 3 and she has service. I have an
| iPhone with eSIM in the same house and do not.
| frantathefranta wrote:
| Me and my wife have the exact same iPhone with the same AT&T
| setup. Also on the same latest iOS version. She doesn't have
| service at our house, I do. Bizarre.
| flerchin wrote:
| I wonder how much an outage like this would end up costing them.
| Maybe it would even be net positive as some of their operating
| costs like data transfer would be down.
| neom wrote:
| A similar scale outage in Canada ended up costing the company
| around $3.80 per subscriber ($170MM CAD). Coincidently, I read
| that's about the same amount that was calculated as impact to
| the Canadian economy ($175MM).
|
| https://www.catchpoint.com/blog/rogers-outage
| rglover wrote:
| Middle Tennessee only getting SOS on iPhone (AT&T).
| jrs235 wrote:
| Have you powered your phone off and back on? Does that restore
| service?
| rglover wrote:
| Just tried. No dice. Still getting SOS.
| internet101010 wrote:
| Same thing two blocks away from AT&T HQ in Dallas. Yes, I
| rebooted.
| SirMaster wrote:
| I'm trying to decide if I am surprised or not that something like
| this relies on a single central point of failure. (well I assume
| it's a single point of failure given how this appears to have
| happened nation wide all at once)
|
| You would think that a nation wide cell service would be more
| distributed and that it wouldn't all go down at once like this.
|
| But maybe there are some good enough reasons that there needs to
| be some central components or systems that make it all work.
| tyre wrote:
| Not necessarily a single point of failure. There could be
| standby servers that were promoted to primary when the primary
| failed, then fell over with the huge spike in load. Or they
| were load balanced, multiple failed, and then the others tipped
| over.
|
| In those cases, there were mitigations, but the failures
| cascaded.
| htalat wrote:
| a family member in Pittsburgh, PA does not have service.
| that_lurker wrote:
| "with some of those impacted unable to reach 911 emergency
| services."
|
| How can one carrier going down affect your ability to make
| emergency calls?
| davio wrote:
| https://arstechnica.com/information-technology/2019/08/centu...
|
| I like that they used this 1850's era tech during the outage:
| https://www.wbur.org/news/2018/12/30/911-outage-fire-boxes-b...
| jsjohnst wrote:
| Have AT&T cell signal in Central Jersey, but calls fail and no
| useable internet data connectivity unless using wifi calling.
| Miner49er wrote:
| Interestingly, I'm on an MVNO that uses AT&T and have no issues.
| aclindsa wrote:
| Interesting. I'm also using an AT&T MVNO, but I am affected.
| indigodaddy wrote:
| Is the outage always reflected by how many bars? I have ATT
| yearly prepaid and currently 3 bars, which is about normal for
| current area..
| nonethewiser wrote:
| Tried to use it? Other comments suggest it's down nationwide
| but you are saying you're up?
| saltyollpheist wrote:
| Having bars doesn't necessarily mean calls can be made or
| received. Mine keeps fluctuating between 0 and 5 bars but
| wherever I try to make a call, it claims I'm not registered
| with my current carrier, which uses AT&T as its backbone.
| nonethewiser wrote:
| yeah thats why I asked if he used it
| saltyollpheist wrote:
| NGL I replied to the wrong person. It's been a long
| morning. My apologies and keep on keeping on.
| 13of40 wrote:
| That's something I've noticed in general with 5G -
| basically five bars means you've got a good connection to
| the tower, but you could have bad service due to a
| bottleneck upstream of that. (At least for data - for voice
| that might be more negligible.)
| selectodude wrote:
| That's been an issue for a lot longer than that, at least
| in dense metro areas. Even back in the 3G days I'd have 5
| bars and no data due to tower congestion.
| browningstreet wrote:
| Reno -- AT&T cell and fiber are working normally
| dmane11 wrote:
| No service on my iPhone 15 pro att e-sim here in Chicago
| kennethrc wrote:
| Phoenix area, sending this from my 5G NRSA-enabled router on AT&T
| bonyt wrote:
| Working here, in a subway station in NYC.
| commandlinefan wrote:
| What's weird is that I have two phones with AT&T service (one
| provided by work, one my own). I have service on the work phone,
| but not my personal phone.
| assimpleaspossi wrote:
| https://www.youtube.com/watch?v=ucdZHR75iCM
| bdcravens wrote:
| My wife and I are same account (AT&T, Texas) and hers is down but
| mine isn't. Only difference is my phone is slightly newer (15 Pro
| Max vs 14 Pro)
| thechao wrote:
| My 12 year old has been lording it over the rest of the family,
| for the same reason.
| breischl wrote:
| Maybe yours is set up for WiFi calling?
| inemesitaffia wrote:
| Did I see you on Reddit? As in both are down now?
| bdcravens wrote:
| No that wasn't me.
| chasd00 wrote:
| In the same boat (AT&T/Texas) but reversed, my wife's newer
| iphone was SOS but my older one was not. Her's came back online
| about 1:30PM central after being down since she woke up around
| 6AM central. afaik both my kid's phones were working, I'm sure
| i would have heard howls of anguish if not. Both of their
| phones are maybe 4-5 generations older iphones.
| BuckYeah wrote:
| Our corporate att phones are all down in multiple large campuses.
| About 10,000 phones total
| tmaly wrote:
| I wonder if this is a bad software update to the cell towers?
| a_band wrote:
| AT&T seems down in Eastern TN too. All phones in SOS mode.
| niblettc wrote:
| Hey neighbor! SOS mode here in north AL too.
| nu11ptr wrote:
| Using myATT app doesn't even show my wireless account anymore. My
| entire family account doesn't even show up as a service. Seems
| like a hack or internal issue that deleted accounts? Can others
| confirm whether they see their accounts?
| zeven7 wrote:
| I can see my account. The first thing I did when I saw my phone
| wasn't working was log in and pay my bill, thinking maybe I had
| missed one or something.
| bmitc wrote:
| Their outage status page is also completely broken. Doesn't
| show anything.
| mh- wrote:
| Many of their APIs appear to be intermittently returning 502s,
| leading to strange behavior in their web/mobile apps.
| radicaldreamer wrote:
| That's just people trying to figure out if their service has
| been disconnected rather than it being a network outage
| deckar01 wrote:
| The account management / status tools were slow and flaky on
| the best days. I wouldn't rule out a little extra traffic
| knocking them out. Correlation is not necessarily causation.
| partiallypro wrote:
| From what I read just a bit ago, basically there is a problem
| with the database of SIM numbers. So, SIMs all just dislodged
| from the network because they lost their network authorization.
| That would lead one to believe it was a botched software push.
| I imagine online accounts get this information somehow which
| could explain the portal being broken. I have a prepaid hotspot
| and it works fine, but none of the "family plan" month to month
| contract phones work. I also wonder if there's a physical SIM
| vs eSIM situation that could explain "newer" models working.
| Kon-Peki wrote:
| In our house this morning, the two phones with physical SIMs
| worked fine and the two phones with eSIMs were SOS mode only.
|
| I could log into my AT&T account just fine and all phones
| showed up correctly.
|
| (I'm submitting this from an AT&T 5G connection, no WiFi
| nearby)
| chasd00 wrote:
| aligns with my experience too. My wife's newer phone was
| sos but my older one was fine. Both ATT.
| kjellsbells wrote:
| That would be consistent with the symptoms. Big telco
| networks are hierarchical with most functions pushed to
| regional data centers with a very small number of services in
| a redundant pair or trio of central data centers. Subscriber
| database (HSS, UDR) would be one such function.
|
| The cause of a failure of the HSS could be manifold, ranging
| from router failures to software bugs to cyber attack
| (databases of 100M+ users being a juicy target).
|
| One slightly scary observation from NANOG was that FirstNet,
| the network that ATT built for first responders, was down.
| That would be ugly if true and I'd expect the FCC to be very
| interested in getting to the bottom of it.
| betaby wrote:
| Are there any technical details?
| fuzzfactor wrote:
| AT&T mobile 5G hotspot lost internet a few hours ago.
|
| Resets futile.
|
| Shows "Mobile Broadband Disconnected"
|
| However can still ping google, with excellent under 10ms
| response.
| wingspar wrote:
| I'm on Cricket, which is a prepaid cell plan 'subsidiary' of
| AT&T, and calls/internet are work fine.
|
| So not a network outage, but account related?
| jtbayly wrote:
| That would make sense, except that wifi calling is still
| working.
| jrochkind1 wrote:
| My Cricket phone is not fine. Android phone, giving me
| regularly repeated alert messages "No voice service.
| Temporarily turned off by your carrier."
|
| I think your experience is just that it's not 100% outages,
| some people still have service.
| seatac76 wrote:
| Verizon is working just fine.
| acefaceZ wrote:
| What is the big deal? Just use your HAM radio.
| xyst wrote:
| I'm more of a smoke signal guy.
| engineer_22 wrote:
| That reminds me, I passed the General and Tech but didn't
| follow through on my application...
| zingababba wrote:
| I agree unironically
| chomp5977 wrote:
| Funny you say this lol I did exactly that
| roamerz wrote:
| I wonder if this extends to the Public Safety part of AT&T -
| FirstNet.
| pgrote wrote:
| Yes, it does. Family member in the medical field is
| experiencing issues.
| roamerz wrote:
| Dang that's a bad look for them. Hopefully they can provide a
| thoughtful and honest postmortem.
| hoppyhoppy2 wrote:
| >medical field
|
| >postmortem
|
| Let's hope that everyone lived so a postmortem won't be
| necessary :)
| EvanAnderson wrote:
| It does. I was the on call support technician for a public
| safety answering point (PSAP-- aka a 911 center) this morning.
| At 04:14 Eastern today I received a call that the law
| enforcement and fire mobile units on FirstNet all disconnected
| from the VPN.
|
| The AT&T land lines (CAMA trunks provided by the ILEC Frontier)
| that handle the 911 service did not fail. Only mobile service
| failed.
| roamerz wrote:
| Yeah I have had to make that call myself many times in the
| past. Never with FirstNet but definitely with AT&T and
| Verizon. It would be awesome if the carriers could put out a
| reliable announcement to the affected accounts that there is
| an outage - it would definitely simplify the pre-support call
| triage an 1AM.
| msrenee wrote:
| Eastern Nebraska here. Woke up to no connection at 4am. It was
| restored by 8 am. Now at 10, it's out again.
|
| Edit: And back again a few minutes later.
| kraig911 wrote:
| My military friends said to buy gas today... Another friend said
| solar flares. I can't tell if it's a foreign actor or nature.
| throwawaaarrgh wrote:
| Occam's razor says it's none of those things. The simplest
| explanation is somebody just fat-fingered a change, which
| happened to coincide with a lot of other things to result in a
| big outage. Happens all the time.
| kevin_nisbet wrote:
| To everyone trying to speculate on the root cause, I haven't seen
| enough information in any of the comments to really draw any
| conclusions. Having worked on several nationwide cellular issues
| in Canada when I worked in telecom, we saw nationwide impacts
| based on any number of causes.
|
| - A new route injected in the network caused the routing engines
| on a type of cellular specific equipment to crash nationwide.
| This took down internet access only from cell devices nationwide.
| But most people didn't notice because it happened at 2AM
| maintenance window and was fortunately discovered and reversed
| before business hours why the routing engine was in a crash loop.
|
| - A tech plugged in some new routers, and the existing core
| routers crashed and rebooted. While the news worthy impact was
| just a regional outage for something like 20 minutes, we
| discovered bugs and side effects from the Pacific to Atlantic
| coasts over the next 12 hours. So when you say you're impacted at
| location x, that data point could be everyone is down in the
| area, many people are having issues, or only one or two people
| have issues spilled over to some other region. This is why seeing
| it does or doesn't work in location x is limited value, as almost
| every outage I've investigated could result in some people still
| having service for various reasons. The question is in a
| particular area is it 100% impact, 50% impact, or 0.001% impact.
|
| - A messaging relay ran into it's configured rate limit. Retries
| in the protocol increased the messaging rate, so we effectively
| had a congestion collapse in that particular protocol. Because
| this was a congestion issue on passing state around, there were
| nationwide impacts, but you still had x% chance of completing
| your message flows and getting service.
|
| And then there was the famous Rogers outage where I don't
| remember them admitting to the full root cause. It's speculated
| that they did an upgrade on change on their routing network,
| which also had the side effect of the problem booting all the
| technicians from the network. Then recovery was difficult because
| the issue took out the nationwide network and broke the ability
| for employees to coordinate (you know because they use the same
| network as all the customers who also can't get service). All the
| CRTC filings I reviewed had all the useful information redacted
| though, so there isn't much we can learn from it.
|
| So it's fun to speculate, but here's hoping at the end of the day
| ATT is more transparent then we are in Canada, so the rest of the
| industry can learn from the experience.
| Scoundreller wrote:
| Rogers, of course, blamed their vendor (Ericsson I believe it
| was). Rogers can do no wrong!
|
| Of course, was fun to see yet another huge org have no back-
| out/failure plan for their potential enterprise-breaking
| changes. No/limited IT 101 stuff here.
|
| The only positive thing we learned was that the big 3 (really
| 2) telcos thought it would be a good idea to give eachother
| emergency backup sims for the other network to key employees in
| case their network went down. They did that in 2015, but better
| late than never.
|
| Fun that Rogers used the same core for wireless and wired
| connections, so many of us were in total blackout, even if we
| used a 3rd party internet provider that ran over Rogers. Like,
| everything including their website was down, corp circuits,
| _everything_ with non-existent comms from Rogers.
|
| Thankfully my org was multi-homed and switched over its
| circuits at 6am so on-site mostly continued without issue.
|
| Also fun where the towers remained just powered on enough for
| phones to stick to them but not be able to do anything, so
| 9-1-1 calls would just fail, instead of failing-over to other
| networks. Seems like a deficiency in the GSM spec (or Rogers
| SIM programming?) that I don't think was actioned on.
|
| https://en.m.wikipedia.org/wiki/2022_Rogers_Communications_o...
| MichaelZuo wrote:
| > Fun that Rogers used the same core for wireless and wired
| connections, so many of us were in total blackout, even if we
| used a 3rd party internet provider that ran over Rogers.
|
| If it ran over Rogers circuits then why wouldn't it go down
| too? Isn't that the case everywhere?
| Scoundreller wrote:
| I just know that a part of Rogers' response was to separate
| their cores between wireless and wireline so that the risk
| of both going down simultaneously would be reduced.
|
| The 3rd party providers aren't white-label resellers, but
| there's obviously some overlapping susceptibilities to
| going down when Rogers breaks something. Depends what they
| break, and in this case, it took them down too.
| kevin_nisbet wrote:
| > Also fun where the towers remained just powered on enough
| for phones to stick to them but not be able to do anything,
| so 9-1-1 calls would just fail, instead of failing-over to
| other networks. Seems like a deficiency in the GSM spec (or
| Rogers SIM programming?) that I don't think was actioned on.
|
| Actually, I think this is going to change after the Rogers
| outage, it's just slowly happening behind the scenes so it's
| not getting much attention these days. The government has
| mandated a lot of industry response to failover between
| providers... we'll see where they land after all the lobbying
| happens. I do think implementations are changing a bit around
| this, mostly in the phones so that they give up and go into a
| network scan if the emergency call is failing.
|
| I worked mostly on core network stuff, so I was a layer
| removed from the towers, but if they hadn't lost management
| access they would've been able to tell the tower to stop
| advertising the network and 911 service. I do understand the
| question of from a vendor implementation perspective of how
| automatic this should be though... because automation in this
| regard does have some of it's own risks and could complicate
| some types of outages or inadvertently trigger and confuse
| recovery of problems.
|
| I'm with you though there should be an automatic mechanism to
| fail over to other network operators, I just haven't thought
| through all the risks with it and I hope the industry is
| taking their time to think through the implications.
| Scoundreller wrote:
| > I do think implementations are changing a bit around
| this, mostly in the phones so that they give up and go into
| a network scan if the emergency call is failing
|
| It seems like this is a global problem, since all Rogers-
| subscribed devices in a Rogers reception area couldn't make
| 9-1-1 calls. But could be a SIM coding issue and not
| afflict other providers elsewhere.
|
| I just always imagined the GSM spec was so resilient that
| you could _always_ make a 9-1-1 call if a working network
| was available but this outage proved that wrong. Surprising
| to learn in 2022.
|
| Of course it's Canada, so I agree with them that the
| thought of letting users failover to a partner for
| everything would thrash the partner's networks. Even though
| Canadian subscriber plans are laughably low in monthly data
| and population density is low (per the telecom's usual
| excuse for our high prices) it turns out the telecoms still
| underbuilt their networks to have less capacity than what
| other networks internationally built out to support plans
| available on the international market (e.g. close to truly
| unlimited data/free long distance calls)
| toast0 wrote:
| > I just always imagined the GSM spec was so resilient
| that you could always make a 9-1-1 call if a working
| network was available but this outage proved that wrong.
| Surprising to learn in 2022.
|
| The X is broken but claims it isn't stops failover
| pattern is strong all over networking. It's not unusual
| to see it in telco root cause analysis.
| kevin_nisbet wrote:
| > I just always imagined the GSM spec was so resilient
| that you could always make a 9-1-1 call if a working
| network was available but this outage proved that wrong.
|
| As I recall it is slightly more nuanced than this and was
| particular to the failure mode, and has a couple of
| different things aligning to create the failure mode.
|
| If you're phone is just blank, no sim card. To make an
| emergency call, it has to just start scanning all the
| supported frequencies. This is very slow, tune radio,
| wait for the scheduled information block that described
| the network on the radio protocol. See if it has the
| emergency services bit enabled. If not, tune to next
| frequency and try again. I used to remember all the
| timers, but almost a decade later I can't remember all
| the network timers for the information blocks.
|
| The sim card interaction, is say you're at home and you
| boot up your phone with 100% clean state. You don't want
| to wait for this scan to complete, so the SIM card gives
| the phone hints about which frequencies the carrier uses,
| so start on frequency x to find the network. But if you
| roam internationally, it can take alot longer to find a
| partner network, and there are some other techs around
| steering to preferred partners, but I don't know that
| those come into play here. I don't know but would be
| surprised if there is a SIM option to try and pin the
| emergency calls to a network, I think it's more likely
| the interaction is this hint on where to start the scan.
|
| The way the rogers network failed, it appears to me it
| caused the towers to stay in a state where they
| advertised in their radio block the network was there,
| and the 911 bit was enabled so the network could be used
| for emergency calls. This is where I don't really have
| the details since they haven't been public about it, how
| much of their network was still available internally.
| Maybe the cell towers could all see each other, that
| network layer was OK, and the signalling equipment was
| all talking to each other as well. That's the part I
| don't really know and have to speculate, as well as the
| tower side since I was a core person. So because the
| towers had enough service to never wilt themselves, they
| kept advertising the network, along with the 911 support.
| But then when you try to activate an emergency call,
| somewhere in the signalling path, as you get from tower
| to signalling system, to the voip equipment, to the
| circuits to the emergency center the outage knocked
| something out. Oh and for all these pieces of 911
| equipment, there are two of everything for redundancy...
| two network paths, two pieces of equipment, etc.
|
| And because they lost admin access to their management
| network, no one could go in manually and tell the towers
| to wilt themselves either.
|
| If the towers had just stopped advertising 911 services,
| the phone would fall back into the network search mode as
| I described when you have no sim card. It just starts
| scanning the frequencies until it see's an information
| block for a network it can talk with the emergency
| support advertised to and does an emergency attach to the
| network that the carriers will all accept (An
| unauthenticated attach for the sole purpose of contacting
| an emergency center).
|
| So my suspicion is because carriers are so used to we
| have two of everything, and all emergency calls are
| marked for priority handling at all layers of the
| equipment (they get high priority bits on all the network
| packets and priority CPU scheduling in all the
| equipment), this particular failure mode where there was
| a fault somewhere down the line, and they lost control of
| the towers to tell them to stop advertising 911 services
| all sort of played together to create the failure mode.
| mjevans wrote:
| Multi-faceted failure mode.
|
| 0) At the network terminal level (mobile phone): at least
| for emergency calls if a given network fails to connect,
| fail over and try other networks. Even if the preferred
| networks claim to provide service.
|
| 1) At the network level: failure thresholds should be
| present. If those thresholds are crossed enter a fail-
| safe state. This should include entering a soft offline /
| overloaded response state.
|
| 2) Where possible critical data paths should cross-route.
| Infra Command and Control and Emergency calls in this
| case. Though if Roger's issue was expired certs or
| something the plans for handling that get complicated.
| Scoundreller wrote:
| it's that "0" level that surprised me the most here.
|
| Days later, Rogers said you _might_ be able to pull out
| /disable your SIM card to call 9-1-1, but then it
| depends: if Rogers is the strongest network, you might
| end up in the same predicament anyway.
| Fatnino wrote:
| Say there is an outage at the 911 call center. Now you try
| to call, don't get through, and your phone writes off that
| tower. Who were you planning to call after 911? Too bad,
| should have placed that call first.
| vlovich123 wrote:
| Your phone would try other towers from other providers.
| If 911 is experiencing an outage that's a separate issue
| that needs to be mitigated at a different layer. Even
| still, 100% uptime is difficult and expensive.
| flippy_flops wrote:
| The speculation is fascinating. For most people, their guess is
| a reflection of themselves. Is there a term for that? This is a
| gross generalization, but I've seen... - Science people
| guessing solar flares - My "right-wing friend" guessed
| international hackers - I, myself, guessed it was a botched
| software release - Someone in this post commented their
| military friend says get gas
|
| And yet, like everyone else, I genuinely feel that I'm probably
| right
| Scoundreller wrote:
| > Someone in this post commented their military friend says
| get gas
|
| The Rogers outage in Canada took out the nationwide debit
| card payment network because that infra depended on Rogers.
| Credit cards still worked, but depends on your station's
| access to make the transaction. And no shortage of shops
| running their POS "in the cloud" and needing to close if they
| lose internet access. I actually did have to lend cash to a
| colleague to buy gas to get home during that Rogers outage.
|
| All it takes is for one pipeline valve to depend on a
| cellular connection for billing to get the whole line
| shutdown.
|
| And ugh, we hope for a botched software upgrade too, but a
| corp cyberattack is so much harder to recover from so can't
| be discounted from the realm of possibilities. I know that's
| where my mind went with Rogers given how thorough their
| outage was.
|
| Was kinda unimaginable for a total outage to happen with no
| org comms ready to go in the pipeline. Your plans are
| supposed to have those comms ready for a bad update that
| you've been planning for weeks. It's a cyberattack where you
| may stay silent. But I know Rogers isn't going to admit fault
| until they find someone else to blame.
| charcircuit wrote:
| PoS devices are usually networked. If you don't validate
| transactions in realtime you would later validate in batch,
| but that has more risk than validating at the time of
| transaction.
| Scoundreller wrote:
| > If you don't validate transactions in realtime you
| would later validate in batch
|
| yeah, a lot of orgs just don't enable that (or don't have
| a process to enable it as required, and have difficulty
| pushing out a notice to do so if the network is down!).
|
| Also can only do offline credit card transactions. Can't
| with our Interac (Canadian-only) debit network. Unsure
| about Visa/Mastercard debit transactions.
| rescbr wrote:
| > Unsure about Visa/Mastercard debit transactions.
|
| AIUI, the debit card itself enforces online confirmation,
| even if the transaction goes through the credit card
| rail.
| ShamelessC wrote:
| > Is there a term for that?
|
| Projecting, biased.
| BuyMyBitcoins wrote:
| My speculation is: "Higher-ups kept demanding that
| technicians 'do more with less' in order to deliver on
| quarterly metrics and now we're finally seeing the cumulative
| result of employees being stretched thin, underpaid, and
| overworked."
|
| You are welcome to infer as to why I'm thinking this way!
| spazx wrote:
| This is my bet; and mayyybe some external bad actors taking
| advantage of the situation on top of that.
| nonethewiser wrote:
| Obviously you're a self loathing executive.
| gnuser wrote:
| The ops team can run the whole company and better without
| the C-Suite is my impression of modern day SV. Agile
| stickers on waterfall gates...
| bregma wrote:
| So how is the job search going?
| mlyle wrote:
| > And yet, like everyone else, I genuinely feel that I'm
| probably right
|
| This is the thing with black swan events. The more pedestrian
| explanations are almost always true, but then there's a tiny
| fraction of the time where you're much, much better off
| having taken a bit of an alarmist view.
| booleandilemma wrote:
| A type of availability bias, maybe?
|
| https://en.wikipedia.org/wiki/Availability_heuristic
| nonethewiser wrote:
| I literally caught myself thinking about a cyberattack merely
| because its sort of exciting (albeit terrible). And then
| realizing despite its prominence in my mind, it's probably
| not the most likely cause (although certainly plausible
| still). And furthermore, that my mind gravitates to that
| without any real information suggesting it over other
| explanations. More about fearing for the worst instead of
| what you want I think.
| giancarlostoro wrote:
| > some people still having service for various reasons.
|
| I assume roaming being one of the top reasons no?
| Y3Jlbmd1dGEK wrote:
| I don't think it has anything to do with routing by looking at
| the comments on down detector. Many people report they are in
| the same household, and one person out of six (in the
| household) experiences the problem while all are on ATT. It
| sounds more like an upgrade that went through halfway, or,
| considering the time it happened, maybe a rollback that went
| only half through.
| progbits wrote:
| No info in NANOG yet but expect some in this thread in the coming
| hours:
| https://mailman.nanog.org/pipermail/nanog/2024-February/2250...
| nathanyz wrote:
| Latest AT&T Statement: "Our network teams took immediate action
| and so far three-quarters of our network has been restored," the
| company said. "We are working as quickly as possible to restore
| service to remaining customers."
|
| Still down for me though.
| IAmGraydon wrote:
| Still down for me too.
| TheCaptain67 wrote:
| down all morning in ATL but back up at 1PM EST
| themaninthedark wrote:
| Back up in Cartersville, north of ATL @ 13:12. Oddly enough
| my text messages say they went through at 12:43 but my
| response to someone's message when once my phone had
| everything roll in at once is timestamped 13:12
| partiallypro wrote:
| Also still down for me here around Nashville.
| DHPersonal wrote:
| Just came back up for me in Oklahoma City, OK.
| Scoundreller wrote:
| 3/4 might just mean the internal facing side, which is still
| progress, but doesn't mean any improvements for end-users.
| wizerdrobe wrote:
| Anecdotally, I woke up to no signal / "SOS" mode on my iPhone
| this morning at around 0600 and had service restored around
| 0830 in South Carolina. However, a coworker in Memphis
| confirmed he was still out of service at 1000 so it's
| regional restoration.
| Scoundreller wrote:
| I always wonder instead of a regional restoration, if they
| would "disable" segments of SIMs/accounts randomly to avoid
| lightning strike (it's not a DDoS...) their network as they
| turn things back on. Depends on what the recovery method
| is, but could be problematic to turn everything back on at
| once.
| fragmede wrote:
| I've heard that called the thundering herd problem.
| chasd00 wrote:
| my wife's phone was SOS when she woke up at about 6AM
| central and finally become operational around 1:30PM
| central.
| mulippy wrote:
| This is affecting roughly 0.02% of the US population; is it
| really that bad?
| bitmasher9 wrote:
| What percentage of the population would need to be impacted
| before a tech failure is relevant on a technical news site?
| k_roy wrote:
| Seven. Exactly 7%
| syntaxing wrote:
| Assuming you're using the 74K reported number by Downdetector,
| that's a self reporting system. The real amount of people
| effected is probably magnitudes beyond that.
| JohnMakin wrote:
| Was out since about midnight pacific time til ~7am. I just used
| my ATT network to respond to a MFA prompt and the device thought
| I was ~120 miles from where I currently am.
| stuart73547373 wrote:
| probably unrelated but a nurse friend mentioned that a healthcare
| system got hacked last night (didn't have further details)
| organsnyder wrote:
| Is that nurse friend senior leadership in their organization?
| Otherwise, it's extremely unlikely that they would have any
| knowledge of an event like this so soon after it occurred. And
| the people who do have this knowledge (should) know not to talk
| about it until after an investigation has been conducted.
|
| Many systems issues are mistakenly thought by non-technical
| users to be "hacking".
| vaxman wrote:
| https://www.usatoday.com/story/news/politics/2024/01/31/fbi-...
|
| Note: This site does not like to entertain contentious topics and
| has rate limited me over my prior post about factory and foundry
| installed malware in Chinese manufactured equipment because I did
| not respond to a suspicious demand for proof, instead opting for
| a sarcastic reply. I can live without being associated with Garry
| Tan's company, YCombinator. (I thought to post this link while
| posting to another site, not because I hangout here.)
| hamandcheese wrote:
| Late yesterday, I tried to place a 911 call while driving on the
| 101 north toward San Francisco.
|
| Before I could complete my report, the call dropped, and for the
| next 10 or so minutes I had no service, only "SOS".
|
| I'm on Verizon, and the timing doesn't match up with this
| headline, but now I'm suspicious.
| organsnyder wrote:
| Sounds like a standard dropped call due to a dead zone.
| gnicholas wrote:
| VZW doesn't have dead zones on 101 in the Bay Area, in my
| experience.
| xyst wrote:
| Sounds typical for Verizon tbh
| rglover wrote:
| Pure speculation, but CISA released this [1] a few weeks back and
| tweeted [2] it out.
|
| [1] https://www.cisa.gov/news-events/cybersecurity-
| advisories/aa...
|
| [2] https://twitter.com/CISACyber/status/1758495005176447361
| 8b16380d wrote:
| Back up for me (Raleigh, NC)
| macshome wrote:
| It just came back for me as well in Winston-Salem, NC
| kkdeligate wrote:
| I live in Houston TX and have 4 lines on my account. 2 of them
| don't have data yet including myself. My att account don't show
| our numbers or the status of the lines. I think the outage
| started about 2am while I was listening to a podcast. Calls and
| text on WiFi works though.
| jakedata wrote:
| Service resumed in the Boston area about 15 minutes ago. No muss
| no fuss, just magically resumed like nothing ever happened. What
| they share of the post-mortem should be interesting.
| alberth wrote:
| I'm on AT&T and impacted (no cell services since I woke up this
| morning).
|
| What's also odd is I'm not even able to log into my ATT.com
| account.
|
| When I try to log in, it states "want to pay your bill".
|
| So my immediate concern was, did my credit card expire and my
| service was turned off.
|
| I'm still hoping that's not the case, and it's just that I'm
| impacted by this outage (and not something else).
| xyst wrote:
| ATT used to be the premium cellular provider in the US. Guess
| they got too big and neglected their ops side.
| theturtle32 wrote:
| Here in Long Beach, CA, my husband and I both had our AT&T phones
| go into SOS mode at about 1:20am local time, and it resolved for
| both of us about an hour later, and is still resolved.
| chevman wrote:
| Outage Over
|
| Status: Restored AT&T FINAL, Service Degradation, Global Smart
| Messaging Suite AT&T Global Smart Messaging Suite
|
| Event description: FINAL, Service Degradation Impacted Services:
| MMS MT Start time: 02-21-2024 22:00 Eastern, 21:00 Central, 19:00
| Pacific End time: 02-22-2024 11:00 Eastern, 10:00 Central, 08:00
| Pacific
|
| Downtime: 780 minutes
|
| Dear Customer, We are writing to inform you that Global Smart
| Messaging Suite is now available. The MMS MT service has been
| restored and our team is currently monitoring Thank you.
|
| AT&T Business Solutions Kind Regards,
|
| The AT&T SMS Service Administrator
| nonethewiser wrote:
| I think a communication like this should include that they are
| investigating the root cause (assuming they aren't completely
| sure) and that they will share it, and state where.
|
| Maybe im reading to much into it but it bothers me that thats
| not in the communication.
| freedomben wrote:
| They most certainly are investigating the root cause, and
| probably there's a witch hunt developing, but as far as
| customers go I would expect AT&T's attitude to be "none of
| your business." I've worked with many of these types of
| companies before, and outside of the occasional cool CS rep,
| their cultures are lots of information hoarders and
| responsibility dodgers. Taking responsibility for a problem
| is a good way to ensure you never get promoted.
| keanebean86 wrote:
| "A recently departed employee had a core router's power
| going through a wall switch. This was done to facilitate
| quick reboots. A cleaning contractor turned off the switch
| thinking it was a light. It took us several hours to
| determine the situation and restore power"
| dv_dt wrote:
| I think the telecom issue playbook is significantly different
| than the SaaS playbook. Not sure if that's just cultural or
| if there are other drivers - maybe paying customer telecom
| interfaces are simpler and more closed than typical SaaS?
| aksss wrote:
| IME, telecom as an industry is highly focused on the RCA,
| ICA, and uptime, and has had that embedded culturally for
| decades. Sharing the information publicly doesn't have much
| value, in the balance, unless there are a string of
| incidents where an acute perception problem needs to be
| addressed. This would more likely result in a marketing and
| advertising strategy rather than the sharing of technical
| RCA details. Additionally, one must consider that not all
| RCA details are fit for public disclosure. _You_ may be
| interested in deets, but John Q. Public is not interested
| beyond "Is it fixed yet?". If you want insider perspective,
| work in/with the industry. It's fascinating stuff.
| wolverine876 wrote:
| Doe that include cellular voice calls?
| doctator wrote:
| back up in NY
| davidjade wrote:
| It's likely unrelated, but bad timing for AT&T as they have
| applied to end landline service in some areas of California.
|
| https://www.wired.com/story/att-landline-california-complain...
| jader201 wrote:
| Something I'm not seeing discussion on:
|
| What is/were the cascading effects of this, particularly for
| drivers?
|
| Many people in buildings were unaffected, as they could fallback
| to wifi. But I imagine this had a pretty broad impact to drivers.
|
| Just a few things I can think of:
|
| - Packages delayed (UPS, FedEx, Amazon, truck drivers, etc.) for
| drivers that relied on their phone's mapping apps to get them to
| their deliveries
|
| - Uber/Lyft/taxi/etc. drivers not able to get directions to their
| pickups/dropoffs
|
| - Traffic worsened because drivers weren't able to optimize their
| routes, or even get directions to their destination
|
| Maybe larger companies have their own infra for this, or have
| redundancy in place (e.g. their own GPS devices)?
|
| I'm curious to hear thoughts on whether these (and others) were
| impacted, or if there are ways they're able to get around this.
|
| Also, unrelated to drivers, I can imagine there is/was a higher
| risk of not getting treated for emergencies due to not being able
| to make calls (I'm not sure whether/how emergency calling was
| impacted).
| standardUser wrote:
| Worth noting that GPS does not rely on cell service.
| Scoundreller wrote:
| The routing does though.
|
| I have Google offline maps downloaded for areas I end up in
| just in this case. Gotta do traffic rerouting the old
| fashioned way though.
|
| Or have an old-school GPS map thingy in your glovebox.
|
| (Also have kiwix and a whole archive of Wikipedia on my
| phone).
|
| I wonder if meshtastic communicators sales took off during
| this. How's LoRa traffic these days?
| freedomben wrote:
| Yes, much that we think of as Google Maps relies on API
| calls made to the backend. Plus this assumes that you
| downloaded the offline maps ahead of time, which in my
| anecdotal experience is not something that most people
| really consider. GMaps does (or did at one time at least)
| have a neat feature of auto-downloading your home area map
| but, the one time I needed it, it didn't work.
| Scoundreller wrote:
| > which in my anecdotal experience is not something that
| most people really consider
|
| Thankfully I'm in Canada where it's not impossible to end
| up in the sticks with no service.
|
| Chewing through your handful of gigabytes/month of data
| wasn't hard. Only in the past year or so have double
| digit gigabyte/month data plans become cost-effective.
|
| And our roaming prices are extortionate, so for jaunts
| over the border (or internationally), I'll sometimes go
| "naked".
| ianburrell wrote:
| Google Maps does offline routing. It doesn't do traffic
| routing but updating routing is better than nothing.
| bombcar wrote:
| The "Here" app or whatever it is called did offline maps
| and offline routing decently enough. It wasn't perfect, but
| it worked for "here to there", even if it didn't find the
| best possible route.
| a_gnostic wrote:
| Carriers have mapping independent of networks. Drivers keep
| personal GPS too. You would lose traffic and road
| conditions, I guess, but nothing proper trip planning
| wouldn't cover.
| Scoundreller wrote:
| > Drivers keep personal GPS too.
|
| Do they? I know there are a lot of old units out there
| but I figure people would have tossed them.
|
| At least I've found Waze has been pretty good at starting
| off with wifi and loading the map of the whole journey
| after coverage was lost with some resilience for
| stops/detours.
| ezfe wrote:
| Apple Maps has offline navigation with historical traffic
| included
| 01HNNWZ0MV43FF wrote:
| I've been using this on Android for a couple years and love
| it: https://en.wikipedia.org/wiki/Organic_Maps
|
| You click a few buttons to download OSM tiles and then it
| does routing. The latest OSM even has a decent amount of
| stores, restaurants, etc., listed.
| treflop wrote:
| I am consistently in areas with zero cellular service and
| I'm reasonably sure Google Maps will route offline. At
| least, I've never switched to another mapping app because I
| couldn't route -- it's more usually because Google Maps is
| more primitive areas is kind of detail-less.
|
| But even if it doesn't, there are a ton of offline map apps
| that use OpenStreetMap data.
| LeifCarrotson wrote:
| Google Maps and now Apple Maps (as of ~6 months ago) have
| offline maps, but not by default. If you enable and download
| them for your area of interest you can use a subset of the
| normal app.
|
| I make sure to have this around my usual area and anytime I
| travel to an area with poor coverage, plus my Garmin watch
| has offline maps and GPS everywhere, but this is not typical.
|
| OSMand usage is even less common.
| tnel77 wrote:
| Offline maps are a life saver in areas with bad coverage.
| One of the first things I setup for a new phone or when I'm
| headed somewhere new on vacation.
| freedomben wrote:
| This is one of the most interesting differences I often
| notice between users who rarely leave the city and those
| who routinely leave. Offline functionality often seems
| unnecessary at best and absurd at worst to the former
| group, while the more rural/remote the person the more
| they value offline functionality. For the most extreme
| example, talk to the average person who lives outside of
| Anchorage or Fairbanks in Alaska, and they only really
| care what the app _can_ do when it 's offline as that is
| it's assumed status when on the go (disclaimer: I moved
| out of alaska a little over 5 years ago so things might
| have changed somewhat).
| ghaff wrote:
| Yeah, if I'm going to travel internationally or if I'm
| somewhere I know I'll have spotty cell service, I'll
| download maps. I should probably be better about doing it
| in local areas where I "assume" things will be fine.
| tnel77 wrote:
| I grew up in a rural area and lived in Colorado for a
| while. Going home or venturing into the mountains often
| resulted in bad service so it just became second nature.
| Good observation!
| maxerickson wrote:
| Lots of people dislike the design choices in OSMAnd, so
| it's worth mentioning that there are lots of apps that use
| OSM data and provide offline maps and routing.
| fhdkweig wrote:
| agreed. On Google Maps app, there is a feature called
| "offline maps" which allows a user to select a rectangle on a
| the map and download all the street info inside it. A whole
| US state can fit in less than a few hundred megabytes. I have
| all the city I live in downloaded so I can go on walks
| without needed to use my data plan.
| a_gnostic wrote:
| Not as useful as back when google maps required a 5GB
| download IIRC
| jader201 wrote:
| That's assuming you have it on and updated before you hit
| the road.
|
| I think it's off by default, and I'm guessing most people
| haven't thought to turn it on, or are even aware of it.
| patmorgan23 wrote:
| I'm pretty sure maps caches the data around you if you've
| used it somewhat recently. It saves Google bandwidth too.
| jader201 wrote:
| I'm not so sure.
|
| Anecdotally, I've made it to a remote destination using
| Maps, then hopped back in the car an hour later (with no
| signal), and it couldn't load anything. This seems to
| happen quite often.
| Scoundreller wrote:
| Maps used to expire after 30 days (no idea why), and the
| auto-updating while on wifi wasn't great unless you were
| in the app forcing it update. Nowadays they last 365d.
| offmycloud wrote:
| GPS on most cell phones uses data connection to download
| current satellite data in order to decrease the time from
| cold start to GPS lock. Lack of cell or WiFi can cause GPS to
| take 5-15 minutes to "search the sky" and download satellite
| data via low bitrate channel under poor signal conditions.
|
| https://en.wikipedia.org/wiki/Assisted_GNSS
|
| Edit: You can think of it as a CDN for the GPS almanac.
| sobriquet9 wrote:
| From cold start. Most starts are not cold. The phone knows
| where it is, approximately, what time it is (within a
| second or so, from built-in RTC), and orbital parameters of
| the satellites overhead (maybe without the latest
| corrections).
|
| My Garmin watch gets a GPS lock in way less than 5 minutes
| without any cellular connection.
| dfadsadsf wrote:
| That's just not true for modern phones. I use iPhone on
| hikes without cellular connection and GPS lock is
| instantaneous. Organic Map app is great for hiking.
| rpd9803 wrote:
| worth noting that without cell service, GPS can reliably give
| you time, lat, long and elevation. So if previously you had
| no actual map downloaded, or an old or out of date map, you'd
| just get a pretty accurate dot on an inaccurate map, or just
| raw coordinates.
| panarky wrote:
| Also can't login to sites that require SMS 2FA.
| danesparza wrote:
| Well ... let's be honest: SMS 2FA shouldn't be a thing.
|
| TOTP or stronger, please.
| SpaethCo wrote:
| TOTP or SMS, it's just another text password you're
| entering in that's fully phishable.
|
| TOTP just "feels" more secure.
| ezfe wrote:
| SMS 2FA is a code that you're entering from a phone
| number. The "risk" is that your phone number can be
| ported without your permission, and then someone else can
| get the code.
|
| TOTP is more secure because it isn't tied to a phone
| number. You're right that it's still phishable but that's
| not the point.
|
| In both cases, the primary benefit to the general
| population is to have a rotating credential that, if one
| website is hacked, is useless on another website.
| throwway120385 wrote:
| TOTP is more secure in that you can't be simjacked by
| someone impersonating you in the cell phone store.
| jjav wrote:
| No, TOTP is far more secure because it has no dependence
| on a third-party who can mess up in many ways (Denial of
| service like in this case by being unavailable,
| Impersonation by allowing SIM swaps or intercepting
| messages directly).
|
| You fully control how to store the TOTP seed and how you
| compute the value, so it is far more secure.
|
| Yes, it can be phished if you fall for that, but it
| removes several attack vectors.
| Scoundreller wrote:
| > You fully control how to store the TOTP seed
|
| Sorta. The seed still needs to be issued to you in some
| way.
| giancarlostoro wrote:
| This really shouldn't be the only way to verify its you if
| its going to prompt you every single time.
| cmg wrote:
| My service came back around 1:30PM in Connecticut. Data and
| calls are working fine. I requested a 2FA code at 2:30 from a
| service that only offers SMS. An hour and a half later, I
| still haven't gotten it.
| dktalks wrote:
| If you use Google Maps, it will automatically prompt you to
| download a map of the area if there is known poor coverage. It
| also has automatic (?) local maps.
| Scoundreller wrote:
| One beef of mine with Google's offline maps is that they're
| only driving maps, and not walking/transit/cycling maps.
| Obviously you can kinda figure out walking paths anyway, but
| since I'm sometimes travelling without roaming access, it's
| unfortunate.
| giantg2 wrote:
| I image it would be hard do transit maps if you weren't
| connected to get the schedule.
| Scoundreller wrote:
| They already get them somehow while "online". Offline
| with beginning & end times and a rough idea of frequency
| should be good enough for local use.
|
| Offline road maps are subject to
| construction/seasonal/holiday route closures/deviations
| too, and so is transit.
| giantg2 wrote:
| Road closure tends to be much more rare. Transit is much
| more variable in the US.
| Scoundreller wrote:
| > Traffic worsened because drivers weren't able to optimize
| their routes, or even get directions to their destination
|
| During Canada's Rogers outage in 2022:
|
| > In Toronto there was some dependency on Rogers. One quarter
| of all traffic signals relied on their cellular network for
| signal timing changes. The Rogers GSM network was also used to
| remotely monitor fire alarms and sprinklers in municipal
| buildings. Public parking payments and public bike services
| were also unavailable.
|
| https://en.m.wikipedia.org/wiki/2022_Rogers_Communications_o...
|
| As it was summer, I recall some park programming for kids had
| to be cancelled because the employees were required to have a
| phone capable of calling 9-1-1 (but sounds like that at least
| still worked here)
| bostonwalker wrote:
| This is putting it mildly. The Interac network went down and
| no one could use their debit cards nationwide.
| Scoundreller wrote:
| I'd put that on Interac single-homing itself without
| redundancy.
|
| Their ops are critical enough you'd expect better from
| them.
|
| Not the kind of shortcut Canadian banking takes for core
| stuff.
| twisteriffic wrote:
| Not all of it. My credit union's interac services still
| worked.
| sandworm101 wrote:
| >> Traffic worsened because drivers weren't able to optimize
| their routes
|
| I'm not sure that is a thing. The vast majority of drivers are
| on familiar routes and are not navigating via electronic means.
|
| Better question: How are the autonomous cars doing? Are they
| parked by the side of the road unable to navigate without cell
| coverage.
| taurath wrote:
| It only takes a few people missing an exit and swerving to
| create a bunch of traffic. So many people are used to not
| navigating manually anymore I can't imagine it doesn't have a
| big effect.
| sandworm101 wrote:
| I have a 20-mile commute. I used my phone on day one at the
| new job, then never again since. It just isn't worth the
| effort for a road I've driven literally hundreds of times
| before. Do people also use google maps to get them from
| their front door to their garage? From the grocery store to
| wherever they parked their cars?
| jader201 wrote:
| It depends on your daily commute.
|
| If I need to drive 20 minutes with most of it on the
| expressway, and they're prone to accidents and there are
| multiple viable routes, I'm 100% going to load it up on
| Maps every trip, if it will save me being delayed 10-60
| minutes every few weeks.
|
| But if I'm going mostly backroads, probably not worth it,
| since you can more easily go around accidents, and
| they're less common.
|
| But again, I'm guessing more city expressway commuters
| use navigation daily than you think.
| dgacmu wrote:
| I do. Mostly from curiosity about which way Google will
| suggest I go; sometimes because the traffic or road-
| closure awareness is useful. Though it's often the case
| that I know about the road closures before it does -- but
| sometimes it surprises me in a pleasant way.
| jader201 wrote:
| > The vast majority of drivers are on familiar routes and are
| not navigating via electronic means.
|
| I've been rerouted due to an accident many times, and I've
| seen the detours get backed up because of people taking more
| optimal routes (without traffic being redirected via other
| means).
|
| I'd be curious to see more data on it, but I would speculate
| it's less than the "vast majority".
|
| > Better question: How are the autonomous cars doing? Are
| they parked by the side of the road unable to navigate
| without cell coverage.
|
| Yeah, that falls under my point about Uber/Lyft/taxis. I
| would speculate there is broader impact from those vs.
| autonomous cars (that are probably still relatively
| uncommon).
| dheera wrote:
| > What is/were the cascading effects of this, particularly for
| drivers?
|
| I wish for an economic system in which all causes could be
| backpropagated to the source and the source be held
| responsible.
|
| If for example I lost 2 hours of my time today because I had to
| fight with Comcast, Comcast should be charged for 2 hours worth
| of my hourly salary.
|
| If I lost a job offer because of bad interview performance
| because of heating issues because of bad maintainence on part
| of landlord, landlord should be charged for the difference in
| time until I get my next job offer or the difference in salary
| until the next job offer.
|
| If I had to fight health insurance for 5 hours on the phone due
| to incorrect bill and that caused me additional stress that
| caused my condition to worsen, health insurance should be held
| liable for the delta effects of that stress.
|
| In this case the cellular operators in question would be held
| liable for the lost incomes of those drivers plus the lost
| incomes of passengers who lost money because they couldn't get
| to their destinations on time or missed flights and had to
| rebook them.
|
| I know this level of backpropagation is hard to implement in
| the real world but it would be awesome if the entire world were
| one big PyTorch model and liabilities could be calculated by
| evaluating gradients.
| GravityLab wrote:
| We need to maintain paper-based systems of information storage
| and retrieval. People should be familiar with a physical map.
| If we are too dependent on the technology, that is a risk.
| ww520 wrote:
| Does this affect voice only? Or affect both voice and cellular
| data?
| defly wrote:
| JFYI in December largest Ukrainian operator Kyivstar was down for
| days after sophisticated Russian hacker attack
| talldatethrow wrote:
| Did anyone have a phone fail during this outage?
|
| My Pixel 4XL was working at 2am as I placed it on the charger
| (even though it had about 80%). Noticed zero bars. Shrugged and
| went to sleep.
|
| Woke up at 9am and the phone is totally non operational.
|
| Yes I know sometimes phones just break, but less rarely while not
| moving, not wet, not dropped, or due to a charging issue.
|
| What a coincidence.
| tonetegeatinst wrote:
| Wondering if we ever get technical write-ups from these type of
| outages?
|
| Is this only effecting non priority accounts?
| Animats wrote:
| This is a serious architectural flaw.
|
| In the entire history of electromechanical switching in the Bell
| System, no central office was ever out of service for more than
| 30 minutes for any reason other than a natural disaster, or, on
| one single occasion, a major fire in NYC.
|
| The AT&T Long Lines system in the 1960s and 1970s had ten
| regional centers, all independent and heavily interconnected.
| There was a control center, in Bedminster, NJ, but it just
| monitored and sent out routing updates every 15 minutes or so.
| All switches could revert to default routing if needed, which
| meant that some calls would not get through under heavy load.
| Most calls would still work.
| lxe wrote:
| The complexity and scale of moderns systems are on another
| scale of magnitude.
| bluepizza wrote:
| This is no justification. They should have another scale of
| resilience.
| lxe wrote:
| Agreed. Not a justification, but an explanation/excuse as
| to why systems are less reliable. You're right on the mark
| -- when reliability doesn't scale with complexity, you get
| this.
| Scoundreller wrote:
| And centralized. Data is cheap (though they won't admit that)
| while big iron cellular core stuff is expensive.
|
| Funny when they billed extra for long distance calls even
| though all calls were routed through one place for a huge
| geographic area. Calling your neighbour could be a hundreds
| of miles round trip over mobile.
| Animats wrote:
| > And centralized.
|
| Yes. Too much of routing is centralized. Since phone
| numbers are no longer locative (the area code and exchange
| number don't map to physical equipment) all calls require a
| lookup. It's not that big a table by modern standards. Tens
| of gigabytes. All switches should have a database slave of
| each telco's phone number routing list, to allow most local
| calls if external database connectivity is lost. It may be
| behind, and some roaming phones won't work. But most would
| get through.
| Animats wrote:
| Watch "Without Fail" (1967), on how the Bell System did
| it.[1]
|
| [1] https://www.youtube.com/watch?v=ZAJpionUxJ8
| Diederich wrote:
| In this context, what's the physical scale of a 'central
| office', as far as regional dimensions? Thanks!
| ok123456 wrote:
| There was the Mother's Day outage of 1990. That was caused by
| someone swapping a break statement for a continue statement in
| some C code that handled the routing, and there was a cascading
| effect.
|
| Then again. That only affected long-distance service.
| Animats wrote:
| There's an outage map.[1] But it's useless. That's just a US map
| of where most people live.
|
| [1] https://www.cbsnews.com/news/outage-map-att-where-cell-
| phone...
| paxys wrote:
| https://xkcd.com/1138/
| chasd00 wrote:
| my wife's phone wasn't working for half the day but mine was.
| Same provider and same location (i'm assuming being in the same
| house meant we probably were connecting to the same tower).
| Browsing around new sites, i think even solar flares are in the
| realm of possibility.
| joshu wrote:
| Someone I know installed an iOS update at the same time; when the
| phone came back up, it was dead. What a frustrating coincidence?
| w10-1 wrote:
| Likely unrelated: I can report that last week my phone number was
| somehow assigned to two devices for ~3 days.
|
| I use a carrier that leases from a major (Verizon). There were
| clearly disconnects between the systems that even tier-3 support
| expected to automatically resolve overnight (i.e., be eventually
| consistent). Still, after 2 such overnight waiting periods
| failed, I got someone to spec a fix and see it through over the
| course of 3 hours. They were clearly surprised that both their
| system and the major system were not reporting correctly, and the
| solution came only after making changes while ignoring status.
|
| I got a sickening feeling that enterprise software layers mated
| with eventually-consistent web persistence across two outsourcing
| organizations to produce a situation where truly no one
| understood what was happening -- or could even figure it out.
| This resolve only when some youngish person had the guts to just
| make changes that should work -- so we're back to the days of
| heroes.
___________________________________________________________________
(page generated 2024-02-22 23:00 UTC)