[HN Gopher] Hetzner subsea fibre cut outage
___________________________________________________________________
Hetzner subsea fibre cut outage
Author : cain
Score : 125 points
Date : 2022-05-24 16:39 UTC (6 hours ago)
(HTM) web link (status.hetzner.com)
(TXT) w3m dump (status.hetzner.com)
| jacooper wrote:
| Aha! That's why my status server has been acting weird saying
| things were offline when they weren't.
| [deleted]
| snsr wrote:
| Best guess this is the cable based on Hetzner investment and
| landings:
|
| https://www.submarinecablemap.com/submarine-cable/c-lion1
| iruoy wrote:
| That seems to be correct.
|
| https://www.hetzner.com/presse-berichte/2016/01/156818
|
| > Hetzner Online GmbH, the first major Central European
| provider of data center services to construct a new data center
| park in Finland, is also a user and co-investor in the Cinia
| CLion1 cable connection.
| bombcar wrote:
| I was today years old when I learned Hetzner has its own undersea
| cables.
| daneel_w wrote:
| It's not their cable.
| ArchOversight wrote:
| They are a co-investor... so its partially their cable.
| daneel_w wrote:
| And as I'm an EU citizen paying top taxes, plenty of which
| went into funding the project, it's partially my cable as
| well.
| OJFord wrote:
| By that logic (that it's not (partially) Hetzner's I
| mean, not the sarcasm) it's nobody's cable then?
| devoutsalsa wrote:
| Fix your damn cable @daniel_w!!!
| Trias11 wrote:
| Lol,
|
| To be specific - you invested in EU taxes, while others
| invested in cable.
| hericium wrote:
| > Recovery of the service is in progress (Last update: 2022-05-24
| 05:49 UTC+0)
|
| > There is currently an ongoing fibercut at our subsea cable. Our
| cable provider is working on fixing this. There is currently no
| ETA. (Last update: 2022-05-23 14:09 UTC+0)
| davidhariri wrote:
| If, like me, you didn't know how these get cut I'll save you the
| trouble. The most likely cause is improper boat anchoring. Never
| shark bites.
| gmcquistin wrote:
| Sharks do bite these cables!
|
| https://amp.theguardian.com/technology/2014/aug/14/google-un...
| [deleted]
| tiffanyh wrote:
| Dumb question: I thought TCP/IP was designed to route around cut
| fibre cuts.
| Arnavion wrote:
| Also to be clear, this is handled by BGP, not by TCP/IP. BGP is
| what ASs use to identify who connects to what so that they can
| route IP appropriately.
| pvg wrote:
| That's pretty much what happens but no amount of routing can
| make up for the loss of capacity when you physically take out a
| part of the network. You end up with less total available
| bandwidth and more latency.
| caymanjim wrote:
| It is, and a cut like this rarely causes a complete outage, but
| it depends on whether and what kind of backup connectivity your
| ISP has, and the connectivity of your destination. Bigger
| entities tend to have more backups. Smaller entities tend to
| have fewer backups, and often the backups are much lower
| bandwidth/quality, so it's typical for service to degrade.
| AtNightWeCode wrote:
| Don't know about Hetzner but cloud traffic these days are not
| routed over the public Internet in that way.
| pchristensen wrote:
| If any part of this interests you, you must read "Mother Earth,
| Motherboard" by Neal Stephenson. It's epic longform from the
| early Wired.
|
| https://www.wired.com/1996/12/ffglass/
| carlsborg wrote:
| Then you will also like this: https://www.submarinecablemap.com
| sbf501 wrote:
| Is it one big horkin' cable that needs to be spliced by divers?
| Or is it hundreds of tiny little fibers?
| jaywalk wrote:
| It's one big cable with eight individual fibers. But they
| aren't fused by divers, the cable will be brought to the
| surface and fused onboard the repair ship.
| daneel_w wrote:
| This isn't Hetzner's own cable, and it's not just "any" cable -
| it's an Alcatel-Lucent + Cinia + EU project, with 150 terabits
| worth of bandwidth. It's a high profile target, but it's just as
| likely that it's just an anchor accident.
| ArchOversight wrote:
| As Hetzner is a co-investor in the cable, it is partially their
| cable.
| R0b0t1 wrote:
| So do they own a solid chunk of it somewhere, or like a
| really long pie-slice sliver down the whole thing?
| jaywalk wrote:
| It's probably split up by wavelengths.
| R0b0t1 wrote:
| So if they only have specific wavelengths, how do they
| keep them in the pie slice of cable they own? I just
| realized they likely don't own a cylindrical section of
| it somewhere because the light has to go in and out of
| it, only thing that makes sense is the long pie slice.
| jaywalk wrote:
| There's no way to own a physical "slice" of fiber optic
| cable. The wavelengths all travel together, and there are
| optical splitters/combiners at each end that
| separate/combine the wavelengths of light. Think of them
| like TV and radio broadcasts. Each station broadcasts on
| it's own frequency (wavelength) over the shared air
| (fiber).
|
| A metaphor with light would be if you had two lights, one
| red and one blue. If you shine them both at the same
| spot, you'll see purple. If you put a prism in that spot,
| it'll split the red and blue light back out separately.
| At a very high level, that's what happens with fiber
| optics.
| CaliforniaKarl wrote:
| The light doesn't really work that way.
|
| u/Denvercoder9 mentioned that there are 8 strands of
| fiber in this particular cable. Each strand can host
| multiple wavelengths of light, that do not really
| interfere with each other. There are different ways of
| dividing up the usable frequency range; here's a site
| with some examples: http://www.3coptics.com/News/11.html
| (Remember, light has a frequency & a wavelength. Human
| vision covers wavelengths from around 400 nm (nanometers,
| the UV end) to 700 nm (the IR end)).
|
| The beams of laser light sent down a fiber strand have a
| wavelength (a central wavelength), and a width. The beam
| essentially needs exclusive ownership of wavelength +-
| width. The diagrams on the linked page show how channels
| are separated to give each channel a set width.
|
| So, when an entity owns a part of a cable, they own a set
| of wavelengths, likely one or more contiguous blocks of
| wavelengths, that they can use for whatever. All owners
| will pay (probably based on the % of wavelengths they
| control) for a company to operate the cable (running the
| stations at each end, where the wavelengths are split out
| on to separate fibers for each owner). The owners will
| also pay for another company to do repairs as needed.
| k__ wrote:
| Since it's a cable, I would imagine the latter.
| Denvercoder9 wrote:
| Neither, it's shared ownership. It's similar to if you buy
| a house with your partner: neither of you owns specific
| rooms, but you own the whole thing together. There's likely
| agreements about which fibers (there's 8 fiber pairs in the
| cable) and/or how much bandwidth everyone can use, though.
| foepys wrote:
| Why only 8? Why don't they put like 100 in there?
| tiernano wrote:
| IIRC, it's mostly to do with power... every Xkm (100?)
| they need to put a power amp and have to splice 100 pairs
| at each point...
| willcipriano wrote:
| They didn't need 500+ terabits?
| bogomipz wrote:
| Well, cost for one. If that fiber isn't being used it's a
| terrible investment. But more generally it's probably not
| needed. The bandwidth of optical fibre is theoretically
| unlimited. A single fibre carries many wavelengths, a
| wave length is like a channel. This is known as
| DWDM(dense wave division multiplexing.) A single
| wavelength commonly caries 100 Gbs. Although recently
| advances have shown 700 Gbs for a single wavelength[1].
| The number of wavelengths and hence bandwidth depends on
| the transmission gear used on both ends of the cable. You
| can upgrade the bandwidth by upgrading the transmission
| gear. The state of the art for years has been 8 pair
| systems but more recently developments have produced 24
| and 36 pair systems.
|
| [1] https://www.lightwaveonline.com/network-design/high-
| speed-ne...
| cure wrote:
| Presumably because the cable would be (much) heavier and
| thicker.
|
| There is some movement towards having more pairs in newer
| cables, e.g. the Grace Hopper will have 16 pairs, cf. htt
| ps://cloud.google.com/blog/products/infrastructure/announ
| ...
| bogomipz wrote:
| Most submarine cables are consortiums. A very recent
| consortium for example is SEA-ME-WE 6:
|
| https://www.submarinenetworks.com/en/systems/asia-europe-
| afr...
| fosefx wrote:
| Is there sth like a status page for submarine cables?
| daneel_w wrote:
| Possibly, but none that I'm aware of.
| [deleted]
| lizardactivist wrote:
| The use of that cable is just to reduce latency. There are many
| other paths between their datacenters, so there won't be any real
| breakage.
| travisporter wrote:
| Can they somehow tell exactly where the cable was cut?
| mortenlarsen wrote:
| This video explains and show the same concept for copper cables
| in a very simple and hands on way: "Cheap and simple TDR using
| an oscilloscope and 74AC14 Schmitt Trigger Inverter"
| https://www.youtube.com/watch?v=9cP6w2odGUc
| jaywalk wrote:
| Some NICs and managed switches can do it as well. Well, I
| should say that _most_ can do it (as in the hardware is
| capable) but only some actually expose the functionality.
| _trampeltier wrote:
| This very old video does it all explain better than all
| todays videos. The models are just so simple, no animations.
|
| https://m.youtube.com/watch?v=DovunOxlY1k
| jlgaddis wrote:
| Yes, using an OTDR [0].
|
| These used to be big, expensive, standalone devices (I'd
| occasionally use one in a previous job). Nowadays, there are
| portable, handheld units specifically for identifying where a
| break has occurred and the same, basic functionality is often
| even built into the gear used at either end of connections
| (e.g., even the "enterprise" switches -- from Brocade and Cisco
| -- that I use at home).
|
| ---
|
| [0]: https://en.wikipedia.org/wiki/Optical_time-
| domain_reflectome...
| polishdude20 wrote:
| Yeah there's special tech for that! Has something to do with
| sending some frequency over the line and measuring how long it
| takes to "reflect" off of the breakage and back to the shore.
| dsr_ wrote:
| The term is time domain reflectometry, TDR. It used to be
| really expensive, and now it's filtered down into medium-cost
| optical adapters.
| bostik wrote:
| In the non-optical domain, that used to be a feature in the
| most expensive cable measuring gears (to the tune of EUR25k
| about 20 years ago).
|
| I had the pleasure of using one of the devices that had
| that when I wired my old house. Borrowed the kit from ...
| an institution that allowed me to get their gear for a
| weekend. (Yes, I knew people in there.) Having the device
| tell me that a given strand in a CAT5e cable was faulty at
| <this many meters> from my location made it surprisingly
| easy to detect where I'd messed up.
| colanderman wrote:
| Many (low-cost) Mikrotik routers have this feature built
| in now: https://wiki.mikrotik.com/wiki/Manual:Interface/E
| thernet#Det...
| dboreham wrote:
| I was going to say that this was invented by none other
| than Heaviside, but it turns out his technique relied on
| static resistance measurement to find the cable fault.
| Similar concept though. And in 1861!
| lordnacho wrote:
| Happens quite often to subsea cables. I was renting a number of
| them and every now and again you'd get a break. You can follow
| the NOC's update emails:
|
| Break detected
|
| Ship has been commissioned
|
| Ship is at harbor
|
| Ship has sailed out to the cut
|
| Fusing the ends
|
| Service restored
| AlecSchueler wrote:
| What's the kind of scenario where you'd be renting an undersea
| cable?
| lordnacho wrote:
| Servers in different financial exchanges that need to talk to
| each other, reasonably fast.
| dabeeeenster wrote:
| How the hell do you fuse the ends of a massive cable like this
| at the bottom of the sea?!
| jaywalk wrote:
| You pull it up to the surface and fuse it on the boat.
| conqueso wrote:
| Find it, pick it up, patch it on deck, lower it back down.
| lordnacho wrote:
| You find the ends, pick them up, and there's a kit that can
| fuse the ends to your new piece. I'm not sure about the
| details, but I think there are repeaters on the wire that
| need to be added as well.
| Scoundreller wrote:
| Do they just fuse any strand to any strand and resort
| everything at the end or?
| toast0 wrote:
| For submarine cables, there aren't that many strands, so
| getting it right is probably worth it, for terrestrial
| fiber, there may be 100+ strands, so they usually can
| sort things out at the ends, and I think you just fuse
| any strand to any strand. Sometimes, only some strands
| break and you can reallocate strands by priority until
| the cable can be fixed.
| 0x0000000 wrote:
| Strands in a bundle would likely have color coded
| sheaths. As long as you're not colorblind, it's pretty
| straightforward matching what gets spliced where.
| dataflow wrote:
| A repeater on the ocean floor? Do they run power cables to
| it alongside the data all the way from the coast?
| detaro wrote:
| Yes, commonly there are high-voltage cables in there.
|
| Nowadays, for shorter runs (that are still long enough to
| need repeaters) there are also purely optical repeaters
| where instead separate fiber strands deliver light from a
| pump laser.
| piptastic wrote:
| This has pretty good information on it, and a couple videos:
|
| https://www.onesteppower.com/post/subsea-cable-repair
| hypertele-Xii wrote:
| > to fix a deep water cable, the ship has to use a grapnel,
| which grabs and cuts the cable, dragging the two loose ends
| to the surface. If needed, one end can then be hooked to a
| buoy and the other end brought on board. Cable has to be
| added to make the repair, since there is not enough slack
| to bring the cable up and cut a piece out. After the cable
| is retrieved and on board, in a repair room that looks like
| a laboratory, engineers repair the cable. Data cables can
| take up to 16 hours to repair, after which they are lowered
| back down to the sea bed in an omega or hairpin pattern to
| accommodate the extra length. There is new technology in
| development that would make in situ repair possible for
| power cables, preventing the need to pull the cables up to
| the surface.
| NickRandom wrote:
| (Second YT search hit so there might be better ones )
| https://www.youtube.com/watch?v=FtEw19JX-UE
| evan_ wrote:
| What's the typical timeline for that cycle? Hours, days, weeks?
| Just curious. I'm sure it costs an astronomical amount of
| money.
| lordnacho wrote:
| Days, but I had one time where the ships were booked up and
| it took longer.
| caymanjim wrote:
| When I lived in Cayman, the main fiber link (Maya-1[0]) was down
| for about a week (anchor drag, I think). Most of the island
| Caribbean has limited backup connectivity. The whole island only
| has one fiber line (around the whole western/southern Gulf of
| Mexico) and a crappy backup copper line to Jamaica. For a while,
| it was like being on a modem again. Forget about streaming
| anything.
|
| This outage Hetzner is reporting isn't a big deal due to robust
| backups, but these things can cripple a small island outpost.
|
| 0: https://www.submarinecablemap.com/submarine-cable/maya-1
| user3939382 wrote:
| > For a while, it was like being on a modem again
|
| Internet in Congo if you're not on the UN satellite
| sizzzzlerz wrote:
| When something like this happens, does the traffic get
| distributed on other operational fibers that have sufficient
| unused bandwidth or do the operators fire up some dark fibers to
| carry the load? Or maybe it's both?
| yuliyp wrote:
| It depends. Generally the larger carriers and consumers of
| fibers have a network that is able to route around outages
| (fiber cuts like this are basically routine) just the same as
| on land. If they don't have enough, then they'll need to either
| reduce demand or rent some capacity from someone else.
| solarkraft wrote:
| > then they'll need to either reduce demand
|
| Worst case everything just gets a bit slower (right?),
| depending on how much traffic went through that cable.
| tinco wrote:
| Bitten by a shark, or snagged on a russian submarine? Whichever
| the case, I just ran an ansible deployment over our cluster
| that's spread over their Frankfurt and Helsinki datacenters, and
| I didn't notice any difference. I guess that's to be expected,
| their line between those datacenters is just an optimization, the
| internet is still functioning as it always is and maybe Hetzner
| gets a slightly larger egress bill from their ISP as the inter-dc
| traffic gets rerouted the general internet.
| nabla9 wrote:
| There are not many sharks that can bite cables in the Baltic
| see due to low salinity, it's border-line freshwater.
| HL33tibCe7 wrote:
| > Bitten by a shark, or snagged on a russian submarine?
|
| In all likelihood much less interesting, probably snagged in a
| fishing net or hit by an anchor.
| barefeg wrote:
| I imagined they would be hurried under the sea bed. Probably
| cheaper to place them on the surface?
| McBeige wrote:
| I imagined it'd too impractical to bury them in case they
| need repairs, but apparently they're buried at least in
| some cases.
|
| According to [1], a plough is dragged along the ocean floor
| behind the cable laying ship. The plough digs a trench that
| the cables are fed into. I can't really tell from the
| pictures whether the trench is covered back up.
|
| [2] shows a really cool animation. Apparently ploughs like
| these can bury cables 2-3 meters into the ocean floor, and
| use a jet of water to help dig. Good stuff.
|
| 1: https://www.quora.com/How-are-major-undersea-cables-
| laid-in-... 2: https://www.youtube.com/watch?v=9hEDTRU_F2s
| HatchedLake721 wrote:
| Can't rule out Russian military dolphins these days.
|
| https://www.theguardian.com/world/2022/apr/27/russia-
| black-s...
| dtx1 wrote:
| > Global Pandemic? Check > Transnational Giga Companies
| controlling the world? Check > Military Dolphins? Check >
| Smuggling Gigabytes of Data via Implants? Not yet.
|
| I rate the current situation as a strong 3/4 Johnny
| Mnemonics!
| konart wrote:
| >Russian
|
| >these days
|
| https://en.wikipedia.org/wiki/Military_marine_mammal
| Mo3 wrote:
| Military dolphins...
| fghorow wrote:
| "With frickin laser beams!"
| user3939382 wrote:
| The models of these cables that I've seen shows them in a
| jacket of steel so I doubt it's a shark. Or that was a joke
| that went over my head...
___________________________________________________________________
(page generated 2022-05-24 23:02 UTC)