[HN Gopher] System separation in the Continental Europe Synchron...
       ___________________________________________________________________
        
       System separation in the Continental Europe Synchronous Area on 8
       January 2021
        
       Author : andreasley
       Score  : 262 points
       Date   : 2021-01-26 09:58 UTC (13 hours ago)
        
 (HTM) web link (www.entsoe.eu)
 (TXT) w3m dump (www.entsoe.eu)
        
       | tyho wrote:
       | How does this affect timekeeping? Normally, if there is a drop in
       | grid frequency, a small increase will be planned later to
       | maintain a long term average frequency of 50Hz so that clocks
       | that keep time by grid frequency stay accurate [0].
       | 
       | The network split seems to have made this impossible, during the
       | split, the cycle counts for the two regions diverged, and the
       | split ended before this was reconciled. Will people in one of the
       | regions have to adjust their clocks?
       | 
       | [0]:
       | https://en.wikipedia.org/wiki/Wide_area_synchronous_grid#Tim...
        
         | i_am_proteus wrote:
         | My suspicion is that NTP (and GPS) have generally superseded
         | the synchronous grid for timekeeping. It has for every
         | application that I know of.
         | 
         | Simultaneously, I suspect some systems still do rely on grid
         | time.
        
           | nottorp wrote:
           | Computers (or anything containing a proper computer) don't
           | care about grid frequency for time keeping.
           | 
           | On the consumer side, your microwave, coffee machine and even
           | bedside clock will.
           | 
           | Not sure on the industrial side.
        
             | read_if_gay_ wrote:
             | > your microwave, coffee machine and even bedside clock
             | will.
             | 
             | I guess a ~0.5% deviation for a couple minutes is going to
             | be tolerable here.
        
               | nottorp wrote:
               | Oh found it. Looks like this went on for more than a
               | couple minutes:
               | 
               | https://www.theguardian.com/world/2018/mar/08/european-
               | clock...
        
               | LeonM wrote:
               | The grid is never exactly 50Hz, it varies quite
               | significantly. On the long run, the grid operators try to
               | average to 50Hz as close as possible, just so that clocks
               | continue to operate somewhat accurate.
               | 
               | In fact, it is even in the FAQ of the article:
               | 
               | "The transmission grids of the countries of Continental
               | Europe are electrically tied together to operate
               | synchronously at the frequency of approximately 50 Hz"
        
               | driverdan wrote:
               | Based on the article the control limits are 49.8 to 50.2
               | Hz (+-0.2hz). That's less than 1%, not very significant.
               | 
               | One of the places I lived in Austin, TX had a range with
               | a clock that used grid frequencies (60hz in the US). It
               | was dead accurate. I only changed it for daylight
               | savings, it never need adjustment.
        
               | klodolph wrote:
               | The US used to guarantee 5,184,000 cycles per day. That's
               | not, like, 1% or anything comparable, that's zero (with
               | temporary deviations). Any clock which referenced line
               | frequency would be dead accurate over the long-term.
               | There's a system called Time Error Correction (TEC) which
               | would do this (refer to WEQ-006 and the like). Obviously
               | this error is measured relative to some reference clock.
               | 
               | I'm not sure what the current status is, but my
               | understanding is that there are efforts to retire this
               | system and allow the speed to drift more.
        
             | jwr wrote:
             | > On the consumer side, your microwave, coffee machine and
             | even bedside clock will
             | 
             | I'd be _very_ surprised. It 's cheaper to build in a
             | 32.768kHz crystal for timekeeping than try to access the
             | grid frequency from the isolated low-voltage circuit.
        
               | garaetjjte wrote:
               | I don't think price is significant factor. It just needs
               | optocoupler, diode and few passives.
        
               | klodolph wrote:
               | You don't need an optocoupler to measure grid frequency.
               | You can just use a resistor. Dirt cheap, cheaper than a
               | crystal.
        
               | garaetjjte wrote:
               | Yes, but OP talked about isolated circuits. It's likely
               | that most appliances don't need to be isolated though.
        
               | sltkr wrote:
               | It is very true.
               | 
               | A similar event happened in Europe a year ago (also due
               | to problems in former Yugoslavia) and all electric clocks
               | in my house (microwave, oven, alarm clocks) went out of
               | sync temporarily. I was just as surprised as you.
               | 
               | Basically, it's exactly the devices that lose track of
               | time in case of a power outage, and that you need to
               | manually adjust for daylight savings time, that are
               | synchronized to the grid. Devices that use a battery
               | (such as laptops, mobile phones, and CMOS clocks in PCs)
               | must necessarily use some other means.
               | 
               | It doesn't seem to be much correlated with price either:
               | my alarm clock cost over $100 and it was still affected.
               | Surely at that price point they could have afforded a
               | crystal oscillator in the design, if they wanted to, but
               | it seems this just isn't typically done.
        
               | laurent92 wrote:
               | As an industry, it requires a special customs declaration
               | + fire safety measures on boats and planes when you
               | transport an oven with a battery inside. That may be the
               | reason for not using battery-backed ovens, and relying on
               | the power supply.
        
               | fuzzer37 wrote:
               | Just curious what kind of alarm clock costs $100
        
               | sltkr wrote:
               | Something like this:
               | https://www.amazon.co.uk/dp/B008LR3KD8/
               | 
               | The idea is that it wakes you up gently by flooding the
               | room with light resembling a natural sunrise. One reason
               | that it might be expensive is that it needs a lot of high
               | luminance hue shifting LEDs, and possibly I overpaid a
               | bit (there do seem to be much cheaper competitors) but it
               | was still a good investment because it does make my
               | mornings a lot more pleasant.
        
               | ficklepickle wrote:
               | One that plays soothing sounds, like dogs barking or
               | seagulls fighting over trash, probably.
               | 
               | It might even project the moon phase on the ceiling.
        
               | iSnow wrote:
               | At a $100 price point, I'd expected a DCF77 receiver tbh,
               | not a crystal oscillator.
        
               | ardy42 wrote:
               | >> On the consumer side, your microwave, coffee machine
               | and even bedside clock will
               | 
               | > I'd be very surprised. It's cheaper to build in a
               | 32.768kHz crystal for timekeeping than try to access the
               | grid frequency from the isolated low-voltage circuit.
               | 
               | IIRC, the grid frequency is usually more accurate.
               | Probably because it's carefully monitored and managed by
               | the power authorities like the OP describes. If you build
               | a clock with its own frequency reference, than any error
               | will accumulate and have to be monitored and managed by
               | the user.
        
               | userbinator wrote:
               | _It 's cheaper to build in a 32.768kHz crystal for
               | timekeeping than try to access the grid frequency from
               | the isolated low-voltage circuit._
               | 
               | Those products don't use isolated power supplies ---
               | normally they use a capacitive dropper. A high-value
               | resistor to an IC pin is sufficient to drive the clock
               | counter, and a resistor is definitely cheaper than a
               | crystal.
        
               | alkonaut wrote:
               | I have a microwave and oven from the same brand, and
               | their digital clocks diverge about 2minutes in 24h. I
               | just don't understand how it's possible to build a device
               | that poorly in 2020 (they are new) and I also wonder
               | which tech they use for timekeeping. Is it possible that
               | _one_ of them uses a crystal and the other uses the 50Hz?
               | That would explain a small difference.
        
               | eqvinox wrote:
               | It's easy enough to mistune a 32768Hz clock crystal by
               | shitty electronics engineering, unfortunately :/
               | 
               | 50Hz might have temporary variations, but it's controlled
               | so that over a longer period of time, you always get the
               | correct number of cycles, e.g. 180000 in an hour.
               | 
               | This also means that going off the 50Hz power grid
               | theoretically has better (or even perfect) long-term
               | accuracy. Also means that if you adjusted your clock for
               | this separation event, you'll have to adjust it back :D
        
               | laurent92 wrote:
               | US current is 60Hz. EU current is 50Hz. Maybe your over
               | maker did some rounding when adjusting by 5/6 and here
               | you go? Btw, does your oven have a 50-60Hz switch?
        
               | thaumasiotes wrote:
               | > I have a microwave and oven from the same brand, and
               | their digital clocks diverge about 2minutes in 24h.
               | 
               | That's much, much worse than a cheap digital watch from
               | 20 years ago.
        
               | andai wrote:
               | Much worse than a digital watch from the _80s._ (Maybe
               | 70s?)
        
               | alkonaut wrote:
               | I know. It's worse than the pendulum clock I have to
               | crank at my grandma's once a week! Progress!
        
               | kevin_thibedeau wrote:
               | It's either a poorly made bottom of the barrel crystal or
               | they cheaped out even further and used a ceramic
               | resonator. That is the price of race to the bottom
               | globalization. Measuring line frequency is a no go
               | because the safety compliance inceases costs.
        
               | nottorp wrote:
               | Maybe, but I bet those designs are very traditional and
               | date from the time where it was cheaper to use grid
               | frequency.
               | 
               | If not, why don't coffee machines that can start on a
               | timer have a backup battery for their clock? That might
               | be cheap enough too. And vital for those moments when
               | there was a 10 second brownout which leads to your coffee
               | not being ready in the morning...
        
               | wongarsu wrote:
               | > why don't coffee machines that can start on a timer
               | have a backup battery for their clock?
               | 
               | As a European the thought wouldn't have come to me.
               | Brownouts just don't happen (see this 0.5% frequency dip
               | making headlines), and a blackout is a once-a-decade
               | event for any given house (probably even less frequent
               | than that).
               | 
               | So your answer is probably that demand for that feature
               | is far from universal, and a coffee machine is much lower
               | stakes than for example an alarm clock.
        
           | labawi wrote:
           | I was under the impression semaphores use grid power as a
           | clock source. And the case of missing payer in southern
           | Balkan got the timing shifted by a few minutes.
        
         | brummm wrote:
         | In Europe, many countries have radio signals that many clocks
         | use to sync their time that is based on some atomic clocks. For
         | example, in Germany these are called Funkuhr and the signal
         | comes from an atomic clock in Braunschweig if I'm not mistaken.
        
           | ficklepickle wrote:
           | We have this in North America as well. The station is called
           | WWVB.
           | 
           | https://www.nist.gov/time-distribution/radio-station-wwvb
        
         | izacus wrote:
         | Clocks on our ovens and some microwaves got out of sync during
         | that period. I haven't seen any other device rely on the
         | network frequency for timekeeping - everything time sensitive
         | is already on NTP and GPS time.
        
           | tzs wrote:
           | OT: Several commenters have mentioned both their oven clocks
           | and microwave clocks getting out of sync. I'm curious as to
           | why people set both of these clocks in the first place.
           | 
           | The microwaves and ovens I've seen do not actually require
           | you to set the clock. None of their cooking features and
           | functionality depends on it. It's just a convenience feature.
           | I doubt that most people actually need two clocks in the
           | kitchen, so why not just set one of them (or neither if you
           | don't need a kitchen clock at all)?
        
             | AdamN wrote:
             | There was this great design for a microwave that a UX
             | designer proposed. One button for +30 seconds, no clock,
             | and a door handle. That's it. Everything else is
             | superfluous.
        
               | ta76893548 wrote:
               | As a person who uses a microwave a lot this sounds
               | terrible, I could live with 30s resolution, but not
               | having adjustable power level would not be acceptable.
               | Cooking anything on 100% power is just a recipe for
               | uneven heating.
        
             | BrandoElFollito wrote:
             | I have in my kitchen + living-room one clock based on NTP
             | on a dashboard, one on a WiFi radio and one on the oven.
             | 
             | Depending on where I am I see one of them so all must be
             | synchronized to the millisecond so that children are not
             | late to school (school is nearby, which exponentially
             | raises the risk of them being late because it is just "4
             | minutes" away).
             | 
             | This is one of the scientific uses of such multiple clocks.
        
             | Denvercoder9 wrote:
             | Lots of ovens and microwaves don't allow you to disable the
             | clock, so the alternative to setting them is to have them
             | run out-of-sync, which is much more irritating than having
             | to set them.
        
               | tzs wrote:
               | Ah...I'd forgotten about the ones that always show the
               | time regardless of whether or not it has been set. I can
               | see how that would get annoying.
               | 
               | I guess I've been lucky in that all the microwaves I've
               | had over the last couple of decades, and the ones we had
               | at work, do not show the time unless you've set it. My
               | current one, which has 7 segment LEDs for the time
               | digits, blanks all the digits, leaving just the colons.
               | My prior one showed "--:--" if time was not set. The one
               | before that had a graphical display and just showed blank
               | where the time would normally go.
        
               | mrmanner wrote:
               | Mine has two alternatives: Clock or blinking red light. I
               | choose the clock...
        
             | nslav wrote:
             | I just got rid of a GE microwave which required me to set
             | both the time and date before heating food after a power
             | outage. I never did figure out what the date was for since
             | it was never displayed and the clock did not automatically
             | adjust for daylight savings time.
             | 
             | On top of that it had a dedicated button for toggling
             | between AM and PM when setting the clock, which served no
             | purpose whatsoever after that point. I wish I knew what the
             | designer was thinking when they came up with that.
        
             | amaccuish wrote:
             | Literally because it annoys me more having it not set
             | properly
        
             | vaduz wrote:
             | I can give you a personal counter-example: my oven requires
             | the time to be set to run - without it, neither the fan nor
             | any of the two heating elements turns on.
             | 
             | It's an old Bosch HBN202S
        
       | alex_duf wrote:
       | I wonder what are the benefits of keeping the grid synchronous?
       | 
       | Would it be possible to have multiple smaller grids, still
       | interconnected, but without being kept in sync?
       | 
       | I'm not sure if what I'm saying is possible or efficient, but
       | converting AC to DC, transmitting the energy, then converting DC
       | to AC so the frequency becomes irrelevant.
        
         | gspr wrote:
         | For one, it's used for timekeeping.
        
           | filmor wrote:
           | That's more of a side-effect, not the goal of a synchronous
           | grid.
        
         | tsar_nikolai wrote:
         | > I wonder what are the benefits of keeping the grid
         | synchronous?
         | 
         | > Wide area synchronous networks improve reliability and permit
         | the pooling of resources. Also, they can level out the load,
         | which reduces the required generating capacity, allow more
         | environmentally-friendly power to be employed; and allow more
         | diverse power generation schemes and permit economies of scale.
         | [0]
         | 
         | [0] https://en.m.wikipedia.org/wiki/Wide_area_synchronous_grid
        
         | calaphos wrote:
         | It is absolutely possible and exactly how connections between
         | the different grids are done, e.g. between the power grids of
         | North America or between the European grid and Russia.
         | 
         | HVDC of course has the disadvantage of extra conversion
         | equipment. HV circuit breakers are also significantly more
         | complex as an arc will form as long as current is flowing. With
         | AC this happens at the zero automatically, nothing like this
         | with DC.
        
           | erk__ wrote:
           | Does not even need to be between countries for example east
           | and west of Denmark are on two different grids which only
           | recently got connected 10 years ago. (Large HVDC under the
           | great belt) I am pretty sure the map on the site is "wrong"
           | as Sjaelland (Zealand) the large eastern island of Denmark is
           | on the nordic grid, and so is Bornholm which nearly
           | exclusivly gets it power from Sweden.
        
             | Symbiote wrote:
             | The map on Wikipedia is better, as it divides Denmark
             | correctly, and also excludes several other large European
             | islands that are not connected to the grid.
             | 
             | https://en.wikipedia.org/wiki/Synchronous_grid_of_Continent
             | a...
        
           | asmithmd1 wrote:
           | You can see on this map the purple DC lines between England
           | and France and Ireland https://www.entsoe.eu/data/map/
           | 
           | I am amazed the grid goes all the way to Iraq
        
         | 7thaccount wrote:
         | Multiple small grids (probably not interconnected) would be
         | like microgrids.
         | 
         | In general, interconnection allows for the sharing of
         | generators. My 30 year old coal unit might not need to run
         | because I can buy electricity from your cheaper gas plant.
        
       | [deleted]
        
       | kuter wrote:
       | There was a incident in Turkey (31 march 2015) that caused a
       | country wide blackout. From what I understand the west side and
       | the east side loose sync. The west side had under supply and the
       | east side had over supply.
        
       | huhnmonster wrote:
       | I was wondering if someone could explain the countermeasures for
       | such an event. Obviously, as the article states, producers are
       | being shut off in the regions with surplus, while drains, who can
       | afford to shut off, are shut off in the deficit regions.
       | 
       | Is this an automatic process? Or is it more like someone from the
       | company's energy provider calls them and tells them to shut off
       | some devices? And is there not a potential problem, that if too
       | many shut of at once, you now have a surplus again? Or is it
       | coordinated by one single entity?
        
         | paulmac_ie wrote:
         | The company I work for provides Demand Response to the Irish
         | grid operator.
         | 
         | In cases like this, our systems would detect the frequency
         | deviation, and shut off loads within 100 milliseonds to reduce
         | the demand on the grid. This helps in cases where demand is
         | greater than supply.
         | 
         | The entire system is automated - the required time frames are
         | so quick that you don't have time for humans to be involved. By
         | the time we're aware that an event has occurred, we've already
         | reduced demand on the grid.
         | 
         | Handling high frequency events where supply is greater than
         | demand is tricker. Sites that have long running generation can
         | be instructed to shut down their generation, but large-scale
         | batteries are probably the best solution in these cases. They
         | can be switched quickly to start charging (if they have spare
         | capacity).
         | 
         | As you've identified, one potential issue is that you can end
         | up over-responding to the event and move from a low frequency
         | event to a high frequency event.
         | 
         | The way we do in in Ireland is that our response is
         | proportional to the frequency nadir. Not everything is tripped
         | off at the same time.
         | 
         | As other posters have noted, the actual frequency deviations
         | that occurred are not that big. 49.7 Hz is not that low
         | compared to normal grid frequency. In fact, some of our systems
         | wouldn't even activate at this level. They would see it, but
         | wouldn't trip off any loads.
        
           | xoob wrote:
           | I'm curious about this. What kinds of load is your company is
           | running? How do you take them offline and online so quickly
           | without affecting production or leading to long restart
           | cycles?
        
         | rtkwe wrote:
         | They talk about it in the article as contracted interruptible
         | services that got shut down in France and Italy to reduce the
         | draw on the grid.
        
         | [deleted]
        
         | bo0tzz wrote:
         | It is an automatic process, coordinated - as far as I'm aware -
         | by two national energy grid providers (Switzerland and Germany)
         | who have been selected for this role.
        
           | fisian wrote:
           | The response is automated and every system operator provides
           | some ressources to react:
           | https://en.wikipedia.org/wiki/Operating_reserve
        
         | Faaak wrote:
         | These are called ancillary services. There are 3 levels:
         | frequency control (for example dams), primary reverse (fast
         | acting power, in less than 10s), and secondary reserve (slow
         | acting: < 15min).
         | 
         | Participants can either be positive (they consume more energy,
         | for example PHES systems), or negative (they inject or consume
         | less energy).
         | 
         | All these ancillary services are paid (annual auction + per
         | case). Nowadays, it's getting bigger with VPP: virtual power
         | plants, which aggregate small loads (i.e. small ~1MW
         | generators) in order to propose a bigger load to TSOs.
         | 
         | It's all automatic
         | 
         | If you're interested:
         | https://www.swissgrid.ch/en/home/customers/ancillary-service...
        
           | Faaak wrote:
           | On top of that, newer solar inverters MUST (at least in the
           | EU) reduce power if frequency rises in order to automatically
           | shed power in cases of extreme supply
        
         | Reason077 wrote:
         | "Shutting off drains" is known as _demand side response_ or
         | DSR. It 's often cheaper, faster, and more environmentally
         | friendly to pay industrial customers to cut their demand by x
         | MW than to fire up x MW of ancillary generation. And as far as
         | grid frequency/balance is concerned, has exactly the same
         | effect.
         | 
         | Often such customers will have flexible demand in the form of
         | non-critical heating or cooling, pumps that only need to run
         | some of the time, etc, which they are very happy to turn off
         | temporarily in return for extra income.
         | 
         | This is indeed an automatic process, triggered in near-real
         | time in response to signals from the grid.
        
         | jnsaff2 wrote:
         | The grid frequency is a measure of energy balance in the grid.
         | So anyone can measure anywhere in the frequency domain whether
         | there is a surplus or deficit of generation. This is due to the
         | fact that the grid itself can't store any energy, it has to be
         | balanced all the time.
         | 
         | Imagine the massive spinning generators as a big mass that slow
         | down just a little bit when you switch on a light and then that
         | generator has to add more steam (or open hydro valve or
         | whatever).
         | 
         | So anyone with an accurate enough measuring device can exactly
         | monitor the state of the grid. We use this device [0] for
         | example.
         | 
         | There are generally frequency containment reserves (FCR) that
         | consist of different ways of generation and have their
         | different reaction times, power and energy capacities.
         | 
         | Hydro for example can react in about 15 seconds, battery
         | inverters in milliseconds. Gas turbines in minutes, coal fired
         | plants in hours.
         | 
         | You can also shed energy by switching off loads (Demand side
         | response).
         | 
         | The system operator is responsible for grid balancing in the
         | short term, they have direct facilities under their control and
         | they have contracts with generators and consumers. And there
         | are markets to bid your generation and flexibility.
         | 
         | The markets in the Nordics for example are:
         | 
         | - FCR-N (Frequency containment Reserve - Normal operations)
         | 
         | - between 49.90 - 49.99 and 50.01 - 50.10 (reaction time up to
         | 20s)
         | 
         | - FCR-D (Disturbance) - between 49.7-49.90 and 50.10 - 50.30
         | (reaction time up to 2 seconds IIRC)
         | 
         | - FFR (Fast Frequency response) - below 49.7 - reaction time
         | 0.6s IIRC
         | 
         | Once a day you bid your capacity for the next 24h (for each
         | hour) and then you measure the grid frequency yourself and when
         | you detect a deviation you activate your response. You get paid
         | for availability and activation separately. There is a ton of
         | qualification and logging you need to do to be able to
         | participate but the activation message is the grid frequency
         | itself, no further communication needed.
         | 
         | Outside frequency regulation there is energy markets where
         | generation and consumption is agreed 24h ahead.
         | 
         | [0] https://www.gobmaier.de/
        
           | bennofs wrote:
           | > Imagine the massive spinning generators as a big mass that
           | slow down just a little bit when you switch on a light and
           | then that generator has to add more steam (or open hydro
           | valve or whatever).
           | 
           | I have always wondered, would that still hold true if the
           | grid was fully solar-based? There would be no rotating mass
           | in that case.
        
             | jnsaff2 wrote:
             | Yes indeed, inverter based generators (solar and batteries)
             | don't have any inertia. But as long as the grid is still AC
             | the frequency will be the same and the inverters would need
             | to compensate internally. Especially with solar where a
             | cloud going over a solar farm can easily knock off a few MW
             | in seconds the volatility of the grid will increase and
             | need for storage (batteries, hydro, etc) alongside Demand
             | Side Response is getting much bigger.
        
             | netflixandkill wrote:
             | Yes, although a significant issue with that is there is no
             | spinning mass to "borrow" inertia from, so while modern
             | inverters at solar sites are good at shaping output phases,
             | they have very little capability to absorb significant
             | frequency deviations.
             | 
             | Grid scale battery systems are often used for voltage or
             | frequency stability as opposed to deep discharging as
             | generation offsets, although that will change eventually if
             | batteries get better enough or really cheap LNG stops being
             | a thing.
        
             | brandmeyer wrote:
             | It does hold true for some systems. Wind turbine generator
             | controls can provide virtual inertia over very short time
             | scales (a second or few) by exchanging energy with the
             | turbine rotor. You can also provide primary frequency
             | reserve in the negative direction (load step-off) using
             | exactly the same ramp control that steam plants use.
             | Typical ramp rate is 100% of rated power for a 5% change in
             | frequency.
             | 
             | However, in order to provide primary frequency reserve in
             | the other direction, you do need additional local storage.
             | You don't need very much. Just 10% of rated power for 15
             | minutes gets you to very deep renewable penetration.
             | 
             | The trouble isn't with the technology, its with the
             | economics. Once you set a sufficiently high price for
             | frequency support and primary frequency reserve, suppliers
             | will show up.
        
       | liversage wrote:
       | Something similar happened in 2018. However, it was not a
       | technical problem but a conflict between the power grid operators
       | in Serbia and Kosovo.
       | 
       | https://www.dw.com/en/clocks-in-europe-are-running-late-beca...
       | 
       | I believe that if you adjusted your clock during the conflict you
       | had to adjust it again when it was resolved as the resolution was
       | to increase the frequency for a period to reverse the loss of
       | frequency.
        
       | GekkePrutser wrote:
       | Aha so this is what caused that big frequency drop people had
       | already reported. Interesting!
        
       | cpach wrote:
       | Relevant Wikipedia article:
       | https://en.wikipedia.org/wiki/Synchronous_grid_of_Continenta...
       | 
       | Must admit I've never heard of this before :-)
        
         | m463 wrote:
         | Funny how the folks on this site will still wade into an
         | article even with a completely unrecognizable (yet curious)
         | title like this one.
        
       | brohee wrote:
       | Infosec slant: such investigations are made way easier by precise
       | timestamping (everything happens in a few milliseconds), but the
       | source of truth (GPS or other GNSS) is usually pretty easily
       | spoofable if you only intend to move the time a few milliseconds.
       | Galileo has a project addressing this
       | (https://ec.europa.eu/transparency/regexpert/?do=groupDetail....)
       | but AFAIK it's not in the signal yet. And once it is, it will
       | take years for the devices doing the satellite time to PTP to be
       | replaced/updated.
        
         | traceroute66 wrote:
         | Surely this is only true to a minimal extent in world where you
         | can already buy timeservers that support multiple independent
         | time sources out of the box (multiple satellite Galileo,GPS
         | etc. + terrestrial radio e.g. DCF77).
         | 
         | IIRC if you're dead serious and money is no object, all the
         | major national time labs also allow you to run a private leased
         | line to their facility for direct checks.
        
           | brohee wrote:
           | As of now all GNSS constellation are somewhat spoofable,
           | especially if you only want to drift a clock a few
           | milliseconds late... And a lot of timeservers are still GPS
           | only so the attack is pretty easy to perform.
           | 
           | The private line idea is maybe doable (but which protocol?
           | NTP doesn't offer the needed precision), but as all
           | substations would need one, my guess is that eventually money
           | would be an object...
        
             | michaelt wrote:
             | _> (but which protocol? NTP doesn 't offer the needed
             | precision)_
             | 
             | You might enjoy reading the documentation for NPLTime [1]
             | which uses PTP over telecom fibre networks.
             | 
             | Of course, their target market is banks with record keeping
             | requirements so it probably costs a fortune.
             | 
             | [1] https://www.npl.co.uk/npltime
        
       | jasonjayr wrote:
       | Previously, from the 8th:
       | https://news.ycombinator.com/item?id=25685646
        
       | contravariant wrote:
       | Seems like an excellent example that preventing a single point of
       | failure means nothing if your system doesn't have the extra
       | capacity to handle a single failure.
        
         | nabla9 wrote:
         | There was multiple failures, not single.
         | 
         | The grid was separated but it didn't cause more power outages.
        
       | londons_explore wrote:
       | If you design an electricity grid to never partition, then you
       | need to keep a substantial headroom of transmission equipment
       | unused to prevent cascading failures leading to a partition in
       | case of failure of just one or two sites.
       | 
       | Instead, an electricity grid should be able to survive any
       | partition like this while still keeping all frequencies within
       | the nominal range. The total cost of such a system is lower in
       | most cases (you need more idling generation capacity or droppable
       | load, but less transmission capacity)
       | 
       | The EU system here failed to do either here.
        
         | frede wrote:
         | The system survived the partitions, there was no widespread
         | blackout. Further, the system is not organized by the EU, but
         | the ENTSO-E and contains e.g. countries from North Afrika.
        
           | cbmuser wrote:
           | There was no blackout _yet_, but it was very close.
           | 
           | According to the German Bundesnetzagentur (the equivalent of
           | the FTC/EIA), the number of times where they have to
           | intervene with the grid due to grid instability is constantly
           | rising due to Germany shutting down nuclear and coal plants.
           | 
           | > https://www.bundesnetzagentur.de/DE/Sachgebiete/Elektrizita
           | e...
        
             | imtringued wrote:
             | According to this chart there has been an insignificant
             | reduction in installed capacity.
             | 
             | https://energy-
             | charts.info/charts/installed_power/chart.htm?...
             | 
             | Nuclear power went from 20GW to 10GW. Hard coal went from
             | 28GW to 22GW.
             | 
             | Ok, but gas went up from 23GW to 29GW. Brown coal stayed
             | the same.
             | 
             | Renewables went up by by 50GW for PV and 50GW for wind.
             | Consider that wind often hits a 50% capacity factor. That
             | is 25GW in additional power just from wind alone.
             | 
             | Some of those plants may not be running continuously but
             | they are still useful for emergency responses.
        
             | Dumbdo wrote:
             | > There was no blackout _yet_, but it was very close.
             | 
             | Where do you get that from? None of the sources reported a
             | close blackout, as far as I understood it there was a lot
             | of emergency capacity left. We weren't even in the
             | emergency frequency range, as the other commenter pointed
             | out.
             | 
             | Even the linked article just states that those
             | interventions got more often after shutting down
             | coal+nuclear, but it's not critical, it _only_ costs money
             | to compensate the operators:
             | https://de.wikipedia.org/wiki/Redispatch_(Stromnetz)
             | 
             | It's probably much less money than all the nuclear
             | subsidies.
        
             | phicoh wrote:
             | It is a choice, spend a fortune on nuclear power that
             | nobody wants in their own backyard, or deal with a less
             | stable grid due to solar and wind energy.
        
               | qayxc wrote:
               | Sigh. It's not a dichotomy at all.
               | 
               | It's perfectly possible to have a stable grid even with
               | solar and wind.
               | 
               | The problem is that neither nuclear nor coal power
               | stations are load following and "base load" has lowered
               | over the years (industry has moved to Asia, devices
               | became more efficient, etc.).
               | 
               | Stand-by power (like natural gas powered generators)
               | hasn't been built up the way the way it should have been.
               | Same goes for smart grid technologies and buffer storage;
               | not to mention the maniacs (especially in South Germany)
               | who basically protest _everything_ - from nuclear power,
               | to wind power, to required infrastructure like north-to-
               | south high-voltage transmission lines.
               | 
               | It's way too oversimplified to reduce the issue to just
               | wind and solar.
        
         | jabiko wrote:
         | I'm not sure this counts as a failure.
         | 
         | The load shedding mechanisms worked, supportive power
         | generation was automatically activated and there was no
         | widespread blackout.
         | 
         | You are right that it appears like there was insufficient
         | capacity at the north-west/south-east separation point and I
         | guess that is going to be investigated but apart from that
         | everything looks like it worked.
        
         | zaarn wrote:
         | The frequencies were in nominal range. The lowest was around
         | 49.7, the highest about 50.6, both of which are within the
         | allowed range of frequency (47.5 - 52.5 is the absolute
         | emergency range, 49-51 is the nominal range, before you start
         | capping power to or from customers).
         | 
         | Romania had a power blackout for a bit, but everything worked
         | as intended and the romanian grid blacking out didn't pull in
         | the rest of the european grid.
         | 
         | A partition between power grid is acceptable to preserve the
         | largest amount of the grid remaining functional. Marrying the
         | partitions together is bothersome but not something that takes
         | forever (took 1 hour in this case).
         | 
         | While 49.7 is very low, most of your devices will not notice
         | and will be fine. Most industrial equipment is largely already
         | coded to handle frequency shifts in favor of keeping the grid
         | stable.
        
       | reitanqild wrote:
       | What puzzles me is why everyone talks about _clocks_..!
       | 
       | Clock synchronisation is a nice side effect, but what amazes me
       | is how it is possible to keep this thing swinging somewhat
       | synchronously year after year, and how huge chunks of it doesn't
       | crash and burn when events like this happens.
       | 
       | (I have worked as a consultant for a power supplier, I was more
       | puzzled afterwards : )
        
         | littlestymaar wrote:
         | > what amazes me is how it is possible to keep this thing
         | swinging somewhat synchronously year after year
         | 
         | There's nothing to do to keep them in sync: every single[1]
         | alternator connected to the electric grid is rotating at the
         | same speed, which is driven by the frequency of the
         | electricity. This is a physical effect.
         | 
         | The big challenge isn't to keep things in sync, it's to keep
         | the supply equals to the demand. (when we don't, the frequency
         | goes up or down depending on which is higher than the other)
         | 
         | [1]: in thermal stations only, wind turbines are not directly
         | connected to the grid because their rotation speed depends on
         | the wind...
        
           | reitanqild wrote:
           | > There's nothing to do to keep them in sync: every single[1]
           | alternator connected to the electric grid is rotating at the
           | same speed, which is driven by the frequency of the
           | electricity. This is a physical effect.
           | 
           | You either know a lot less than me or you know more and
           | understand it better. I honestly can't tell : )
           | 
           | It feeks like my brain collapses just from starting to think
           | about thousands of kilometers of grid, thousands of power
           | stations, millions of consumers etc. (Not that the amount of
           | them matters, just the supply or demand they deliver/drive.)
           | 
           | In the interest of learning, here is my model that I map the
           | things I do understand to:
           | 
           | The way I (as a hardware/software engineer who started in
           | electronics and had a brief course on power electrics) reason
           | about it is torque, as if they are kind of sharing a common
           | shaft and applying torque to it while consumers are braking
           | it.
           | 
           | This simplified mental model would explain - kind of - how it
           | can stay in sync, only in reality we are talking not a linear
           | axle or shaft but this continent-wide grid where gigawatts of
           | supply and demand can occur within minutes or even less and
           | the phase differs by quite a lot over the span of the
           | continent. (In my mental model this is the axle twisting.)
           | 
           | Also at 50Hz the wavelength of light in vacuum should be
           | around 6000km if I typed correctly and DDG understood
           | correctly. A rule of thumb we learned (in high frequency
           | electronics back in electronics engineering) was that once
           | you cross a tenth of that the normal rules doesn't quite
           | apply. If this can be applied to power grids (an I think it
           | can) it becomes even more complex I guess. (The
           | simplification we can apply in "small" circuits is that we
           | can pretend evey point in the circuit is at the same point of
           | the phase at the same time.)
           | 
           | Keep in mind everyone: these are just my models. I'll be
           | delighted if littlesymaar (or someone else) knows this
           | extremely well and manages to enlighten me because it would
           | be fun to really "get" it.
           | 
           | Edits: A lot.
           | 
           | Also: Just operating one power plant can be complex: I
           | remember one presentation from former students or something
           | about how important it was in Eastern Europe back then to be
           | ready to cut immediately if neighbouring plants failed or cut
           | so that your plant wouldn't suddenly oversupply and burn out.
        
             | nine_k wrote:
             | I think the idea of a common shaft, only provided by
             | electricity, is a good approximation.
             | 
             | But the shaft of such length is not very "rigid", and the
             | task is to keep it from wringing and breaking by keeping
             | all parts of it rotating at the same speed.
             | 
             | (As a side note: have you heard of selsyns?)
        
             | brandmeyer wrote:
             | Something that might help: It isn't one axle twisting. Any
             | one source or sink on the grid can model themselves as a
             | single shaft coupled to the rest of the grid, but for
             | conceptualizing it as a whole you really do have to think
             | of it as a system of coupled machines.
             | 
             | One thing that might help with the stability intuition: The
             | generators themselves are synchronous machines, but they
             | have parasitic induction machines deliberately installed in
             | the form of damper bars. Those damper bars mean that the
             | vibrating mode between the generator rotor and the stator
             | field is well-damped. Similarly, the vast majority of total
             | load is in the form of induction motors which naturally
             | have a damped response between the motor rotor and stator
             | field. So resonating patterns and shocks are quite
             | difficult to set up. Even sharp step inputs are attenuated
             | to be not-sharp over short distances.
             | 
             | In the US, one common test for grid step response is
             | dropping an entire nuclear power plant. Not one reactor:
             | the entire site. So several GW of electric supply is
             | dropped instantaneously. The limit typically isn't any kind
             | of oscillation, its the ability of primary frequency
             | reserve to pick up the slack before you start triggering
             | under-frequency trips.
        
         | LargoLasskhyfv wrote:
         | Tell us _more_ about that :)
        
           | reitanqild wrote:
           | I cannot really but I have put my lack of knowledge on public
           | display here for someone else to help me and you :
           | https://news.ycombinator.com/item?id=25917037
           | 
           | : )
        
             | LargoLasskhyfv wrote:
             | Thanks. I've already read that ;) But let me say this.
             | About 10 to 15 years ago I've been more interested in
             | transmission lines because of "Energiewende" and such, came
             | across HVDC, how that would enable smart grids, possibly
             | protection against solar/geomagnetic storms because of
             | faster switching times, and so on. I looked into how they
             | built those HVDC-lines in China, how that is not new at
             | all, but eased by modern solid-state power semiconductors,
             | and so on. And looked further into patents, modulation
             | schemes, even many 'papers'(pdf) and also very technical
             | books(skimmed those only).
             | 
             | Anyway, what I learned was that the grid has _many_ very
             | nonlinear phenomena which are extremely hard to model and
             | predict, because of the dynamics. Much more complicated
             | than the simplistic answers some here have given you.
             | So...relax?
             | 
             | :-)
        
       | LargoLasskhyfv wrote:
       | You know what I'm missing? The almost realtime map from ucte.org
       | which updated every 3 to 5 seconds on my Pentium3 933 with only
       | 512MB Ram in the browser.
       | 
       | I think it needed some plugin, either Java or Flash, but you
       | could zoom in or out, either to the whole of Europe, or down to
       | the substations in the cities and towns.
       | 
       | It was like google maps or similar. AND it had _all_ the
       | transmission lines, power plants, and substations.
       | 
       | And color coded lines, with KW/MW/GW and arrows indicating the
       | direction of energy flow on them.
       | 
       | Anybody could view it without having an account there. At least I
       | did, from time to time.
       | 
       | This way of viewing it in realtime is now gone, or at least not
       | accessible to the general public anymore.
       | 
       | Does anybody know some equivalent?
       | 
       | Sites with bargraphs and charts need not apply.
        
         | Reason077 wrote:
         | ENTSO-E is the successor organisation to UCTE [1], with its
         | functions transferred to ENTSO-E in 2009. So if anyone has the
         | data, it would be them. Interesting that the ucte.org website
         | still seems to be online and frozen in time!
         | 
         | [1] https://www.entsoe.eu/news-events/former-
         | associations/#union...
        
           | andreasley wrote:
           | While ENTSO-E does have a map [1], it's not as nice as the
           | one described above.
           | 
           | [1] https://www.entsoe.eu/data/map/
        
             | LargoLasskhyfv wrote:
             | Yes, does not compare, unfortunately.
        
               | noobermin wrote:
               | I hate to commit red-herring but this is another example
               | of how computing is worse today in some ways.
        
               | [deleted]
        
           | LargoLasskhyfv wrote:
           | Yeah, I noticed that some years ago. They don't have that
           | offer anymore. Interestingly, looking at the article, the
           | cutouts are in the exact style of that former realtime map.
           | 
           | But static. Not living. That was once there, and available to
           | the general public. Personally, I view that as artificial
           | scarcity of information, regarding energy politics.
           | 'Smokescreening' so to speak.
        
             | EE84M3i wrote:
             | Is it possible it's no longer available for security
             | reasons?
        
               | LargoLasskhyfv wrote:
               | I don't know. I've thought about that. OTOH if some
               | terrorist would like to know the choke points of the
               | grid, he could deduce that from openstreetmaps also,
               | where the transmission lines and substations are
               | available for viewing.
        
               | intrepidhero wrote:
               | The physical layout of the transmission system is
               | basically impossible to obscure, but does not necessarily
               | give the attacker insight into power flows. Having access
               | to real time power flow data would enable an attacker to
               | design a cascading failure scenario, making an attack
               | MUCH more effective. The last component would be knowing
               | operational response procedures. Which seem to be fairly
               | transparent in Europe? If so I'm surprised more care
               | isn't taken to secure them.
        
               | WanderPanda wrote:
               | It is probably right, but I have to say that I hate that
               | kind of security through obscurity
        
               | LargoLasskhyfv wrote:
               | Especially when there is this https://de.wikipedia.org/wi
               | ki/Stromausfall_in_Europa_im_Nove... which was caused by
               | switching this off, https://de.wikipedia.org/wiki/380-kV-
               | Ems-Freileitungskreuzun... which is rather similar to
               | https://de.wikipedia.org/wiki/Elbekreuzung_2 :-)
               | 
               | Even if there are surveillance cameras on those masts,
               | there aren't on the masts leading to it. So you could
               | ride your bicycle there, have some fun with thermite, and
               | finally enjoy the stars again!1!!
               | 
               | /me giggles (j/k)
        
               | javajosh wrote:
               | Yes, and honestly someone determined to do damage like
               | that would be the sort to be patient and methodical
               | enough to find targets with or without the map. But I do
               | understand the instinct to want to at least increase the
               | barrier to destruction, the problem is that it ALSO
               | increases the barrier to understanding (and so protection
               | and enhancement). So, an understandable but bad trade
               | off, IMHO.
        
               | LargoLasskhyfv wrote:
               | I didn't see it from that point of view. It was just
               | interesting to watch for me, out of the corner of my
               | eyes, while doing other stuff on other systems and
               | screens. I did that somewhere between 2005 and 2008,
               | whenever I felt like it :-) With a focus on Germany in
               | general, and my town especially. So I had it mostly
               | zoomed just so that Germany was in view, with small parts
               | of neighbouring countries visible. We didn't have that
               | much solar and wind at the times. But sometimes one could
               | see the direction of flow switch. From/to France for
               | instance, or Denmark. And I think I've seen that to/from
               | Poland and Czech also.
               | 
               | Anyway, you could get a feeling for how things were
               | depending on time of year/day, weather, and so on.
               | Intuitevely, because presented spatially and realtime.
               | 
               | And you don't get that spatial intuitiveness from the
               | substitutes with (often delayed) charts & graphs.
               | 
               | Which makes it impossible to counter bullshit by
               | politicians or other parties with vested interests when
               | they are saying this and that, and you could have said:
               | _Ahem, that is not entirely correct, because there and
               | then it was like so, and not what you say!_
               | 
               | And the loss of that ability is making me angry!
               | 
               | edit: Regarding the 'intuitiveness' of spatial vs. bars &
               | charts. Even when the possibility to filter for im/export
               | between countries exists, it does not show over which
               | lines it went. Which could be useful in discussions about
               | extending/upgrading lines, or building new ones. It's
               | just not there.
        
         | layoutIfNeeded wrote:
         | Someday someone will reimplement it in React, and you will only
         | need an i9-9900k with 64Gb to run it somewhat smoothly.
        
           | LargoLasskhyfv wrote:
           | Where would someone get the realtime data? Technically it
           | _has to_ exist, otherwise frequency regulation wouldn 't
           | work. The fact that they don't show it in the way I
           | described, shows that they don't want to anymore, because
           | they don't need to because of different organizational
           | structure, or the structure has been made that way for
           | ideologicel reasons by politics, i.e. 'smokescreening'.
        
         | rkangel wrote:
         | Is it possible that not making that level of data available is
         | purposeful from a security point of view?
         | 
         | Western European countries (among others) worry about threats
         | to their "Critical National Infrastructure", whether in person
         | (terrorist with a bomb) or as a cyber attack. Providing a big
         | map with all transmission lines, power plants and substations
         | seems like it would be a big help to an unsophisticated
         | terrorist, showing them what to blow up.
        
           | LargoLasskhyfv wrote:
           | I don't think so, because you can see that on openstreetmap.
           | I mean the transmission lines and substations.
        
         | Havoc wrote:
         | Like this:
         | 
         | https://www.electricitymap.org/map
        
           | LargoLasskhyfv wrote:
           | While that is looking nice, (and I don't know how 'realtime'
           | it is) it lacks the structure of the transmission nets, and
           | the zoomability down to substation level in the towns/cities.
           | I can't remember exactly anymore, it didn't show every
           | transformer in town, but the places where the high voltage
           | overland transmission lines entered into town, and were
           | transformed down to more managable levels were there. I think
           | down to 10.000/50.000 Volts. That was different regionally.
           | For instance Portugal vs. Germany.
        
         | theandrewbailey wrote:
         | That's a lost art nowadays, doing all that on a Pentium 3.
         | Today, you'd need about 512 GB of RAM and a supercomputer to do
         | the same.
         | 
         | \s
        
           | LargoLasskhyfv wrote:
           | Hrrm. I did that under some Linux and KDE 3.x, maybe even
           | with the KDE Browser called Konqeror. That was usable at the
           | times :-) 2005 to 2008 that must have been.
        
         | pmayrgundter wrote:
         | Did it look like this?
         | 
         | https://www.researchgate.net/figure/Energy-trade-change-in-t...
         | 
         | Seems like they're referring to it but also the comments
         | suggest the data was from simulation of optimal market
         | exchange, not actual electrical flows
        
           | LargoLasskhyfv wrote:
           | Nope. That is abstract. It looked exactly like the 2 blurred
           | cutouts from the Entsoe article, where they put in the
           | probable causes textmarker style. Interestingly it is hard to
           | find screenshots of good quality or at all. Seems to have
           | been a very special interest of mine?
        
       | [deleted]
        
       | AnssiH wrote:
       | AFAIK the continental synchronous area frequency is long-term-
       | synchronized to atomic time (i.e. the cumulative grid time must
       | stay within X seconds of the reference clock).
       | 
       | Looking at the (partial) graph suggests that the south-east part
       | ran faster for the entire duration of the disconnection.
       | 
       | Does this mean grid-synced clocks in the south-east part are now
       | permanently ahead of the atomic reference, or are they planning a
       | mitigation (which I assume would have to mean disconnecting again
       | and running the SE part slower for a bit)?
        
         | gspr wrote:
         | > Does this mean grid-synced clocks in the south-east part are
         | now permanently ahead of the atomic reference, or are they
         | planning a mitigation (which I assume would have to mean
         | disconnecting again and running the SE part slower for a bit)?
         | 
         | That's the standard procedure. Maintaining an exact frequency
         | can be hard, but _counting cycles_ is easy. So the frequency
         | will slew to slowly compensate back.
         | 
         | Edit: Here's a Tom Scott video on a UK-Continental divergence
         | that happened a while back:
         | https://www.youtube.com/watch?v=bij-JjzCa7o
        
           | garaetjjte wrote:
           | UK is connected through DC lines, so frequency is
           | independent. In this case they would have to separate the
           | grid again to be able to correct this.
        
       ___________________________________________________________________
       (page generated 2021-01-26 23:01 UTC)