[HN Gopher] 2025 Iberia Blackout Report [pdf]
___________________________________________________________________
2025 Iberia Blackout Report [pdf]
Author : leymed
Score : 91 points
Date : 2025-06-23 18:34 UTC (4 hours ago)
(HTM) web link (media.licdn.com)
(TXT) w3m dump (media.licdn.com)
| fuoqi wrote:
| >Given the timing of this drop in solar generation, in the middle
| of the production ramp-up and without any meteorological
| phenomena to explain it, the most plausible explanation is that
| it is due to market reasons (prices)
|
| >These changes in production can be significant (if the price
| signals from the markets are sufficiently strong) and affect the
| energy flows in the networks and the stresses in the nodes
|
| >Regarding the correlation between changes in generation and
| voltage: if the generation operating at power factor falls (with
| the existing regulation, the renewable generation), there is a
| decrease in the reactive energy absorbed by these installations
| (since it is reduced proportionally to the reduction in
| generation). Additionally, as the energy transmitted by the
| networks decreases due to this reduction in generation, the
| capacitive effects of the electrical circuits increase as they
| become more discharged, which causes an increase in reactive
| energy. Both effects (higher reactive power production by the
| circuits and lower reactive power absorption) push the voltages
| upwards.
|
| As expected, renewables played the leading role in
| destabilization of the grid which led to the collapse. Of course,
| these instabilities could be worked around on the grid level
| using various (not cheap) measures, so you may argue that it's
| not renewables' "fault", but the fact still stands: above the
| certain threshold of renewable generation the current grid
| architecture becomes increasingly unstable.
|
| I wrote exactly about it in this comment
| https://news.ycombinator.com/item?id=43831523 and its child
| comment, but the fans of renewables just flagged it.
| matsemann wrote:
| But the quote literally spells out it was market forces, not
| some instability in solar generation?
|
| Your other comment probably got flagged because it started with
| a huge straw man and had multiple unwarranted jabs in it.
| fuoqi wrote:
| Temporary negative prices have been caused by the renewable
| generation which exceeded the grid demand at the time, which
| then evolved into the nasty feedback loop caused by the
| reaction of renewable generation to those conditions. You
| simply do not get such situation with traditional generation,
| it's the direct consequence of the intermittent nature of
| renewables and its high ratio in the total generation.
|
| Also, have you read after the market part? Please watch this
| video https://www.youtube.com/watch?v=7G4ipM2qjfw if the last
| quote is gibberish to you. It discusses somewhat different
| issues, but the point still stands.
| pkilgore wrote:
| Where is the market for someone to get paid to pump water
| into a reservoir and let it fall down later for $$$?
| pdpi wrote:
| That market exists, but the window of time here is like
| twenty minutes. Pumps have inertia and take time to spin
| up, you can't HFT load and generation.
| shakow wrote:
| True, but the market moves fast because renewables (or, more
| precisely, wind & solar) move fast.
|
| There is not much fast trading to be done on a
| nuke/gas/coal/hydro powerplant ramping up or down, but there
| is a lot of instability (and thus market volatility) to be
| found in fast varying solar/wind conditions.
| wavefunction wrote:
| You quoted
|
| >the most plausible explanation is that it is due to market
| reasons (prices)
|
| Seems to be market conditions or manipulations or
| inefficiencies in the market.
| pdpi wrote:
| What I'm reading from that quote is that the issue wasn't
| renewables as such, but an issue of power generation reacting
| too quickly and too intensely to price fluctuations.
| "Renewables" only matter insofar as they're the sort of
| generation that, under the current regulatory regime, get to
| react to those pricing changes.
| baq wrote:
| The report goes to great lengths to avoid certain words or
| phrases. The market failed here, it didn't price in risk of
| grid collapse correctly.
| Yeask wrote:
| That is a simple and great explanation.
| fuoqi wrote:
| See the sibling comment:
| https://news.ycombinator.com/item?id=44360082
| baq wrote:
| Should've said 'not enough spinning mass' and it'd be perfectly
| fine for the politically correct and mean the same thing. This
| was highlighted as a risk for years and it finally
| materialized.
| philipkglass wrote:
| According to the operator report linked in another comment by
| leymed [1], the problem was not a lack of spinning mass
| (inertia) but voltage instability. From page 16 of the PDF:
|
| _The incident was NOT caused by a lack of system inertia.
| Rather, it was triggered by a voltage issue and the cascading
| disconnection of renewable generation plants, as previously
| indicated. Higher inertia would have only resulted in a
| slightly slower frequency decline. However, due to the
| massive generation loss caused by voltage instability, the
| system would still have been unrecoverable._
|
| [1] https://news.ycombinator.com/item?id=44360052
| baq wrote:
| Obviously I'm as good of a grid operator as I was a stealth
| bomber expert on the weekend, but superficially that just
| doesn't seem right. Maybe I'm underestimating how much
| spinning mass would be required, but that still qualifies
| as 'not enough was present'.
| tofflos wrote:
| It's a difficult read.
|
| Cybersecurity and digital systems was not the issue but gets
| thirteen pages of proposed measures. I feel this could have been
| left out.
|
| Electric System Operation was the issue and gets seven pages of
| proposed measures.
| leymed wrote:
| Check this shorter report by the operator:
|
| https://d1n1o4zeyfu21r.cloudfront.net/WEB_Incident_%2028A_Sp...
| AnotherGoodName wrote:
| Reads very similar to some blackouts we had in Australia. Weakly
| connected grids with vast geographical distances leading to
| oscillations that took down the grid.
|
| https://en.wikipedia.org/wiki/2016_South_Australian_blackout
|
| Completely solved with lithium based grid storage at key
| locations btw. This grid storage has also been massively
| profitable for it's owners
| https://en.wikipedia.org/wiki/Hornsdale_Power_Reserve#Revenu...
|
| Australia currently has 4 of the 5 largest battery storage
| systems under construction as a result of this profit
| opportunity;
| https://en.wikipedia.org/wiki/Battery_energy_storage_system#...
|
| You can also read numerous stories of how Australia's lithium ion
| grid storage systems have prevented blackouts in many cases.
| https://www.teslarati.com/tesla-big-battery-south-australia-...
| The fact is that the batteries responsiveness is the fastest of
| any system at correcting gaps like this. 50/60hz is nothing for a
| lithium ion battery nor are brief periods of multi-gigawatt
| draw/dumping as needed.
|
| There's even articles that if Europe investing in battery storage
| systems like Australia they'd have avoided this.
| https://reneweconomy.com.au/no-batteries-no-flexibility-spai...
| londons_explore wrote:
| > nor are brief periods of multi-gigawatt draw/dumping as
| needed.
|
| Actually this is typically an issue for grid batteries.
|
| Spinning generators can easily briefly go to 10x the rated
| current for a second or so to smooth out big anomalies.
|
| Stationary batteries inverters can't do 10x current spikes ever
| - the max they can get to is more like 1.2x for a few seconds.
|
| That means you end up needing a _lot_ of batteries to provide
| the same spinning reserve as one regular power station.
| AnotherGoodName wrote:
| Collectively Australia's battery storage systems will be able
| to beat any single power plant for peak output in Australia
| once fully built out based on pure numbers. But for these
| sorts of grid oscillations the more important thing is the
| localization of generation. Which obviously favours the
| batteries over large centralized power stations in any case.
| giantg2 wrote:
| Seems like pumped hydro offers a nice compromise.
| AnotherGoodName wrote:
| Fwiw the hornsdale battery linked above cost AUD$172
| million and can provide 2000MW of equivalent inertia. Link:
| https://www.energymagazine.com.au/sa-approves-world-first-
| ba...
|
| That equivalent inertia can only be done for short periods
| but that's exactly what grids need in stability - there's
| generally no lack of total generation, just a need to jump
| in and smooth out spikes.
|
| You can't build a dam for that price, nor could you do it
| in under 100 days from contract signing as that battery was
| built. Batteries are definitely the answer here. The 'more
| spinning mass' answers don't make sense since Australia
| literally solved the above problem in a much cheaper way
| already.
| giantg2 wrote:
| You don't need to build a dam, you just need the pipes
| and pumps for an existing dam (or elevated natural
| basin).
| ajross wrote:
| That... doesn't sound correct. Inverters are the cheap part,
| you can literally wire as many as you want in parallel.
| Batteries have _immense_ power availability, with most
| chemistries you can trivially deliver the entire capacity in
| half an hour or so (more like 5 minutes with lithium cells).
|
| Basically I'm dubious. I'm sure there are grids somewhere
| that have misprovisioned their inverter capacity, but I don't
| buy that battery facilities are inherently unable to buffer
| spikes. Is there a cite I can read?
| AnotherGoodName wrote:
| Agreed. The relatively small battery substation linked
| above can output 2GW of equivalent inertia generation (a
| measure to align batteries to inertial power systems) when
| needed. That's an entire power station they can match for
| short periods of time. Link:
| https://www.energymagazine.com.au/sa-approves-world-first-
| ba...
|
| Australia's largest power plant has 2.9GW of inertial
| generation assuming all generators are running at 100%. As
| in the small battery substation alone comes close to the
| countries largest power station. I'm not sure where the
| idea that lithium ion can't dump power quickly comes from.
| They are absolutely phenomenal at it. Australia's building
| dozens of these substations too since they are so cheap and
| reduce overall power costs. It's a win from all points of
| view.
| probablypower wrote:
| You can google "system inertia" as a starting point.
| rcarmo wrote:
| Page 130 is where the actual human readable summary is. Although
| the previous pages were pretty detailed in explaining the
| cumulative instabilities.
|
| Sadly, some news outlets are probably only going to look at the
| recommendations and read "cybersecurity" and (even though they
| are common sense recommendations) assume there might be more to
| say about the matter.
| decimalenough wrote:
| Don't worry, some news outlets will summarize this as
| "renewables = bad" regardless of what the report actually says.
|
| Oh wait, they already did:
| https://www.telegraph.co.uk/business/2025/06/18/renewable-en...
|
| Ed: Do I need a /s tag here or something? My point was that we
| shouldn't worry too much about about the presentation of the
| report, its actual contents will be spun to suit any narrative
| regardless.
| baq wrote:
| Flame bait journalism is one of the things you can count on
| in any circumstance. If you replace that 'renewables = bad'
| with 'politicians = idiots' OTOH... sometimes the elected
| representatives should listen to unelected physicists and
| engineers. Grid stuff is one of those things.
| decimalenough wrote:
| > _Non-confidential version of the report of the committee for
| the analysis of the circumstances surrounding the electricity
| crisis of the April 28, 2025_
|
| Now I'm curious about what's in the confidential version of the
| report.
| londons_explore wrote:
| Based on the redacted bits, it is mostly company names and
| locations.
| londons_explore wrote:
| It doesn't look like this report really identifies the root
| causes...
|
| I would like to see: "We have simulated the complete 200 and 400
| kV grid of the iberian peninsula and western europe, and can
| reproduce the situation that occurred. Any one of the following
| changes would have prevented the issue, and we suggest
| implementing them all for redundancy. This simulation will be re-
| run every day from now on to identify future cases similar
| incidents could occur"
| baq wrote:
| The engineers knew exactly what's going to happen, the report
| is politically redacted. Very unfortunate, but completely
| expected.
| JanneVee wrote:
| When skimming through the report I got to think of the
| oscillation problem in RIP routing protocol. Although it isn't
| the same thing, but it shows the complexity of the problem to
| anyone who thinks there is a single solution to it.
| gred wrote:
| Why so many pages of "Recommendation: implement multi-factor
| authentication" and other IT security irrelevancies? Did they
| need to pad out the number of pages?
| diggan wrote:
| > In the systems with network traffic evaluation probes, no
| records consistent with unauthorized activity have been
| observed, such as lateral movements, network traces or file
| movements for vulnerability exploitation or privilege
| escalation, among others.
|
| > However, as is common in networks and information systems in
| any sector, other risks have been identified, such as
| vulnerabilities, deficiencies or inadequate configurations of
| security measures, which may expose networks and systems to
| potential risks, for which a series of measures are proposed.
| diggan wrote:
| Seems the money shot starts at page 131:
|
| > The ultimate cause of the peninsular electrical zero on April
| 28th was a phenomenon of overvoltages in the form of a "chain
| reaction" in which high voltages cause generation disconnections,
| which in turn causes new increases in voltage and thus new
| disconnections, and so on.
|
| > 1. The system showed insufficient dynamic voltage control
| capabilities sufficient to maintain stable voltage
|
| > 2. A series of rhythmic oscillations significantly conditioned
| the system, modifying its configuration and increasing the
| difficulties for voltage stabilization.
|
| If I understand it correctly (and like software, typical), it was
| a positive feedback-loop. Since there wasn't enough voltage
| control, some other station had to be added but got overloaded
| instead, also turning off, and then on to the next station.
|
| Late addition: It was very helpful for me to read through the
| "ANNEX X. BRIEF BASICS OF THE ELECTRIC SYSTEM" (page 168) before
| trying to read the report itself, as it explains a lot of things
| that the rest of the report (rightly) assumes you already know.
| leymed wrote:
| I think your interpretation is correct. The voltage control is
| done at the high level of the grid, meaning the control covers
| bigger generation stations and major substations. Even if it's
| small generator, rotating machinery, you won't have strict
| voltage control other than its own AVR. The problem I see here
| is that we embed smaller individual generations at the lower
| level, where they pump the generated power to the grid at the
| medium voltage level. When you have majority of your generation
| at this level, you won't have strict control over voltage and
| even frequency, I assume. I'm still digesting the report, but
| what I am after is whether they really neglected it and if it
| is not possible to do voltage control with 50% generation
| coming from renewable and through medium voltage level, aka
| lower level.
___________________________________________________________________
(page generated 2025-06-23 23:00 UTC)