[HN Gopher] The gigantic and unregulated power plants in the cloud
___________________________________________________________________
The gigantic and unregulated power plants in the cloud
Author : ahubert
Score : 303 points
Date : 2024-08-19 15:45 UTC (7 hours ago)
(HTM) web link (berthub.eu)
(TXT) w3m dump (berthub.eu)
| samstave wrote:
| I posted this question to HN 7 months ago, more around
| DataCenters:
|
| > _In the increasingly interconnected global economy, the
| reliance on Cloud Services raises questions about the national
| security implications of data centers. As these critical economic
| infrastructure sites, often strategically located underground,
| underwater, or in remote-cold locales, play a pivotal role,
| considerations arise regarding the role of military forces in
| safeguarding their security. While physical security measures and
| location obscurity provide some protection, the integration of AI
| into various aspects of daily life and the pervasive influence of
| cloud-based technologies on devices, as evident in CES GPT-
| enabled products, further accentuates the importance of these
| infrastructure sites._
|
| > _Notably, instances such as the seizure of a college thesis
| mapping communication lines in the U.S. underscore the
| sensitivity of disclosing key communications infrastructure._
|
| > _Companies like AWS, running data centers for the Department of
| Defense (DoD) and Intelligence Community (IC), demonstrate close
| collaboration between private entities and defense agencies. The
| question remains: are major cloud service providers actively
| involved in a national security strategy to protect the private
| internet infrastructure that underpins the global economy, or
| does the responsibility solely rest with individual companies?_
|
| ---
|
| And then I posted this, based on an HNers post about mapping out
| Nuclear Power Plants:
|
| https://news.ycombinator.com/item?id=41189056
|
| _[We can easily map the infrastructure of the cloud and AI --
| and their supply chains - and these are increasingly of National
| Security Concern:]_
|
| ((Not to mention the actual powerplants being built to
| exclusively provide datacenter power))
|
| Now, if we add the layers of the SubmarinCableMap [0]
| DataCenterMap [1] - and we begin to track shipments
|
| And
|
| https://i.imgur.com/zO0yz6J.png -- Left is nuke, top = cables,
| bottom = datacenters. I went to ImportYeti to look into the
| NVIDIA shipments: https://i.imgur.com/k9018EC.png
|
| And you look at the suppliers that are coming from Taiwan, such
| as the water-coolers and power cables to sus out where they may
| be shipping to, https://i.imgur.com/B5iWFQ1.png -- but instead,
| it would be better to find shipping lables for datacenters that
| are receiving containers from Taiwain, and the same suppliers as
| NVIDIA for things such as power cables. While the free data is
| out of date on ImportYeti - it gives a good supply line idea for
| NVIDIA... with the goal to find out which datacenters that are
| getting such shipments, you can begin to measure the footprint of
| AI as it grows, and which nuke plants they are likely powered
| from.
|
| Then, looking into whatever reporting one may access for the
| consumption/util of the nuke's capacity in various regions, we
| can estimate the power footprint of growing Global Compute.
|
| DataCenterNews and all sorts of datasets are available - and now
| the ability to create this crawler/tracker is likely full
| implementable
|
| https://i.imgur.com/gsM75dz.png https://i.imgur.com/a7nGGKh.png
|
| [0] https://www.submarinecablemap.com/
|
| [1] https://www.datacentermap.com/
| dataflow wrote:
| > It's also possible that the manufacturer gets hacked, and
| subsequently sends out attacker controlled and wrong software
| updates to the inverters, with possibly dire consequences.
|
| > There are also people that claim that the many Chinese
| companies managing our power panels for us might intentionally
| want to harm us. Who knows.
|
| Wait, seriously? The European power system relies on Chinese
| companies not messing it up remotely? And the debate is over
| whether the companies will stay nice? For heaven's sake, isn't it
| obvious that during a war the Chinese government can force them
| to just destroy the continent's power system remotely? How is
| this not seen as a extreme continental security risk?
| formerly_proven wrote:
| Same continent that bought energy for decades from its
| strategic enemy. Coincidence? Probably not. Boundless naivete
| and corruption? Also yes.
| ragebol wrote:
| Russia wasn't an enemy for a while. The belief was that
| engaging with them would ensure they wouldn't be an enemy
| again. That failed.
|
| Germany was an enemy once as well
| amelius wrote:
| They already can by simply turning open some power mosfets in
| their fleet of EVs.
| bilbo0s wrote:
| Yeah.
|
| I'm not sure everyone is really thinking clearly here.
|
| Don't get me wrong, they should get rid of this practice of
| cloud monitoring. A consumer should be able to access
| monitoring over the internet without an intermediary. They
| should, of course, be allowed to contract with an
| intermediary if that is their desire.
|
| But the security argument?
|
| Yeah, that ship has sailed. Total war, means total war. Your
| power grid, your internet, your communications, and your
| fossil fuel deliveries will all see material disruption. I
| wouldn't count on being able to stop those disruptions by
| banning a few web sites. (And frankly, during total war,
| those disruptions would be the least of your problems in any
| case.)
|
| Best bet for places like Europe, China, the US and Russia is,
| just don't do total war with each other. If you choose to do
| it anyway, then you can see what you can expect from that in
| the documents filed under "Play stupid games, win stupid
| prizes."
| crazygringo wrote:
| You're turning war into a black-and-white "total war"
| situation. Total war is rare, and no -- no ships have
| sailed.
|
| It's easy to imagine a scenario where something happens
| between China and Taiwan, Europe gets involved in a way
| that majorly pisses off China, and China decides to
| sabotage Europe's grid in response.
|
| Nothing about that is "total war" with Europe, and it's not
| like Europe is going to escalate with nukes either because
| that would be wildly disproportionate.
|
| But it's a major vulnerability that should be fixed as
| quickly as possible. It's negligent for that to even be an
| option for China, because it certainly doesn't seem like
| Europe can do anything similar to the grid in China.
|
| Your idea that security vulnerabilities don't matter, that
| "that ship has sailed", is false and irresponsible.
| bilbo0s wrote:
| You've totally missed the point.
|
| No one advocated ignoring the vulnerability. I, myself,
| specifically stated that monitoring should be direct.
| Consumers should unilaterally decide where, when and how
| their assets are monitored.
|
| The material point on security is that there are many,
| many methods of disrupting a power grid. Even when you
| are looking for plausible deniability, shutting down
| solar panels from cloud website doesn't make a list of
| your top 10 options. (In fact, it won't make the list in
| those scenarios precisely _because_ you are looking for
| plausible deniability.)
|
| Let's imagine a power grid as modern societies know them
| today, except all consumers monitor their solar panels
| themselves, and none of those consumers outsource this
| monitoring function to any third party foreign or
| domestic. Power grids can still be materially disrupted
| in this scenario. Especially in the case of total war.
| Obviously in the case of open war. And particularly in
| the case of cold war.
|
| As I said, I advocate consumers disconnecting any power
| generation functions from networks. But if I'm in the
| seat coming up with post conflict, or even simply
| emergency recovery, operating assumptions, I'm not
| counting on those panels generating power. It's just
| irresponsible to do so. In total war EMP will knock most
| of that generation off line where you're luck enough not
| to have it eliminated entirely. In cold or open war,
| disruptions to distribution can and will render that
| generation useless. (Just ask Ukraine.)
|
| Consumer cloud, or even personal, monitoring of solar
| panels does not enhance, nor does it degrade, your
| adversary's ability to disrupt your power grid when your
| adversary is at that super power level. If you believe it
| does, you're either not looking at the full spectrum of
| what you're calling "vulnerabilities" extant in the
| infrastructure of modern societies. Or you're
| underestimating the full spectrum of capabilities of
| modern military powers. Both, frankly, are fatal mistakes
| in the types of crises we're postulating.
| crazygringo wrote:
| No, your point was clearly stated:
|
| > _But the security argument? Yeah, that ship has sailed.
| Total war, means total war._
|
| Those are your words.
|
| I'm saying, focusing on total war is irresponsible and
| leads you to draw false conclusions. In the real world,
| limited conflicts are what we're dealing with 99.9+% of
| the time, thank goodness.
|
| And now in your new comment, for some reason you're
| focusing on "plausible deniability" which is another red
| herring. If China wants to disrupt Europe's grid, it
| doesn't care about plausible deniability -- the entire
| point is to publicly retaliatiate. It just needs to do
| it, as easily as possible. The idea that relying on a
| cloud vulnerability "doesn't make a list of your top 10
| options" doesn't make any sense at all. It might very
| well be the #1 option, or one of three tactics employed
| simultaneously.
| lifestyleguru wrote:
| Another security issue are all these cheap always connected IP
| cameras from China. Meantime the most recent achievement of EU
| lawmakers is cap permanently attached to a bottle. No wonder,
| as at least in case of my country we are sending the most
| corrupted sleazy individuals to the EU parliament and
| commission.
| afh1 wrote:
| Shutting off nuclear to rely on gas from Russia was not seen as
| an extreme continental security risk. This is nothing...
| shagie wrote:
| > It's also possible that the manufacturer gets hacked, and
| subsequently sends out attacker controlled and wrong software
| updates to the inverters, with possibly dire consequences.
|
| Idaho National Lab is one of those places that researches this.
| https://inl.gov - their domains are energy (primarily nuclear
| and integrated) and national security ... and securing the grid
| is the intersection of that.
|
| And some time back... https://www.wired.com/story/how-30-lines-
| of-code-blew-up-27-... (
| https://web.archive.org/web/20201101002448/https://www.wired...
| ) . The story is from 2020. The event is from 2007.
|
| The test footage linked in the article is on YouTube -
| https://youtu.be/LM8kLaJ2NDU
|
| The wikipedia article on the test:
| https://en.wikipedia.org/wiki/Aurora_Generator_Test
|
| From the wired article the key part of how it broke:
|
| > A protective relay attached to that generator was designed to
| prevent it from connecting to the rest of the power system
| without first syncing to that exact rhythm: 60 hertz. But
| Assante's hacker in Idaho Falls had just reprogrammed that
| safeguard device, flipping its logic on its head.
|
| > At 11:33 am and 23 seconds, the protective relay observed
| that the generator was perfectly synced. But then its corrupted
| brain did the opposite of what it was meant to do: It opened a
| circuit breaker to disconnect the machine.
|
| > When the generator was detached from the larger circuit of
| Idaho National Laboratory's electrical grid and relieved of the
| burden of sharing its energy with that vast system, it
| instantly began to accelerate, spinning faster, like a pack of
| horses that had been let loose from its carriage. As soon as
| the protective relay observed that the generator's rotation had
| sped up to be fully out of sync with the rest of the grid, its
| maliciously flipped logic immediately reconnected it to the
| grid's machinery.
| yetihehe wrote:
| >The owner of the panels and inverters can meanwhile establish a
| connection with that manufacturer using an app or website, and
| _via the manufacturer_ see how their own panels are doing
|
| > It wasn't necessary from a technical standpoint to let
| everything run through the manufacturer's servers, but it was
| chosen to do it this way.
|
| (emphasis from article)
|
| I'm working on IoT cloud system. It was chosen to be done this
| way because netither consumers nor installers have any expertise
| whatsoever to setup their own network or any devices to be
| acessible from outside (and they want their panels to be
| accessible when they are outside their home). I can do it, most
| readers of HN could do it, but typical consumer or installer
| can't. Sad but true.
| Nextgrid wrote:
| The cloud can operate as a dumb TURN relay relaying
| E2E-encrypted traffic. Then the worst the cloud can do is deny
| service to remote management (and even then, local management
| would still work), but it wouldn't be able to send direct
| control commands to the equipment since they don't have the
| authentication nor encryption keys.
|
| This also makes it simpler from a programming point of view -
| instead of having separate cloud sync & local control
| protocols, you just have one local protocol and you merely
| tunnel it through the (dumb) cloud if you can't connect
| directly.
| yetihehe wrote:
| It could, but this requires to store historical data about
| usage on devices. If you store that encrypted data in cloud,
| then getting it to your mobile phone is super slow. If you
| store it in cloud, you can get historical data even if your
| device is dead or has 256 BYTES of memory and 1 megabit of
| flash storage. We have such devices, very effective at
| managing local municipal heating network and controlling
| several thermal controllers each via rs232 or rs485.
| Fortunately we preemptively moved everything into VPN'ed
| mobile network, we need special approval to touch anything on
| that network and can't connect without them granting access,
| so after EU started moving with cybersecurity this year, we
| are covered.
|
| > This also makes it simpler from a programming point of view
| - instead of having separate cloud sync & local control
| protocols, you just have one local protocol and you merely
| tunnel it through the (dumb) cloud if you can't connect
| directly.
|
| Having only cloud protocol is even simpler, I've done all of
| the above (I do backend and our firmwares).
| Nextgrid wrote:
| > we preemptively moved everything into VPN'ed mobile
| network
|
| Unless your device itself is handling the VPN, I have bad
| news for you if you trust the mobile network to not open
| your devices up to malicious attackers:
| https://berthub.eu/articles/posts/5g-elephant-in-the-room/
| yetihehe wrote:
| We consider "they hacked the mobile network VPN's AND had
| time to reverse our protocol before being booted out of
| network" as too high a level to be resolved by us. If
| someone has enough resources to do this, he will also
| just hack into standard-level secured server at municipal
| office and there will probably be no one there to stop
| him or discover what went wrong.
| adrianN wrote:
| Do you at least fuzz your software?
| DoctorOetker wrote:
| reversing the protocol can be done in advance, if they
| order your product
| wmf wrote:
| I don't think E2E is simpler to program if you want to get it
| right. There are entire companies whose raison d'etre is
| actually managing keys properly (e.g. Signal, Tailscale).
| akira2501 wrote:
| This should be the basic model. A fully third party TURN
| service. You pay $20/mo to keep your home connected, and all
| devices and providers can use a standard protocol, and users
| remain fully in control of their data.
| bdamm wrote:
| For IoT stuff in general; I can do it, and I don't want to
| because I'd rather spend my time doing other things (although
| yeah, I totally did learn everything I could about my solar
| array, because it is a source of power, after all. But for the
| other stuff...)
| danielovichdk wrote:
| These plants or farms are usually built around and on top of
| industrial IEC protocols and SCADA controllers which is a lot
| more low level than what any cloud IoT privider offers.
|
| I have done a controller for a 40 foot container battery and it
| wasn't like we received any API from Hitachi (battery
| manufactor). We had to write everything ourselves.
| pheatherlite wrote:
| If we've learned anything from the security cam and baby cam
| scandals, then it's that convenience is king and we as a
| society would rather risk everything than be arsed to take few
| additional steps to setup/learn something to prevent such basic
| breaches. We (the society) don't even want to change the
| default password on most things.
| yetihehe wrote:
| > We (the society) don't even want to change the default
| password on most things.
|
| Like you wouldn't believe.
|
| My most memorable case of insecure IoT devices - wifi socket
| was sending wifi ssid and password of the network in
| cleartext in every ping packet to chinese servers.
| titzer wrote:
| People gonna be people. It's up to engineers and product
| designers to make things user friendly but also safe-by-
| default. If something needs to be configured, then provide
| instructions on how to configure it. Instead of pretending
| that it's society's fault (can't be _arsed_ ), maybe ask why
| the IT industry can't make instructions that are written out
| --explicit, fairly standard, and easy to follow--like the
| manual for putting together a piece of furniture. Or why the
| stupid device doesn't come with a randomly-generated strong
| password taped to it.
| akira2501 wrote:
| > and we as a society would rather risk everything than be
| arsed to take few additional steps
|
| Large manufacturers would like you to think this. It would
| provide them a convenient excuse for not even trying to
| differentiate the market along these lines.
|
| > We (the society) don't even want to change the default
| password on most things.
|
| Actually.. I just want to use my device _first_ and not go
| through some manufacturer controlled song and dance of dark
| patterns.
|
| In my experience, if you don't pre load the user with this
| garbage, and then wait for them to have an actual _need_ that
| depends on the feature, they're FAR more compliant with
| following even lengthy instructions to get it done.
|
| It's more a problem of aligned benefits and timing than
| anything else.
| lucianbr wrote:
| What's the reasoning for not allowing both control paths, via
| cloud but also locally? So that people who can and want to,
| will use the local control.
| yetihehe wrote:
| Cheapness. It would require to be at least semi secure,
| application on phone would need to find those devices locally
| and it should be synchronized with cloud anyway,
| synchronization is error prone and we had problems with
| devices sometimes responding twice or very slowly through
| local interface (through cloud was much faster, no idea why,
| not our firmware). Also not enough people requesting that
| feature, most don't care and think that losing internet is
| not often enough to warrant worrying about this.
| MostlyStable wrote:
| Why not offer an either/or rather than both? Some people (I
| am one of them) actively do not want these kinds of things
| to be managed through the cloud servers. I don't want it to
| sync, I want to fully turn that off. I want to locally
| host, and I'm willing to take responsibility for that
| feature, including when it breaks. All I want is access to
| whatever the data reporting and control APIs are.
|
| I get that I'm a tiny minority, and that very few customers
| want what I want. But A) it seems like giving me what I
| want should be very cheap (i.e should not entail ongoing
| customer support costs beyond normal, and in fact would get
| rid of the small cloud hosting cost) and B) I'd be willing
| to pay a premium to get it.
| wmf wrote:
| In some areas like cameras there are a decent number of
| cloud-free alternatives. Hopefully as the IOT market
| grows we'll get cloud-free versions of everything.
|
| I think you're too optimistic about costs though.
| Providing any support at all, even one-time during the
| install, is expensive and cloud-free IOT is going to
| require support due to home networks being broken.
| MostlyStable wrote:
| Yes, support is expensive, but what I am proposing will,
| if anything, reduce support. I'm imagining something
| where, if I opt into local control, I am giving up all
| rights to any support that is not related to the core
| functionality of the device. For example the solar
| panels/inverters in the article. If I opt in to local
| control, then the only support I am entitled to is the
| solar panels stop generating power or if the inverter
| stops inverting. Anything that is network related is no
| longer the companies problem, because I have assumed
| complete responsibility for that. I'd even be willing to
| agree that, in the case that I ever decide I don't want
| local control, and I want to switch to the cloud hosting,
| that I will _pay_ for the support required to switch me
| back over.
|
| So if my home network breaks, that is not their problem.
| And they don't need to set it up, they just need to make
| it possible for _me_ to set up, including figuring out
| how to make it work with my potentially broken home
| network. If it requires a new router because mine doesn
| 't provide some necessary functionality? Not their
| problem. Etc. Etc.
| wmf wrote:
| Consumer electronics doesn't work that way. If people
| can't get a product to work they will return it to the
| retailer and when the retailer gets a lot of returns they
| will penalize the company or drop them completely.
| lucianbr wrote:
| I have some shelly devices which manage to do all that, and
| cost next to nothing. Work with local rest services or
| cloud, password protection, TLS. Sure, it costs more than
| zero, but not much.
|
| In the end, freedom goes away because we could not be arsed
| to ask for it at least, let alone fight.
| toast0 wrote:
| The real answer is it's more than twice the work to have both
| paths, and there's not enough demand for it.
|
| That said, Apple Homekit integration is local network based,
| so products that do that and the typical manufacturer cloud
| system have done both paths.
|
| Homekit is a pain to use without Apple hardware/software, but
| there you go. (There's a plugin for HomeAssistant, but I'm
| still classifying that as a pain)
| akira2501 wrote:
| I have a weather station.
|
| It can connect to standard cloud weather service providers
| and I can view my data there.
|
| I can also just redirect that exact same protocol to any
| other host or IP I specify.
|
| They built it once and just gave me the ability to control
| WHERE that data goes. It's honestly not that hard.
| michaelt wrote:
| Often there are two control paths. Sometimes more! Plenty of
| inverters will quite happily give you an RS232 port
| specification and you can create your own dongle!
|
| However, for _purpose of the security of the nation 's power
| grid,_ I don't just need _my_ inverter to be secure, I need
| _pretty much everyone 's_ inverter to be secure. If an attack
| bricks 95% of solar inverters, the fact the nerdiest 5% of
| users have their inverters airgapped won't stop the grid
| having a lot of problems.
| lucianbr wrote:
| > RS232 port specification and you can create your own
| dongle!
|
| This is just a way of pretending to give access while
| making it as hard as possible. We are talking about a
| device that is already connected to the network. The local
| path is not some rest services, but a serial port for which
| I need to fabricate some hardware? Don't piss on me and
| tell me it's raining.
| michaelt wrote:
| Perhaps I wasn't clear - when I say "Sometimes more!" I
| mean many cheap chinese inverters actually support four
| options:
|
| 1. Cloud management with their app.
|
| 2. Wifi management without the cloud (when you're on your
| home wifi).
|
| 3. Unplug the wifi dongle from the inverter for a fully
| offline system. You don't really need your inverter on
| the internet anyway.
|
| 4. Unplug the wifi dongle and DIY whatever you want, the
| dongle's just a serial-to-wifi converter.
|
| That's not to say the security of any of this stuff is
| good, of course. In fact the security is pretty bad! But
| you can for sure get inverters with multiple options for
| non-cloud operation.
| bee_rider wrote:
| Also administering a bunch of IOT systems is a pain. If
| something is an open source community project, ok, I'll play.
| If somebody is selling a product they are responsible for
| making sure it works.
| Gibbon1 wrote:
| You could put an sql database on a local device and just
| access it remotely like anything else. But you are correct
| you're stuck with administering each and everyone one of
| them.
|
| The standard go to a raz pi solution will up and die every
| few months. And half the time you'll need physical access to
| get it back. It takes a lot of work to develop an embedded
| system that has enough reliability.
| MathMonkeyMan wrote:
| To be fair, I can do it only if I have time and physical access
| to the network. Home routers have different gateway IPs,
| different web interfaces, different password policies (e.g.
| there might be an admin password and an additional password for
| changing anything), etc.
|
| It reminds me of <https://xkcd.com/627/>, but when you're
| launching a product that isn't good enough.
|
| It's hard enough to open up a port even with uPNP (typically
| disabled) and other made-for-purpose tech. Torrent clients end
| up trying to poke holes and such. Service discovery might work
| via local UDP broadcast, or it might not. LAN clients might
| live at 10.* or 192.* or be isolated by default. It's easier to
| just go onto the public internet and contact some mysterious
| server. Botnet by design.
| kkfx wrote:
| Victron (to cite an NL vendor) actually can perfectly operate
| in LAN only via MQTT and ModBUS also offering a (bad) WebUI
| locally for settings pretty anything, including a display for
| the said WebUI in a framebuffer with an embedded mini-keyboard.
| It's up to the installer decide to go with their cloud offer or
| not.
|
| The sole remark I have against them (beside the not so good
| software quality it's the impossibility for individual owners
| to do offline updates, we can upgrade via VRM portal but not
| downloading fw and flash it locally even if the needed device
| is on sale, because they offer fw files only to registered
| vendors.
|
| Fronius (to remain in the EU) have a local WebUI witch need a
| connection only for fw updates, even if differently from
| Victron it's not a Debian based system with sources available
| but a closed source one, they unfortunately offer only a very
| limited REST API and a very slow ModBUS but still anything con
| be do locally.
|
| I'm not sure, since I haven't any myself by SMA (Germany) and
| Enphase (USA) seems to been able to operate offline as well.
|
| Stated that, yes, you are damn very right in saying most
| installers have no competence, thankfully where I live self-
| installation is allowed (at least so far), but that's simply
| demand better UIs and training for them perhaps avoiding the
| current state of the industry with an immense amount of CRAP at
| OEM level, with most "state of art" systems not at all designed
| to be used in good ways (see below) and absurdly high prices to
| the customer at a level it's not interesting installing p.v...
| 4 years ago I paid my system 11.500EUR for 5kWp/8kWh LFP, the
| smallest offer to have it designed and built by someone else
| was ~30.000EUR the most expensive ~50.000EUR and all the 6
| offers I tried shows some unpleasant issues and incompetence.
|
| About OEMs just observe how ABSURD is that there is no damn DC-
| to-DC direct car charger. Most EVs now have 400V batteries, the
| same of stationary batteries, with equal BMS comms. Why the
| hell not sell an MPPT-to-CSS combo direct solution? Ok, we do
| not ONLY charge from the Sun, than it's perfectly possible have
| a compo charging station with DC for p.v. and AC for the grid,
| switching from one to another as needed. It's ~30% energy lost
| in double conversion.
|
| Why no DC-to-DC high power appliance who still run DC
| internally (A/C, hot-water heat-pump heaters etc)?
|
| Why not a modern standard protocol for integration of anything
| instead of building walled gardens?
|
| Long story short OEMs have choose the cloud model partially
| because most installers are electricians able to use desktop
| holding the mouse with one hand and clicking with the other,
| but also because they have no intention to made user-
| interesting solution in an open market...
| oezi wrote:
| The key failure is that despite the IPv6 transition we don't
| have static IPs at home and can start hosting servers at home.
|
| Certainly this requires a lot of progress to secure the IOT
| space, but we can allow the enshitification of clouds to
| continue.
| dathos wrote:
| I live off-grid, power and water wise, and it really irked me
| that the monitoring coming with my inverter is only available
| online. Even when there is a network available the app will not
| work. I fixed this by getting a raspberry pi connected and
| reading it from there, but if I disconnect the inverter from the
| internet it will create a new network so now there is always an
| open network in the middle of nowhere with no option to disable
| it.
|
| I'm thinking about screwing it open and desoldering the wifi
| module but honestly I'll replace it in the next couple of years
| so I'd rather not kill myself by making a mistake.
| ansible wrote:
| The high-voltage side should be separated from the electronics,
| so it shouldn't be dangerous if you are observant.
|
| It may be sufficient to just disconnect the antennas from the
| WiFi module, that will help prevent any network connections.
| Nextgrid wrote:
| Disconnecting the antenna would still have leakage at close
| range. Grounding the antenna might be a better option. But in
| practice, the dangers highlighted by the article only surface
| when an attacker has control of many solar plants at scale.
|
| Compromising an individual one by getting close-range
| physical access will be a local annoyance but wouldn't scale
| to a level where it can threaten the grid, so it limits the
| pool of potential attackers to local vandals (which can
| achieve their goals easier by just throwing rocks at your
| panels).
| ijustlovemath wrote:
| Without an antenna, even at close range, initial handshakes
| will fail or be unreliable.
| serial_dev wrote:
| Disclaimer, ymmw, if you have no clue about these systems
| (average people), you can still easily kill yourself in the
| process.
| grecy wrote:
| What inverter do you have? Many like the Fronius have a
| removable networking card.
| m463 wrote:
| why can't people just make stuff and sell it?
| lotsofpulp wrote:
| In a developed country, there are lots of regulations and
| liabilities you are exposed to once you start selling
| something.
| Spivak wrote:
| Because humans are an ongoing cost and no one has figured how
| to sell non-consumable slowly depreciating goods as one-off
| purchases and keep paying your employees once you saturate
| your market.
|
| Option 1: Artificially sell the thing as an ongoing cost.
|
| Option 2: Artificially make the depreciation cycle faster.
| Get consumers to regularly replace it anyway with upgrades or
| trend changes.
|
| Option 3: Make ongoing money from the item via a side-channel
| (tvs are great at this one)
|
| Option 4: Manufacture and sell a huge number of different
| goods across market segments and weather the slow
| depreciation cycle (Oxo does this).
|
| Option 5: Sell some consumable good you can get recurring
| revenue from along side the item (Coffee pods, printer ink)
|
| Option 6: Make up the money on maintenance, repairs, and
| financing. Become a bank.
|
| Option 7: Make your money in some other sustainable
| profitable business and drop the product once you've gotten
| what you can for it.
|
| All of these kinda suck and option 1 is easy to implement.
| formerly_proven wrote:
| Most newer solar inverters can't even be set up without internet
| and most functions are only available with an always-on internet
| connection. This is also true for EU companies like SMA for
| example.
| grecy wrote:
| I just installed a Fronius inverter (made in Austria) and 6.8kW
| of panels.
|
| The inverter itself functions perfectly fine without an
| internet connection, and will display instantaneous power
| output on the screen. I could just be content with that and
| look at my monthly power bill to see how much I generated and
| how much I used each month and never connect it to the
| internet.
|
| To get any kind of data logging & history from the inverter, it
| must be internet connected (wifi or ethernet). And all of that
| is through the manufacturer's website, which constantly nags me
| to "upgrade to pro" for some obscure feature that I'll never
| use.
| ano-ther wrote:
| Eye opening for me. One of the arguments for renewable energy
| (besides emissions) has always been its potential for
| decentralizing power generation. Makes it more resilient,
| democratizes the means of production etc.
|
| This article shows that we inadvertently introduced new choke
| points. And of course the global security environment makes it
| more worrisome.
| panki27 wrote:
| Hmm, almost like what happend to the internet... the idea being
| "everything is decentralized", but now +80% of traffic passes
| through Cloudflare and over 90% of mails come from 2 providers!
| paxys wrote:
| Cloudflare absolutely does not control 80% of internet
| traffic. I have no idea where you got that number from.
| ezfe wrote:
| This article says 80% of known websites, which are 19% of
| all websites. Probably where it came from.
|
| https://w3techs.com/technologies/details/cn-cloudflare
| paxys wrote:
| That's still the _number_ of websites, not their traffic.
| A personal blog hosted on Cloudflare and google.com are
| not both the same.
| realusername wrote:
| It never made any sense anyways, nothing can really escape the
| economy of scale, whatever the technology being used.
| lupire wrote:
| Solar is not the same as renewable.
|
| Renewable and decentralized are different axes.
| kkfx wrote:
| Yes, p.v. have opened the way for semi-autonomy depending on
| where you live BUT ruling class really dislike this, they want
| slave not Citizens and tie people to service it's a very good
| way of making slaves who can't revolt.
|
| That's why instead of pushing self consumption and semi-
| autonomous systems we push grid-tied and cloud-ties crap, to be
| tied to someone else service, slave of that. It's the "in 2030
| you'll own nothing" already a reality in modern cars, connected
| to the OEM with a much higher access than the formal owner,
| much modern IoT and cloud+mobile crap. People do not even
| understand they do now own, until it's too late.
|
| Another simple example: in most of the world banks between them
| have open standard to automatic exchange transaction, in EU
| that's OpenBank APIs, with signed XML and JSON feeds. There is
| NO REASON to block customers for directly use such APIs from a
| personal desktop client. All banks I know block such usage. So
| you do not have all your transactions signed by the bank on
| your iron, you have NOTHING in hand. In case of "serious
| issues" you have nothing to prove what you have on your bank,
| what you have done with your money. In the past we have had
| paper stuff to prove, we now have signed XML/JSON witch is even
| better than paper being much harder to falsify, but no, we miss
| because 99% must own nothing.
|
| We have connected cars with a SIM inside, but instead of having
| the car offering APIs and a client or perhaps even a WebUI,
| directly to their formal owner we have to pass through their
| OEM, the real substantial owner. And we can't even disconnect
| the car. In the EU it's even illegal for new car to be
| disconnected since the emergency e-call service must be active
| on all new cars.
|
| And so on.
| adolph wrote:
| _The short version: most consumer and business solar panels are
| centrally managed by a handful of companies, mostly from
| countries outside of Europe. In the Netherlands alone, these
| solar panels generate an output equivalent to at least 25 medium
| sized nuclear power plants. There are almost no rules or laws in
| Europe governing these central administrators. . . . The same
| thing goes for heat pumps, home batteries, and EV charging
| points._
|
| Seems to me that this is very similar to the situation with IoT
| only with higher stakes. I appreciate this article's presentation
| of inverter and grid trust.
|
| Beyond trusting customer inverters to do the right thing, I
| wonder if there is a method for safing a grid at the hardware
| level. Naive question: could there be a grid provider device that
| prevents overcurrent or incorrectly clocked cycles?
| kwhitefoot wrote:
| The utility company fuse between the property and the 240 V
| distribution system should prevent overcurrent. If the
| frequency or phase of the inverter is wrong the inverter might
| die first unless the network is already down.
|
| There isn't really any practical way to prevent overvoltage
| though. So a rogue controller in charge of all the solar
| systems in a street might be able to do quite a lot of damage
| to consumer devices.
|
| A problem from the utility point of view is that they can no
| longer guarantee that the 240 V side of the distribution system
| is safe to work on just by tripping a breaker on either side of
| the distribution transformer. So all work on the 240 V
| distribution system has to be done with the assumption that the
| system is live.
|
| Eventually regulations will be updated, if necessary, to deal
| with large numbers of solar installations on domestic
| buildings.
| cesarb wrote:
| > The utility company fuse between the property and the 240 V
| distribution system should prevent overcurrent. If the
| frequency or phase of the inverter is wrong the inverter
| might die first unless the network is already down.
|
| To put it more simply: if the phase is wrong, the effect is
| the same as a short circuit, which fuses and circuit breakers
| protect against. If the frequency is wrong, the phase will
| become wrong after a number of cycles.
|
| > There isn't really any practical way to prevent overvoltage
| though. So a rogue controller in charge of all the solar
| systems in a street might be able to do quite a lot of damage
| to consumer devices.
|
| There is, it's called a surge protector or surge protective
| device (SPD). It converts any overvoltage above a certain
| level into a short to ground, which then trips the fuse or
| circuit breaker. It's often used as a protection against
| lightning-induced currents.
|
| > A problem from the utility point of view is that they can
| no longer guarantee that the 240 V side of the distribution
| system is safe to work on just by tripping a breaker on
| either side of the distribution transformer. So all work on
| the 240 V distribution system has to be done with the
| assumption that the system is live.
|
| From what I've seen, the utility workers usually ground the
| wiring when working on it (they have a special-purpose device
| for that). Once it's safely connected to ground, it's no
| longer live.
| ThrowawayTestr wrote:
| If the general public knew how fragile the power grid is no body
| would be able to sleep at night.
| asynchronous wrote:
| Not to be that guy, but the DOE is arguably one of the most
| important federal agencies in the US, and they treat the
| problem with the correct amount of focus, research and
| dedication. It's just a very hard problem. The grid is no less
| secure or less resilient than it was 50 years ago, the main
| problem is that people are more dependent on it. Almost no one
| buys a personal generator before an outage happens anymore,
| despite it being one of the cheapest ways to get resiliency.
| twoodfin wrote:
| Isn't the right place to fix this at the junction between the
| plants and the grid? Regulate the grid utilities into a gateway
| role, and require all inverter control & telemetry traffic to
| pass through them.
|
| This seems likely to be more fruitful than attempting to regulate
| 400 Chinese panel manufacturers.
|
| What am I missing?
| itishappy wrote:
| You're thinking about this right, just at a utility scale.
|
| The "plants" in this context are homes and businesses. The
| junction points between plants and grid are the inverters sold
| by the panel manufacturers.
| WaitWaitWha wrote:
| Q: Are there no regulatory requirements for power plants of any
| kinds in EU, specially around cybersecurity?
|
| I do not allow any system into my environments (at home and at
| work) that _requires_ a third party data connection function.
|
| There are way too many incidents where a provider, cloud or
| otherwise which required connection failed for various reasons.
|
| (e.g., Cisco Spark Board, Xerox ConnectKey, Google Cloud Print,
| WeWork's Connected devices, Lattice Egnines, MS Groove Music
| Pass, Shyp, Adobe Business Catalyst, Samsara, Zune, FuelBand,
| Anki Vector Robot, Google Stadia, Pebble)
|
| Despite this, I am very leery of regulating _solar power_
| specifically.
| afh1 wrote:
| Smartphones don't count?
| WaitWaitWha wrote:
| Apologies, but do not understand the question.
|
| Are you suggesting using smart phones should count in "not
| allowing it in"? Then yes, I try to where possible. I do not
| _depend_ on a smart phone. All functionality that are
| operationally necessary can be done elsewhere without major
| delays or impact.
| afh1 wrote:
| Interesting. How do you handle MFA, do you have a special
| device for that? Your bank/brokerage don't require their
| app?
| numpad0 wrote:
| How would one practically verify and certify cybersecurity of a
| product? Even payment smartcards sometimes come with non-
| malicious maintenance backdoors. There seem to be little to no
| academic theoretical basis to this whole software security
| thing.
| g_p wrote:
| Given the challenges of techniques like TLS interception
| (i.e. through pinning and other good security features),
| about the only measure I can see left is network isolation.
|
| You can set up a local network that has no WAN connectivity
| on it. About anything else is difficult to verify even the
| most basic of security properties. Certifying is another step
| up (although you could argue certifying is just a third party
| saying something passed a finite list of tests) - the real
| challenge is defining a meaningful certification scheme.
|
| There has been some good work towards consumer IoT device
| security (i.e. the 13 steps approach from the UK), that
| covers some of the lowest hanging fruit -
| https://www.gov.uk/government/publications/code-of-
| practice-...
|
| The trouble is that these set out principles, but it's hard
| to validate those principles without having about the same
| amount of knowledge as required to build an equivalent system
| in the first place.
|
| If you at least know the system is not connected to a WAN,
| you can limit the assurance required (look for WiFi
| funcitonality, new SSIDs, and attempts to connect to open
| networks), but at a certain point you need to be able to
| trust the vendor (else they could put a hard-coded "time
| bomb" into the code for the solutions they develop).
|
| I don't see much value in the academic/theoretical approaches
| to verification (for a consumer or stakeholder concerned by
| issues like these), as they tend to operate on an unrealistic
| set of assumptions (i.e. source code or similar levels of
| unrealistic access) - the reality is it could take a few days
| for a good embedded device hacker to even get binary firmware
| extracted from a device, and source code is likely a dream
| for products built to the lowest price overseas and imported.
| Kon-Peki wrote:
| > Incidentally, why are all those panels centrally connected
| anyway? I'd like to know what my panels are doing, but you don't
| need the internet for that.
|
| This is because of the market for carbon credits. When you
| installed your PV panels, someone estimated how much electricity
| they would generate over the next 10-15 years. Tradable carbon
| credits were created based on that estimate and went into the
| marketplace. And for the next 10-15 years they have to verify
| that the electricity was actually generated, or else someone has
| to pay back some money. Did you read the fine print on your
| contract? It is probably you that has to pay it back. You didn't
| know that one of the "rebates" you got was actually a pre-payment
| for those credits?!? Should have read the fine print ;)
|
| Oh yeah, BTW: that "rebate" was only _your_ portion of the
| credits. The installer got some of it (and doesn 't have to pay
| back anything), the person that filled out the paperwork you
| didn't know existed got some (and doesn't have to pay back
| anything)...
| kkfx wrote:
| That's why my system (Victron + Fronius) is offline, monitored
| with HA, BYD battery if there is no secret in-hw backdoor in my
| home server can't reach the internet as well. HA can, via
| wireguard, to act/monitor when I'm outside my home witch might be
| a serious threat but it's pretty easy to cut it off if needed.
|
| There is a more important part, while with p.v. we still can go
| offline, with car's we can't. My car is connected and I can't do
| NOTHING to manage it, it's managed by it OEM behind me and that's
| a much bigger threat since single cars can paralyze the nation if
| properly blocked in critical points of the road network.
|
| At a largest scale that's the reason we can't have a national
| smart grid but only individual smart microgrid, meaning p.v.
| should be used only for self-consumption NOT grid-tied like in
| California.
| SnorkelTan wrote:
| If solar panels can be turned off, why are utility companies
| having to sell excess power at a loss? Why can't they tell the
| solar farms to reduce their output by the required amount?
| bjornsing wrote:
| As I understand it: because the incentives are wrong.
|
| Owners of small scale solar panel installations are payed a
| fixed price per kWh in many EU countries, regardless of the
| market price. The taxpayers pick up the tab I guess.
| trebligdivad wrote:
| In theory someone somewhere should be incentivised to spend
| money on building storage systems so that they then have to pay
| less money in the future in excess days.
| sanderjd wrote:
| Solar power does get curtailed pretty often, but there isn't
| one uniform solution to the problem, different utilities /
| markets / grids have chosen different solutions to this.
| kkfx wrote:
| It's worst actually, at least in France, if you inject to the
| grid you have to pay an "energy transport fee", even if you
| inject for free (only recently self-made systems are allowed to
| sell energy, before they can only donate or not inject at all)
| and the injected energy is now paid less than the cheapest
| price to the customers (6 cent/kWh for ground based p.v., 10
| cent for on-roof p.v.). So well, we do not harm large utility
| business.
|
| What harm on scale is the variable output especially from small
| p.v. utilities built out of incentives NOT personal power
| plants, the grid is sized with some large power plants serving
| a large set of customers, their absorption vary but if the grid
| is vast (and not too vast) enough variation tend to be slow on
| average, let's say 50MW PP experience 100-200kW demand
| variation in very short time. They can compensate easily
| keeping the grid frequency stable. With a significant amount of
| grid injecting p.v. variation might be MUCH bigger creating
| significant stability issues where injection goes up too
| quickly making the frequency skyrocketing and large PP can't
| decrease their output fast enough risking disconnection witch
| in turn might put large p.v. plants offline suddenly creating a
| cascading effect of large blackouts.
|
| That's the real issue with grid-connected and tied renewables
| and another reason why we need to go toward self-consumption
| NOT injection.
| pas wrote:
| as far as I understand there's a market based solution.
| producers bid prices for time slots (consumers too, but that's
| less important from the perspective of a solar power plant) and
| if they win the contract is live, they need to input for that
| slot. if they miss (go over or under) they get paid less (and
| of course a penalty is possible too, theoretically it's the
| same)
|
| this incentivizes better capacity and availability forecasting
| for solar installations, and preserves the usual dynamics of
| the open energy market.
|
| ..
|
| the problem is with these super small ones, where initially
| states just let people connect it, because it's green, yey.
| (but now DSOs started to make connecting waay harder. and
| regulators are investigating, eg. in Spain. [0])
|
| of course the non-residential installations already usually
| need aFRR capability. (eg. this is the case in Hungary.)
|
| and there's already a market for "reserves" in the EU. (but the
| interconnection rate is below the target 15% as far as I know.
| but still, there are intra-state markets, etc.) and we can see
| that when solar is high the reserve prices are surging. [1]
|
| [0] https://caneurope.org/content/uploads/2024/04/Rooftop-
| Solar-...
|
| [1] https://gemenergyanalytics.substack.com/p/european-power-
| res...
| timClicks wrote:
| Does anyone know of an inverter manufacturer that doesn't require
| this? Ideally, one that offers micro inverters for each panel.
| leymed wrote:
| In greater scale, meaning power plants not the PV installed at
| houses, these things are taken more seriously and after
| purchase of equipment the control and automation of plant are
| in your hands. For example, Woodward, ABB have products with
| capacity up to 0.5 MW of single inverter.
|
| Micro inverter for each panel would be very costly. In 1 MW
| plant you will have around 4000 panels, communicating with that
| amount electronic devices would be a headache.
| delroth wrote:
| > In the Netherlands alone, these solar panels generate a power
| output equivalent to at least 25 medium sized nuclear power
| plants.
|
| Since this didn't pass the smell test: the author is looking at
| nameplate capacity, which is a completely useless metric for
| variable electricity production sources (a solar panel in my
| sunless basement has the same nameplate capacity as the same
| panel installed in the Sahara desert).
|
| Looking at actual yearly energy generation data, this is more
| like 1.5 times the generation of an average nuclear power plant
| (NL solar production in 2023: 21TWh, US nuclear production in
| 2021: 778TWh by 54 plants).
|
| Which maybe puts more into perspective the actual risks involved
| here. I'm not saying there shouldn't be more regulations and
| significantly better security practices, but otoh you could
| likely drive a big truck into the right power poles and cause a
| similar sized outage.
| epistasis wrote:
| You are talking about energy, which is not the same thing as
| power. TWh == energy, GW == power.
|
| The distinction is important, especially in the Netherlands,
| which has a capacity factor of only about 10%-15%, whereas most
| of the US will be at least 20%-25%, which is twice as high.
|
| I'm not sure of the typical number of reactors in the
| Netherlands, but using the US average of 1.6/power plant may
| not be the most representative comparison.
| delroth wrote:
| I have no idea what you're talking about, since nowhere did I
| use solar capacity factor data nor did I look at number of
| reactors per plant.
| epistasis wrote:
| You are using both with your energy generated numbers.
| That's where they come from.
|
| Your solar TWh comes from 25GW at ~15% capacity factor, and
| to get your nuclear numbers you're looking at 1.6GW for
| each of nuclear "plants" when each reactor is usually about
| 1GW or less. There are ~90 reactors in the US, at 54
| plants. The article is assuming 1 reactor per plant for the
| Netherlands.
| kkfx wrote:
| The point is about instant power injected, not energy, the
| point is that keep an AC grid at the right frequency it's a
| tricky business because energy production and consumption
| must match.
|
| Too much from production the frequency skyrocket, little
| production the frequency plunge.
|
| Now classic grids are designed on large areas to average the
| load for big power plants, this way those plant see small
| instantaneous change in their output demand, let's say a 50MW
| power plant see 100-300kW instantaneous change, that's
| something they can handle quick enough. With massive p.v.,
| eolic etc grid demand might change MUCH more for big power
| plant, like a 50MW P.P. need to scale back or power up of
| 10MW suddenly and that's way too much to sustain. When this
| happen if the demand is too much the frequency plunge, grid
| dispatcher operators have to cut off large areas to lower the
| demand (so called rolling blackouts), when the demand drop
| too quickly the frequency skyrocket and large PP can't scale
| back fast enough so they simply disconnect. Disconnecting the
| generation fall and the frequency stabilize, unfortunately
| most p.v. is grid tied, if a p.p. disconnect most p.v.
| inverters who have seen the frequency spike disconnect as
| well creating a cascading effect of quickly alternating too
| low and too high frequency causing vast area blackouts.
|
| Long story short a potential attack is simply planting a
| command "at solar noon of 26 June stop injecting to the grid,
| keep not injecting till solar noon + 5'", with just "1 second
| or so" (due to eventual time sync issues) all inverters of a
| certain brand might stop injecting, making the generation
| fall, a bit of rolling blackouts and large pp compensate
| quickly. Than the 5' counter stop, all inverters restart
| injecting en-masse, while the large pp are full power as
| well, the frequency skyrocket, large pp disconnect causing
| most grid-tied inverter to follow them, there are large
| change an entire geographic segment of a grid fall.
| Interconnection operators in such little time do not know
| what to do and quickly the blackout might became even larger
| with almost all interconnection going down to protect active
| parts of the grid, causing more frequency instability and so
| more blackouts.
|
| Such attack might led to some days without power.
| 1053r wrote:
| For the purposes of information security, the nameplate
| capacity is the correct number to consider for a very simple
| reason: we must defend as if hackers will pick the absolute
| worst moment to attack the grid. That is the moment when the
| sun is shining and it's absolutely cloudless across
| Netherlands, California, Germany, or wherever their target grid
| is.
|
| At that moment, the attacker will not only blast the grid with
| the full output of the solar panels, but they will also put any
| attached batteries into full discharge mode as well, bypassing
| any safeties built into the firmware with new firmware. We must
| consider the worst case, which is that the attacker is trying
| to not only physically break the inverters, but the batteries,
| solar panels, blow fuses, and burn out substations. (Consider
| that if the inverters burn out and start fires, that's a
| feature for the attacker rather than a bug!)
|
| So yes, not only is it 25 medium sized nuclear power plants,
| it's probably much higher than that! And worse, that number is
| growing exponentially with each year of the renewable
| transition.
|
| This was probably the scariest security expose in a long time.
| It's much much worse than some zero-day for iphones.
|
| A bad iPhone bug might kill a few people who can't call
| emergency services, and cause a couple billion of diffuse
| economic damage across the world. This set of bugs might kill
| tens of thousands by blowing up substations and causing outages
| at thousands to millions of homes, businesses, and factories
| during a heat wave. And the economic damage will not only be
| much higher, it will be concentrated.
| hn_throwaway_99 wrote:
| While I agree that the important metric to consider is peak
| output and not average output, I would still guess that in a
| country like the Netherlands that peak output is nowhere near
| nameplate capacity.
| Retric wrote:
| You can get close to peak output just about anywhere,
| assuming the panels are angled rather than laying flat. You
| just can't get it for very long in most locations.
| idiotsecant wrote:
| This is wildly overstating the issue. Hackers are not going
| to break into hundreds of separate sites, compromise
| inverters, compromise relay protection, compromise SCADA
| systems, and execute a perfectly timed attack. Even if they
| did, these are _distributed_ resources, they don 't all go
| through a single substation and I doubt any one site could
| cause any major harm to any one substation.
|
| Instead, they're going to get a few guys with guns and shoot
| some step of transformers and drive away.
|
| The problem with infosec people is they tend to wildly
| overestimate cyber attack potential and wildly underestimate
| the equivalent of the 5 dollar wrench attack.
| Gud wrote:
| Most(more or less all of them) grid operators can operate
| their network remotely from a single control room.
|
| I suspect most grids are extremely easy to hack(never
| tried, don't bite the hand that feed you etc).
|
| Info sec is just a hobby of mine. I install high voltage
| switch gear for a living.
| TwiztidK wrote:
| > I suspect most grids are extremely easy to hack
|
| I'd expect the opposite. All companies controlling
| equipment that is part of the "Bulk Electric System" have
| to be NERC CIP compliant and are audited regularly with
| large fines for non-compliance. Doesn't guarantee perfect
| (or even good security) but it's more likely to be a
| priority.
| 1053r wrote:
| This isn't hundreds of separate sites that have to be
| hacked individually. This is fewer than 10 clouds with no
| security to speak of and the ability to push evil firmware
| to millions of inverters worldwide, where in a few years at
| the current rate of manufacturing growth, it will be 10s,
| and then 100s of millions of inverters.
|
| Yeah, the potato cannon filled with aluminum chaff or
| medium caliber semi-automatic rifle can take down a
| substation. But this is millions of homes and businesses,
| which can all have an evil firmware that triggers within
| seconds of each other. (There will inevitably be some
| internal clocks that are off by days/months/years, so it's
| not like it will happen without warning, but noticing the
| warning might be difficult.)
|
| And the growth in sales is exponential!
| ethbr1 wrote:
| > _medium caliber semi-automatic rifle_
|
| Technically, anything that can put a hole in an oil-
| filled transformer.
| https://en.m.wikipedia.org/wiki/Transformer_types#Liquid-
| coo...
|
| You don't need to break it... just crack the radiator
| enough for all the circulating fluid to drain, then it
| overheats.
| EdJiang wrote:
| Important to point out this isn't just theory, it's
| actually happened (in the SF Bay Area!) with a regular
| rifle.
|
| https://en.wikipedia.org/wiki/Metcalf_sniper_attack
|
| https://www.npr.org/sections/thetwo-
| way/2014/02/05/272015606...
| ethbr1 wrote:
| Also in the north GA mountains in the 1970s.
| g_p wrote:
| They don't need to break into separate sites though - the
| issue at hand is that a single failure in the centralised
| "control plane" from the vendor (i.e. the API server that
| talks to consumers' apps) can be incredibly vulnerable.
|
| Here's a recent example where a 512-bit RSA signing key was
| being used to sign JWTs, allowing a "master" JWT to be
| signed and minted, giving control of every system on that
| vendor's control system.
|
| https://rya.nc/vpp-hack.html
| bramblerose wrote:
| The failure mode is much simpler: you don't need to
| physically break anything, you just need to drop 10GW of
| production from the grid (send a "turn off" command to all
| solar inverters) leading to a cascade of failures. Getting
| the grid back online is a laboreous manual process which will
| take (a lot of) time. Think
| https://en.wikipedia.org/wiki/Northeast_blackout_of_2003 or
| https://en.wikipedia.org/wiki/2021_Texas_power_crisis .
| tivert wrote:
| > Getting the grid back online is a laboreous manual
| process which will take (a lot of) time. Think...
|
| It would be even _more_ laborious and take _more_ time to
| bring things back online if the attacker manages to damage
| or destroy equipment with an overload like the GP
| describes.
| msandford wrote:
| The "turning the grid up to 11" attack isn't really
| possible. I know it seems like it is, but the inverters
| will only advance frequency so much before they back off,
| the inverters will only increase voltage so much. Etc.
| Sounds scary, isn't practical.
|
| Turning everything off when the panels are at peak
| output? That lets frequency sag enough that plants start
| tripping offline to protect themselves and the grid and
| it'll cascade across the continent in just a few minutes.
| Then you have a black start which might take months.
|
| There's an excellent video on how catastrophic a black
| start is.
| https://youtu.be/uOSnQM1Zu4w?si=x0dA7X7-19CJm6Kf
| mschuster91 wrote:
| > We must consider the worst case, which is that the attacker
| is trying to not only physically break the inverters, but the
| batteries, solar panels, blow fuses, and burn out
| substations.
|
| Power transformers have a loooooooot of thermal wiggle room
| before they fail in such a way and usually have non-
| computerized triggers for associated breakers, and (at least
| if done to code, which is not a given I'll admit) so do
| inverters and every other part. If you try to burn them out,
| the fuses will fail physically before they'll be a fire
| hazard.
| 1053r wrote:
| This is true, especially for low frequency (high mass)
| inverters. The inverters that are covered here are
| overwhelmingly high frequency (low mass) inverters. We hope
| that they practiced great electrical engineering and
| layered multiple layers of physical safeguards on top of
| the software based controls built into the firmware.
|
| Of course a company that skimped to the point of total
| neglect on software security would never skimp anywhere
| else, right? Right?
|
| :crossed-fingers: <- This is what we are relying on here.
|
| And even if they did all the right things with their
| physical safety, the attackers can still brick the
| inverters with bad firmware and make them require a high
| skill firmware restore at a minimum and turn them into
| e-waste and require an re-install from a licensed
| electrician at a maximum.
| mschuster91 wrote:
| > Of course a company that skimped to the point of total
| neglect on software security would never skimp anywhere
| else, right? Right?
|
| At least in Europe, product safety organizations and
| regulatory agencies have taken up work to identify issues
| with stuff violating electrical codes (e.g. [1] [2]) and
| getting it recalled/pulled off the market.
|
| Sadly there is no equivalent on the software side - it's
| easy enough to verify if a product meets electrical
| codes, but almost impossible to check firmware even if
| you have the full source code.
|
| [1] https://www.bundesnetzagentur.de/SharedDocs/Pressemit
| teilung...
|
| [2] https://www.t-online.de/heim-
| garten/aktuelles/id_100212010/s...
| t0mas88 wrote:
| The risk is not turning all solar installations "on maximum".
| That happens nearly every summer day between 1 and 2pm.
| Automatic shutoff when the grid voltage is rising can be
| disabled, but more than 9 out of 10 consumer solar
| installations in the Netherlands deliver their maximum output
| on such a day for most of the summer, not running into the
| maximum voltage protections.
|
| The big risk is turning them all off at the same time, while
| under maximum load. That will cause a brown-out that no other
| power generator can pick up that quickly. If the grid
| frequency drops far enough big parts of the grid will
| disconnect and cause blackouts to industry or whole areas.
|
| It will take a lot of time to recover from that situation.
| Especially if it's done to the neighbouring grids as well so
| they can't step in to pick up some of the load.
| eldaisfish wrote:
| you are splitting hairs about the wrong issue.
|
| When it is sunny in the netherlands, it is likely sunny
| everywhere in NL because of how small the country is.
|
| This is the situation where having so much solar power capacity
| (kW) is dangerous.
|
| The risk scales with energy output but it would not term
| nameplate capacity a "completely useless metric".
| hinkley wrote:
| I dunno. I lived next to a small inland sea most of my adult
| life. The number of times someone on the other side of town
| asserted it was raining when in fact it was not was quite
| high.
|
| Every adult in Seattle eventually has to learn that if you
| have an activity planned on the other side of town, if you
| cancel it because it's raining at your house you're not going
| to get anything done. You have to phone a friend or just show
| up and _then_ decide if you're going to cancel due to
| weather.
|
| Now to be fair, in the case of Seattle, there's a mountain
| that multiplies this effect north versus south. NL doesn't
| have that, but if you look at the weather satellite at the
| time of my writing, there are long narrow strips of precip
| over England that are taller but much narrower than NL.
| lucianbr wrote:
| > When it is sunny in the netherlands, it is likely sunny
| everywhere in NL because of how small the country is.
|
| Often friends of mine who live in my city report rain when I
| see none, or no rain when it's raining outside my window.
| That's to say nothing of a location 30km away, where
| basically anything can happen. Do we live on the same planet?
| hinkley wrote:
| If memory serves, and I'll admit it's pretty fuzzy, the US
| tends to make ridiculously large nuclear reactors and Europe
| has an easier regulatory situation so they make more of them
| and smaller.
|
| So in addition to the other stuff people mentioned, you might
| be off by another factor of 2 there. They also said "medium
| sized" so let's call it 3.
| cesarb wrote:
| > the author is looking at nameplate capacity, which is a
| completely useless metric for variable electricity production
| sources
|
| For solar panels, the nameplate capacity is usually also the
| power generated at the peak production time, which is the
| moment when an attacker turning off all inverters at the same
| time would have the most impact.
|
| That is: for an attack (or any other failure), the most
| important metric is not the total power produced, but the
| instantaneous power production, which is the amount which has
| to be absorbed by the "spinning reserve" of other power plants
| when one power plant suddenly goes offline.
| wiredfool wrote:
| No, the nameplate capacity is what a solar panel will produce
| under perfect lighting, independent of the site where it's
| installed.
|
| The peak theoretical power output of a solar panel depends on
| where it's installed, inclination, temperature, elevation,
| and so on. The actual peak power is going to take weather and
| dirty panels into account.
|
| 1kw nameplate in Ireland (or the Netherlands) is never going
| to give you an instantaneous 1kw output -- you're going to be
| lucky to see 60% of that.
| bramblerose wrote:
| It's the power output that is relevant for the failure mode
| described in the article, not the yearly production. And in
| terms of power output, 20GW is an incredibly common number for
| peak solar production (see e.g. https://energieopwek.nl/ at the
| end of Jul this year) in summer. Borssele (the medium-sized
| power plant named in the article) has a 485MWe net output. So
| yes, we _are_ talking about >25 mid-sized nuclear power plants!
| neilv wrote:
| > _0.002 MW - Small set of technical standards, no diplomas or
| certificates required_
|
| Be careful with this language, especially when you're involving
| politicians and the non-technical.
|
| The current atrocity of criminally negligent IT infrastructure
| right now is _mostly created and driven by_ people with diplomas,
| including from the most prestigious schools. (And a top HN story
| over the weekend was one of the most famous tech company execs,
| turned government advisor, advising students at Stanford to
| behave unethically, and then get enough money to pay lawyers to
| make the consequences go away.)
|
| And most of the certificates we do have are are individual
| certifications that are largely nonsense vendor training and
| lock-in, and these same people are then assembling and operating
| systems from the criminally negligent vendors. And our IT
| practices certifications are largely inadequate compliance
| theatre, to let people off the hook for actual sufficient
| competence.
|
| My best guess for how to start to fix this is to hold companies
| accountable. For example, CrowdStrike (not the worst offender,
| but recent example): treat it as negligence, hold them liable for
| all costs, which I'd guess might destroy the stock, and make
| C-suite and upper parts of the org chart fear prison time as a
| very serious investigation proceeds. I'd guess seeing that the
| game has changed would start to align investors and executives at
| other companies. What could follow next (with growing pains) is a
| big shakeup of the rest of the org chart and practices -- as
| companies figure out that they have to kill off all the culture
| of job-hopping, resume-driven-development, Leetcode fratbro
| culture, IT vendor shop fiefdoms, etc. I'd guess some companies
| will be wiped out as they flail around, since they'll still have
| too many people wired to play the old game, who will see no
| career option other than to try to fake it till they make it at
| the new, responsible game (ironically, and self-defeatingly,
| taking the company down with them).
| hinkley wrote:
| Put another way: it's far too easy and common for certification
| to encourage rote memorization. And only rote memorization. No
| higher order reasoning is imparted.
|
| Knowledge without reasoning is how you get mired in
| bureaucracy.
| neilv wrote:
| I think the larger problem is _alignment_.
|
| BS gatekeeping rituals and compliance-for-sale theatre are
| arguably just symptoms -- of companies and individuals not
| being aligned with developing trustworthy systems.
| WalterBright wrote:
| Punishment is not the answer, you'll just drive out of the
| industry lots of competent people. Punishment also means that
| nobody will admit to mistakes, will not fix mistakes (because
| that implies guilt), and the covering up of mistakes.
|
| Punishment for mistakes is what led to the Chernobyl disaster.
| neilv wrote:
| Flight safety works so well because the personnel are aligned
| with safety and professionalism, and the FAA has an important
| program in place to protect people from being punished for
| behaving professionally. And IIRC you're familiar with
| aircraft manufacturer alignment with safety.
|
| But I'm concerned about the entire field of software, which
| doesn't have that sense of responsibility, and I don't see
| how it would get it. However, software industry -- both
| companies and workers -- are guided almost entirely by money.
| To the point that it's often hard to explain to many people
| in HN discussions on why it would be good to behave in any
| other way than complete mercenary self interest. So I don't
| see any way to get alignment other than to link money to it.
| If people see that as punishment, so be it.
| WalterBright wrote:
| Every business is guided almost entirely by money. The
| purpose of a business is to make money.
|
| Organizations based on other incentives don't work, or work
| very poorly.
| pas wrote:
| in your later comment you mention alignment, but the reason is
| that there's an enormous market discontinuity between doing the
| "super-duper right thing" and doing the profitable thing ...
| due to network effect(s).
|
| we see competition in cloud/IaaS providers because they
| actually need to build datacenters and networks and so there's
| some price floor, but when it comes to "antivirus" CrowdStrike
| was able to corner the market basically, and downstream from
| them not a lot of organizations/clients/costumers can justify
| having actual independent hot-spare backups (or having special
| procedures for updating CS signatures by only allowing it to
| phone home on a test env first)
|
| the cultural symptoms you describe in so much detail are
| basically the froth (the economic inefficiencies afforded) on
| top of all the actual economic activity that's sloshing around
| various cost-benefit optimum points.
|
| and it's very hard to move away from this, because in general
| IT is standardized enough that any business that needs some
| kind of IT-as-a-service will be basically forced to pick based
| on cost, and will basically pick whatever others in their
| sector pick -- and even if there are multiple providers the
| will usually converge on the same technology (because it's
| software) -- thus this minimizes the financial risk for
| clients/customers/downstream, even if the actual
| global/systemic risk increases.
| _trampeltier wrote:
| I don't remember when and where exactly (and didn't found it in a
| quick search), but there was already an incident, where an
| automatic update failed. I think it was something with the
| country code, so it was a bit isolated and not all over the
| world.
| shermantanktop wrote:
| This article repeatedly cites the need for personnel to have
| diplomas, certificates, and other ceremonial bits of paper.
|
| This focus on paper qualification to mitigate risk seems a very
| European approach. Not saying it is wrong - it is just not
| emphasized as strongly elsewhere. And while it seems like a good
| fit for a slow-moving industry with high expectations of safety,
| the solar/wind world is not a slow-moving industry.
| g_p wrote:
| A good point - perhaps the focus is too heavy on paperwork or
| "measurable compliance".
|
| From experience in this sector though, I think the real issue
| is a lack of technical awareness and competency with enough
| breadth to extend into the "digital" domain - often products
| like these are developed by people from the "power" domain (who
| don't necessarily recognise off the top of their head that
| 512-bit RSA is a #badthing and not enough to use to protect
| aggregated energy systems that are controllable from a single
| location).
|
| Clearly formal diplomas/certificates are not needed for that -
| some practical hands-on knowledge and experience would help a
| lot there.
|
| When a product gets a network interface on it, or runs
| programmable firmware, we should hear discussions about A/B
| boot, signatures, key revocation, crypto agility to enable post
| quantum cryptography algorithms, etc. Instead, the focus will
| be on low-cost development of a mobile app, controlled via the
| lowest-possible-cost vendor server back-end API that gets the
| product shipped to market quickly.
|
| Let's not even go near the "embedded system" mindset of not
| patching and staying up to date - embedded systems are a good
| place to meet Linux 2.4 or 2.6, even today... Vendors ship
| whatever their CPU chipset vendor gives them as a board support
| package, generally as a "tossed over the wall" lump of code.
|
| I doubt many of these issues (which seem to be commercial/price
| driven) will be resolved through paperwork, as you say.
| WalterBright wrote:
| > It's also possible to install new software (firmware) on the
| inverters via the manufacturer, either automatically or manually.
|
| As always, the vulnerability of enabling remote updates. When
| will people learn? Updates should only be possible if there's a
| physical switch (not a software switch) on the device. If it's
| "off", no updates are possible.
|
| Isn't the most devastating attack vector remotely installing
| malware? With a hardware switch, none of that malware will
| survive a reboot of the device.
|
| I remember when hard disk drives came with a write-enable jumper.
| Then, once you've made a backup, the jumper is removed. Then it
| is impossible to accidentally or maliciously write over your
| precious backup.
| DoctorOetker wrote:
| That doesn't protect against supply chain attacks.
| WalterBright wrote:
| Neither does remote updating. But you'll still need physical
| access to the supply chain to compromise it, and that's not
| possible for some hacker in a basement.
| DoctorOetker wrote:
| I never claimed remote updating would prevent supply chain
| attacks.
|
| I was responding to:
|
| > With a hardware switch, none of that malware will survive
| a reboot of the device.
|
| A reboot of the inverter would not prevent a supply chain
| attack using MPPT measurement electronics for an optical
| backdoor channel.
| WalterBright wrote:
| So don't put the backdoor channel in without a physical
| switch.
| DoctorOetker wrote:
| Attackers don't ask permission.
|
| The hardware backdoor channel is present anyway because
| MPPT needs it.
|
| The software can abuse the measurements to listen for
| optically transmitted commands.
| DoctorOetker wrote:
| > But you'll still need physical access to the supply chain
| to compromise it, and that's not possible for some hacker
| in a basement.
|
| I forgot to respond to this sentence in the sibling
| response.
|
| Supply chain attacks can be executed by intermediaries of
| the supply chain, or by manufacturers themselves: develop
| the capability to deny a foreign nation its energy
| infrastructure. The manufacturer is not a hacker in a
| basement. Manufacturers can be pressured by their local
| gorvernments, militaries, 3 letter agencies, ...
|
| A precautionary principle would induce potential target
| nations to surreptitiously catalogue the inverter boards,
| sort them by most-GW serving type, and consider which
| control traces to cut to control the internal energy
| transfers in its inductors, capacitors, ... from a trusted
| parasite board. Just develop and test a few parasite boards
| for the most common inverters, and preferably have critical
| stock ready.
|
| The main value in inverters is the power switches,
| inductors, capacitors, ... it would be cheaper to reroute
| the control to a trusted controller in the event of a
| calamity. We would survive fine, but it will be a painful
| few days.
| veganmosfet wrote:
| Related to this topic, some research results about cybersecurity
| of solar inverters:
|
| https://github.com/veganmosfet/Balcony_in_the_cloud
|
| https://github.com/veganmosfet/SolarFlareSec
|
| https://github.com/veganmosfet/CyberEclipse
|
| https://github.com/veganmosfet/SecureWatt
|
| A big mess, but it's getting slowly better...
| shahzaibmushtaq wrote:
| The second figure explains a lot like everything.
|
| Cloud-based management platforms should not oversee inverters
| directly.
| kuon wrote:
| My installer put a solaredge inverter, it took some real efforts
| to keep it off the cloud while injecting the data in my grafana.
| I can do it because I am a network engineer, but it should be
| easier.
|
| Anyway, I agree that there should be a regulation that forbid
| remote management, and you can only consult data in a read only
| manner remotely (you could air gap the inverter with the internet
| gateway using a one way rs232 connection where the inverted just
| write continuously). And if grid operators need to be able to
| turn solar off, they should install relays controlled by their
| infrastructure.
| Derbasti wrote:
| There's a reason why I took my inverter offline after making sure
| that it was installed correctly. A cheap power meter now serves
| to measure my power generation instead.
| DoctorOetker wrote:
| Taking it offline doesn't protect against supply chain attacks
| in the form of built-in kill switches. A satellite could
| transmit signed instructions by modulating light below the
| noise floor, inverters must sense the voltage/current state of
| the PV panels anyway for MPPT to work.
|
| Only deep inspection of the silicon and code can improve the
| situation.
|
| Perhaps Western blocks could develop provably secure silicon IP
| and code, formally verified, and perform continuous random
| sampling on imported goods, including full multilayer silicon
| inspection; publish it for free and refuse to import products
| that don't cooperate.
| BenjiWiebe wrote:
| I'm curious about the feasibility of modulating light onto a
| solar panel. I feel it would not be feasible, except possibly
| onto a single panel at a time over a long time period. Just a
| gut feeling based off radio stuff (GPS).
| DoctorOetker wrote:
| GPS can provide the coherent reference, if you mean
| transmitting signal (say sound) while the panel is
| illuminated by the sun theres youtube videos of people
| doing that, with a laser pointer, but in sunlight and
| without information theoretic justified modulation scheme.
|
| Nothing prevents the satellite to transmit the commands at
| night, if that feels more convincing to you.
|
| Ask yourself what is the active area of a photodiode in
| your TV/... ? What is the active area of your light-bucket
| on a roof?
| mikewarot wrote:
| It irks me endlessly that we live in the worst timeline, where
| the computer equivalent of fuses and circuit breakers are almost
| completely unknown. Instead we trust code blindly.
|
| This results in almost all of the situations threads here
| address.
|
| In a better timeline, everyone has stable and secure OSs on all
| their devices, and the default is for everything to be locally
| networked, with optional monitoring from the outside via a data
| diode.
| DoctorOetker wrote:
| it's incredibly hard to implement a data diode for PV systems,
| enemy satellites can modulate light (like a TV remote, but
| lower baudrate to stay below the noise floor) and an inverter
| could decode it and respond accordingly.
|
| They measure the PV panels anyway for MPPT.
| isoprophlex wrote:
| If the west for some reason starts to vigorously argue with China
| over something, we're all completely fucked. They'll just tell
| our cheap EVs to forget how to brake, melt the firmware in our
| cellular towers/chips, and toggle our PV inverters off and on at
| a shitty time.
| leymed wrote:
| Reading through comments I saw a lot of comments confusing cloud
| security with electrical safety of a system. Electrical
| protections are completely separate from communication line/
| internet, has to be hard wired. As the size of plant/substation
| increases the automation and control system (again completely
| different thing from electrical protection) has its own internet
| system. Burning down substation, exploding transformer through
| solar panel is very very unrealistic.
|
| On top of that PVs installed at homes are insignificant to cause
| such troubles. As the size of installation increases, you will
| have different connection agreement and certain requirements. You
| can't install 15 MW and connect through inverters that are used
| at home, which is 100 kW at most. Even 15 MW is insignificant
| change for a grid.
| dpedu wrote:
| I can make my computer wildly vary the amount of power it is
| drawing by performing different things in software. Max out the
| CPU and GPU load and it will instantly change from drawing ~100
| watts to 500 or more.
|
| There have been plenty of botnets in the past. Some even in the
| millions of computers. If such a botnet decided to make every
| node's power draw fluctuate per above, wouldn't this cause the
| same type of problem? Is there a reason we've never seen this
| happen despite large enough networks of hacked machines existing?
| berkes wrote:
| The Netherlands (about which the article mainly is) has 8.4
| million households, let's presume they own average of one such
| PC you mention. A delta of 400W would mean a total consumption
| delta of 3.36GigaWatt. That's "peanuts" to cover.
|
| And that presumes an attacker can switch on/off all 8.4million
| computers in a small timeframe. 100% of them would need to be
| on, online and hacked.
|
| I don't think this is a realistic problem.
|
| Tesla F-ing up an OTA update that suddenly switches all
| charging Tesla's off, is probably a theoretical worse scenario.
| dpedu wrote:
| I don't doubt that that many watts is easy to cover -
| eventually. The problem is that it can be instantly turned on
| and off, whereas the grid takes time to shed load or add
| capacity.
|
| I found a figure on Wikipedia saying that the NL's 4.7GW
| worth of offshore wind capacity is 16% of their total
| electricity demand nationwide. 4.7/.16 = 30GW total, so this
| theorized computer load attack would represent about 10% of
| their grid's total capacity. Can their grid add and shed that
| much load that quickly? That's the part I doubt.
| lysecret wrote:
| Same is true for heat pumps.
| bww wrote:
| The author seems to imply, as if it were generally understood and
| accepted, that the reason nuclear reactors are heavily regulated
| is because they produce a lot of energy.
|
| Perhaps that's a component, but one really doesn't need to think
| about it too hard to identify better explanations for why this
| particular energy source is held to unusually high regulatory
| standards.
|
| I don't have an opinion as to whether other large-scale sources
| of energy should be held to similar standards, but to suggest
| that solar energy's failure modes are comparable to nuclear
| energy seems intentionally misleading.
| gwbas1c wrote:
| Here's the critical point:
|
| > In the Netherlands alone, these solar panels generate a power
| output equivalent to at least 25 medium sized nuclear power
| plants.
|
| > Because everything runs through the manufacturer, they are
| able to turn all panels on and off. Or install software on the
| inverters so that the wrong current flows into the grid. Now, a
| manufacturer won't do this intentionally, but it is easy enough
| to mess this up.
|
| > As an interim step, we might need to demand that control
| panels stick to providing pretty graphs, and make it impossible
| to remotely switch panels/loaders/batteries on or off.
|
| Basically, if a hacker were to make all batteries (or panels)
| suddenly switch between full discharge and full charge every
| second or so, it would tear down the electric grid. Voltage and
| frequency would swing rapidly, and whatever plants are riding
| load would struggle.
|
| This could create a massive power outage; but there is a huge
| risk that this could damage power plants and other
| infrastructure.
| trollied wrote:
| There's a great Practical Engineering video about the problems
| solar can cause to power grids:
| https://www.youtube.com/watch?v=7G4ipM2qjfw
| davedx wrote:
| Wait what. I don't know if my inverter does what they say. For
| one thing the vendor went bankrupt so there is no cloud dashboard
| anymore. For another there are hundreds of inverter vendors not
| one single one. And I am highly sceptical the basic dashboard
| showing solar generation has some sinister inverter backdoor
| killswitch when the article seems to provide _no evidence of
| such_? Seriously?
|
| Edit: did some research and apparently it varies - many modern
| inverters can be remotely controlled by manufacturers - if
| they're setup to allow it and are internet connected.
|
| The article is still sensationalist about the risks though
___________________________________________________________________
(page generated 2024-08-19 23:00 UTC)