[HN Gopher] Cooling system leak led to Victorian Big Battery fire
___________________________________________________________________
Cooling system leak led to Victorian Big Battery fire
Author : bjowen
Score : 107 points
Date : 2021-09-28 10:06 UTC (12 hours ago)
(HTM) web link (esv.vic.gov.au)
(TXT) w3m dump (esv.vic.gov.au)
| william2021 wrote:
| Short circuit that led to a fire in an electronic component
| bserge wrote:
| Should've used metal flanges instead of plastic heh
| spockz wrote:
| > There were further contributory factors with the Megapack in
| question being switched into an off-line service mode, resulting
| in the protection systems being inactive.
|
| A short in an offline device? That seems weird.
|
| > A 24-hour delay in connecting the batteries to the supervisory
| control and data acquisition (SCADA) system also meant there was
| no active monitoring of the Megapack alarms.
|
| So a new pack was installed and connected but not monitored? What
| is the rationale behind that? Failures shouldn't happen to new
| devices?
| michaelt wrote:
| _> So a new pack was installed and connected but not monitored?
| What is the rationale behind that?_
|
| Any time a house is built, there's some time between the
| ceiling going up and the fire alarm installation being
| completed.
|
| Nothing wrong with that - if it's done properly and carefully.
| jlg23 wrote:
| Department #1 does the hardware setup, signs off their forms
| and the next day department #2 integrates the new hardware into
| monitoring?
| tomalpha wrote:
| An extract from the summary page suggests a few things went
| wrong, but it seems like the fire was caused by liquid and
| electricity mixing:
|
| > ESV found a Megapack cooling system leak caused a short circuit
| resulting in overheating that led to a fire in a nearby battery
| compartment, which consequently damaged two Megapacks.
|
| > There were further contributory factors with the Megapack in
| question being switched into an off-line service mode, resulting
| in the protection systems being inactive.
|
| > A 24-hour delay in connecting the batteries to the supervisory
| control and data acquisition (SCADA) system also meant there was
| no active monitoring of the Megapack alarms.
| [deleted]
| sitkack wrote:
| I would assume that each megapack would have zigbee for
| telemetry which should be always transmitting, even if the pack
| were in the off state.
|
| And the cooling system should be continuously monitored for
| faults. Not just inspected after filling.
| WJW wrote:
| That it's hardware instead of software doesn't mean it can't
| be hacked together.
| sitkack wrote:
| Thermals are the number one aspect of battery engineering.
| For something that is to run under load in the Victorian
| heat, even more so.
|
| This isn't some weekend hackathon contest, Tesla is
| supposed to have the worlds best engineering.
|
| https://en.wikipedia.org/wiki/Tesla_Megapack
| WJW wrote:
| Why would Tesla have the "worlds best" engineering? I
| don't think I have ever seen anyone ever claim that
| except for perhaps Tesla PR representatives. How would
| you even measure "best" in such a field? Cheapest? Best
| quality? Best value-for-money? Safest? Highest power
| density?
| aaron695 wrote:
| There are 212 megapacks.
|
| I would imagine this would be really fucking hard.
|
| Most larger, more expensive engineering units don't have
| this.
|
| I'm open to the idea, if it exists or there are similar
| setups then that would show it's possible.
|
| Else I'd guess this is like creating Facebook in a weekend.
| Easy to say. Regulations, security, false positives, unique
| software.
| smcleod wrote:
| > "no active monitoring"
|
| What the actual... how is something like this even allowed to
| be operational without monitoring?
| krisoft wrote:
| It wasn't operational, it was being comissioned.
|
| You folks are acting as if some huge crime against humanity
| happened here. In reality nobody got hurt, the company who
| messed up will pay to put it right. They already indicated
| (in the linked report) that they learned from the incident
| and changed multiple things in their systems and procedures
| as a result.
| wolrah wrote:
| > It wasn't operational, it was being comissioned.
|
| You're telling me that you don't see a problem with the
| idea of starting up a bleeding edge system intended to
| store incredible amounts of energy without the monitoring
| systems connected?
|
| In general if you don't have a way to monitor an
| industrial-scale system you go in to your fail-safe state.
| krisoft wrote:
| > intended to store incredible amounts of energy
|
| Each of these Megapacks can contain 1.5% of the energy
| contained in a petrol station. :) Let's not get ahead of
| ourselves, these are realy nice and big batteries. That
| being said the energies stored in them are far from
| incredible.
|
| These boxes were by all accounts manufactured in Nevada.
| They have been moved all the way to Victoria, lowered to
| the ground and hooked to the site's wiring. They were not
| "charged" yet. The next step as part of the comissioning
| was to hook them up to the monitoring system before they
| are checked out.
|
| No matter when you hook them up to be monitored there is
| a step just before it when they are not monitored yet.
|
| > In general if you don't have a way to monitor an
| industrial-scale system you go in to your fail-safe
| state.
|
| That is indeed a great idea. And if you read the report
| you can see that after they hooked up each box they put
| them manually into "maintenance" mode. This mode de-
| energises all systems in the box. It totaly makes sense
| that they thought this procedure is the correct one to
| reach the fail-safe state you mention. Turns out they
| were wrong. They learned this now and changed their
| procedures.
| nomercy400 wrote:
| This. Lessons learned, improvements implemented.
| throwaway0a5e wrote:
| Nobody ever got internet virtue points by keeping their
| cool. You gotta be Outraged(TM).
| smcleod wrote:
| It's a live (active) power source even if it's being
| commissioned. I don't think it's a crime against humanity
| but it sure created pollution and waste.
| londons_explore wrote:
| I am very surprised in this conclusion... Normally if you have
| two high voltage wires being immersed in water, a large current
| flows, vaporising the water, and then the current stops when
| all the water has been evaporated.
|
| For the current to flow for long enough to heat the battery
| packs to cause a fire, there must have been a jet of water
| flowing for a long period of time. And no fusible links between
| that place and the battery.
|
| It really seems like either very bad luck, or bad system
| design, or both.
| mannykannot wrote:
| The article puts it this way: "ESV found a Megapack cooling
| system leak caused a short circuit resulting in overheating
| that led to a fire in a nearby battery compartment." Nothing
| here implies that the damaging heat generation occurred at
| the location of the short circuit. It could have been at any
| point in the new current path created by the short circuit,
| depending on its resistance and the share of the overload
| current it carried.
| michaelt wrote:
| A cooling system for something that's on fire sounds like
| exactly where there _should_ be a jet of water flowing for a
| long period of time :)
| algo_trader wrote:
| Off topic: can we do a floating (mega-)mega-pack, and use
| passive water cooling? This should help costs, efficiency AND
| prolong life time
|
| Of course issues with sweet vs salt water
| londons_explore wrote:
| Sure you can do it, but it won't be reliable or cost
| effective.
|
| Passive water cooling in natural water tends to lead to
| seaweed and all kinds of sea creatures blocking up all heat
| exchange surfaces.
|
| In the case of a megapack, cooling isn't critical to safety
| - as long as you're happy to stop using a battery when too
| hot, it won't get hotter. The reason to cool is is so you
| can keep using the battery.
| Gepsens wrote:
| offline monitoring is what caused tchernobyl, when will ppl
| learn ?
| nathancahill wrote:
| Were you going for Chernobyl?
| jjoonathan wrote:
| Were you going for tche[?]rnob[?]y[?]l?
| cube00 wrote:
| _> Shorter connection times to the SCADA system to help alert
| Tesla with specific alarms._
|
| Interesting that even after installation Tesla are still involved
| in operationally managing the megapacks.
| toomuchtodo wrote:
| Tesla's model is they both sell the Megapack system and then
| orchestrate it over its lifetime using their Autobidder
| platform. The owner is responsible for financing, integration
| with on site generation, and working with the local grid
| operator.
| aetherspawn wrote:
| > "A new battery module isolation loss alarm has been added."
|
| This is the worst fault possible in an EV 101 scenario right here
| (touch metal -> instant death), so I'm surprised that with all
| Tesla's experience, they didn't consider an isolation leak as a
| serious fault. And I'm surprised that they allow the battery
| monitoring system to be disabled during servicing, especially the
| IMD (insulation monitoring device).
| londons_explore wrote:
| I am constantly surprised how many systems have no isolation
| when it would be easy and cheap to add.
|
| For example, typical solar inverter systems don't have
| isolation between the AC and DC sides. That means any chafed
| wire can be deadly. They do at least have leakage detection,
| but set at a 300mA level, which probably won't save your life.
|
| Considering solar inverters already uses high frequency
| switching inside the per-string MPPT boost converter, it would
| have cost mere cents to put two windings on that inductor
| rather than one and get isolation with no downside.
|
| The cynic in me says manufacturers don't add safety features
| not required by law, even if zero cost, because in the future
| laws might be updated to require that safety feature,
| preventing reuse of hardware already sold, which is good for
| business.
| Filligree wrote:
| > For example, typical solar inverter systems don't have
| isolation between the AC and DC sides. That means any chafed
| wire can be deadly. They do at least have leakage detection,
| but set at a 300mA level, which probably won't save your
| life.
|
| This is very yikes. I just installed such a system.
|
| Do you know any way to test if that's a problem? And is it
| possible to fix without opening the inverter?
| londons_explore wrote:
| It's normally set that high by design because the solar
| panels have substantial capacitance to ground (they are
| large flat conductors sat just a few cm off the ground
| after all). The capacitance to ground wouldn't be an issue
| if not for the fact the inverter is non-isolated. That
| means current can flow to/from the grid to charge and
| discharge that capacitor, which is indistinguishable from
| AC leakage.
|
| Some inverters will allow you to lower the leakage
| threshold (usually with settings in a service menu), but
| then you're limited to just attaching 3 or so panels or
| getting false isolation trips.
| Filligree wrote:
| In my case the panels are, in fact, mounted on wooden
| brackets a good 1.5 meters off the ground. On the other
| hand, there's 20 m^2 of them.
| Reason077 wrote:
| Huh? 300 mA should barely give you a tingle unless you're
| talking about some _very_ high voltages. What am I missing
| here?
| throwaway9870 wrote:
| A basic knowledge of electricity. 100-200mA will kill you.
|
| https://academic.oup.com/ptj/article-
| abstract/46/9/968/46379...
|
| I have heard people say "that can't be right, because a car
| battery can source 100+ amps and touching those terminals
| doesn't kill me!" The reason is because you have a high
| enough impedance that 100mA doesn't flow from a 12V source
| through your body.
| Reason077 wrote:
| Embarrassingly, I have managed to shock myself several
| times during my life on 240V mains. Once quite recently
| while working on a light fitting at a friend's flat and
| not realising that someone had flipped the circuit back
| on at the circuit breaker!
|
| That shock was certainly quite unpleasant, but despite
| touching my sweaty skin against live wires, it seems to
| have been far from fatal.
|
| Surely the dangerous situations are live-to-earth shocks,
| which could potentially run across your whole body
| (rather than just a hand or finger, as in my case). But
| that's why we have RCD/GFCI protection, right? Those will
| trip with tiny fault currents, certainly much less than
| 300 mA.
|
| Secondly, aren't "Type B" RCDs required on solar
| inverters? Those will detect and trip on DC fault
| currents as low as 30 mA, not 300!
| nielsbot wrote:
| you need "Lock Out Tag Out" lol
|
| https://en.wikipedia.org/wiki/Lockout-tagout
| james400 wrote:
| "It's not the voltage that kills you, it's the amps!"
| Wouldn't power (W) (or joules) be a better way to phrase
| it? Because as you said, you can't have 100mA flowing
| through your heart unless you ALSO have stiff enough
| source to actually supply that current - which implies a
| high compliance voltage, can't escape ohms law.
| joncrocks wrote:
| "It's the volts that jolt, but the mills (mA) that kills"
| - The phrase a schoolteacher used when I was a lad.
| jrockway wrote:
| V = I * R
| jhgb wrote:
| The voltage across the heart will be rather bounded if
| you fix the current at 100mA, though. Hearts will not
| differ significantly from each other.
| Johnny555 wrote:
| Not really, whether it takes 50V with wet skin or 500V
| with dry skin, it's the current that kills you, not the
| watts
| kortex wrote:
| No, it's not the power, either, as low resistant paths
| can be extremely dangerous (such as wet skin). Really,
| it's a 2 dimensional space that makes for a less catchy
| aphorism. "It's not the voltage that kills you, it's the
| combination of voltage and conductivity".
|
| Honestly I kinda hate that saying. It's like "it's not
| the fall that kills you, it's the sudden decelleration on
| impact." For most situations people encounter, it's high
| voltage sources with more than enough power which are
| dangerous.
| jfrunyon wrote:
| "The combination of voltage and conductivity" is ...
| current, no?
| jackweirdy wrote:
| Here is medhi sadaghdar (electro boom) putting 11v at
| 150A through his tongue
| https://www.youtube.com/watch?v=XDf2nhfxVzg
| aaronmdjones wrote:
| GP mentioned impedance, not resistance. You won't feel
| anything touching the 12/24V terminals in your car
| because they're DC. Your skin has a high resistance, but
| when faced with AC, has a low impedance. This makes AC
| much more dangerous, as you conduct 120V AC much better
| than 120V DC.
|
| Even 50mA will likely kill you, as outlined on the "death
| graph" as we electricians call it. This is independent of
| voltage.
|
| https://upload.wikimedia.org/wikipedia/commons/7/7f/IEC_T
| S_6...
|
| Blue is harmless and imperceptible, green is harmless,
| yellow is harmful, red is fibrillation.
|
| Edit: This illustrates why 30mA (or less) RCD protection
| is common (and required) in various jurisdictions,
| depending on the kind of circuits you're serving.
| Reason077 wrote:
| Right, and in particular, "Type B" RCD protection is
| required for solar inverters in various jurisdictions.
| These provide protection against DC fault currents.
| jhgb wrote:
| > This makes AC much more dangerous, as you conduct 120V
| AC much better than 120V DC.
|
| But at higher frequencies, you should experience the skin
| effect, right?
| chockablock wrote:
| > A domestic power supply voltage (110 or 230 V), 50 or 60
| Hz alternating current (AC) through the chest for a
| fraction of a second may induce ventricular fibrillation at
| currents as low as 30 milliamperes (mA).With direct current
| (DC), 300 to 500 mA is required.
|
| https://en.m.wikipedia.org/wiki/Electrical_injury
| catmanjan wrote:
| So you'd have to have a wire pressed against your chest
| with your back on the ground... Unlikely but yes possible
| hazard when working on solar panels
| jfrunyon wrote:
| Think about the path the electricity would have to follow
| between, say, an arm and a leg, or even both arms.
| rainbowzootsuit wrote:
| These, and the GFCI outlets/breakers are calibrated
| against injury for adults. The thresholds are lower for
| children.
|
| Mike Holt is an excellent instructor on the NEC and has
| quite a few videos that speak about this topic.
|
| https://m.youtube.com/user/MikeHoltNEC
| shadowpho wrote:
| I spent a bit working on various power supplies and sadly
| isolation is non-trivial nor cheap. Adding isolation
| (transformer vs inductor) requires changing control signals,
| control mechanism, size. It costs far more then mere cents,
| and loses some efficiency.
| baybal2 wrote:
| Aren't you are mistaking galvanic isolation with insulation?
| aetherspawn wrote:
| Galvanic isolation is useful because it means that you have
| to touch 2 specific points on the affected unit
| simultaneously to die, rather than 1 point. Galvanically
| isolated systems can detect leaks on both the positive and
| negative sides by using a dual channel IMD. Any insulation
| issue is bad, but this method of detection can usually
| detect it before it's able to instantly kill someone per-
| se. Electric vehicles are an example of a galvanically
| isolated system (from earth) with dual channel IMD, hence
| you can have massive leaks such as 8-20mA before it becomes
| a safety issue.
| baybal2 wrote:
| Yes..., but what difference it makes to this fire?
|
| Are you working in EE?
| aetherspawn wrote:
| Galvanic isolation probably wouldn't have prevented this
| fire, it sounds like the cells just shorted together.
| They would have had to short over several cells, since a
| non-trivial voltage is required to break down coolant, so
| having maintenance contactors to break the pack up into
| 20V or so segments may have prevented this fire.
|
| Using a mineral oil type coolant that is not electrically
| conductive (such as used in Tesla vehicles) would have
| also helped to prevent this fire.
|
| The batteries should be galvanically isolated from the
| casing so that if liquid causes a current flow from any
| point in the pack, then the maintenance contactors can be
| opened and the opposite side current path for the leak is
| interrupted. We can only assume that they are already
| (galvanically isolated) since anything else would be
| bizarre.
| [deleted]
| dhsysusbsjsi wrote:
| Reading this it's a classic systems failure: multiple defensive
| barriers gone wrong through poor safety planning. It's
| reminiscent of the Victorian ESSO gas plant accident
| https://en.wikipedia.org/wiki/Esso_Longford_gas_explosion
| krisoft wrote:
| In what way is it reminding you of that incident?
|
| The paralels I can see:
|
| - both involve energy installations.
|
| - both happened in the same state.
|
| The differences I can see:
|
| - one involves a gas utility the other an electric one
|
| - one has killed and hurt workers while the other did not
| killed nor injured anyone.
|
| - one happened after several years of operation, the other
| happened during installation.
|
| - one left the state without energy service for 20 days, the
| other did not affect service at all.
|
| Don't get me wrong. I love reading about how complex things go
| wrong, one of my favourite pastimes. So I thank you for your
| link. It just doesn't seem that similar to me. Not sure you
| would be citing the same case for example have they not
| happened in the same state.
|
| If I have to choose a failure which the Big Battery fire
| reminds me of I would rather choose the Florida International
| University pedestrian bridge collapse. Why? It seems these
| batteries had adequate safety functions and monitoring planned
| in for their operation. Just these systems were not activated
| during the installation in the right order. Similarly with the
| bridge it seems they did calculations with the full bridge in
| place, but nobody seems to have checked if the intermediate
| steps during the construction will stay up too.
|
| Another similarility (to my untrained eyes) is the aim to
| install a system fast. The bridge project wanted to be a
| flagship for accelerated bridge building, and the big selling
| point of these Megapack installations is how fast they can be
| comissioned. With the bridge project we know that this demand
| for speediness was a contributing factor. Was it maybe also
| with the Big Battery fire too?
| throwaway0a5e wrote:
| >In what way is it reminding you of that incident?
|
| It's just in group signaling. "Hey this reminds me of
| <vaguely related thing>" is a more polite way of saying "look
| I know about X, I'm like you, gimme dat virtue points"
|
| You see this on literally every virtue points (i.e. up-vote)
| based form of social media that's big enough that any given
| participant can blend in with the crowd.
|
| The behavior is endemic these days you can't even post a
| picture of a rotten and falling down deck on the internet
| these days without some jerk derailing everything by dropping
| a link to the wikipedia page on the hyatt regency and then
| several more riding their coattails with quotes copied from
| the page.
|
| He probably doesn't even consciously realize he's doing it.
| walrus01 wrote:
| for people who might say 'oh, but these batteries are so
| dangerous!'
|
| How many cooling system leaks (radiator+water pump related
| plumbing) result in vehicle engine fires per year?
|
| I imagine the largest insurance companies have reasonably good
| data on this as a cause of total-loss of a vehicle.
|
| How many times have you seen in person, or seen a photo of a
| burned out RV that somebody pushed too hard up a mountain pass
| without keeping an eye on the engine temperature?
| pengaru wrote:
| Overheating ICEs blow head gaskets and warp cylinder heads,
| they don't generally start fires.
|
| Fuel leaks and/or exhaust system failures, especially on
| turbocharged vehicles however...
| driverdan wrote:
| > How many cooling system leaks (radiator+water pump related
| plumbing) result in vehicle engine fires per year?
|
| Almost none. Overheating is highly unlikely to cause a fire.
| Fuel leaks cause fires.
| DoingIsLearning wrote:
| Read the doc, this has nothing to do with lithium batteries as
| a component. This was a System Requirements/Hazard Analysis
| failure.
|
| The fault lies square with Tesla. No amount of RV analogies can
| dilute this.
| walrus01 wrote:
| The point was that plenty of other energy dense things with
| radiator-based liquid cooling loops lack proper monitoring
| systems.
|
| Not excusing Tesla but the actual number of heat producing
| things that catch on fire every year (in general) because of
| cooling pump/cooling loop failure is quite a lot. In many
| categories of equipment.
| phaedrus wrote:
| You have a gross misconception of the relative temperatures of
| components in internal combustion engines, and you're using a
| faulty conclusion based on the same misunderstanding as
| supposed supporting evidence.
|
| I assure you an RV or other gas/diesel vehicle does NOT
| suddenly go "pop!" and produce an engine fire when the needle
| reaches the right hand side of the temperature gauge.
|
| The temperature gauge warns you when the coolant is at risk of
| boiling; it's not a warning that the engine is about to
| spontaneously combust.
| walrus01 wrote:
| right, that's why every burned out at the roadside RV I've
| seen in my lifetime (probably more than a dozen now) was on
| the upward slope of a high mountain pass, or at the peak...
|
| one of the points I was trying to make is that people don't
| maintain liquid cooling loops in general, and that's one of
| the more common instances of it.
|
| do you live somewhere flat?
|
| go ask the towtruck drivers who work the coquihalla highway
| in BC how many vehicle fires they see every year, and _where_
| those vehicle fires occur.
| nraynaud wrote:
| how much would you ask to be paid to pressure test the coolant
| system of a giant bomb in situ?
| baybal2 wrote:
| One advantage of LFP batteries is that they can survive much
| higher temperatures without degradation.
|
| If you can make battery packs air cooled, that's a big, big
| weight saver.
| userbinator wrote:
| From the title I first thought this would be about a battery fire
| that happened in the late 1800s.
| bjowen wrote:
| In a much cooler reality, it is, and it's the other Tesla.
| RicoElectrico wrote:
| So, they'll have to take note from their nuclear colleagues'
| playbook? About passive safety.
| cotillion wrote:
| Well, the conclusions seem to suggest they have passive safety.
|
| "The affected Megapacks failed safely despite total loss."
| epistasis wrote:
| I think that people understand the level of engineering and
| technology that goes into batteries and their designs these
| days. There's a ton of talent, a ton of continuous
| innovation, it in the end it too often gets relegated to a
| black box for energy in and energy out. Which is in some ways
| a testament to the quality of the engineering.
| karlkloss wrote:
| Why? The factory didn't burn down, and there was no widespread
| radioactive contamination.
|
| It's a battery pack, not a stash of U235 above the critical
| mass. Every recycling facility is more dangerous.
| ashtonkem wrote:
| Because fires are bad, and they can kill workers and shut
| down facilities?
___________________________________________________________________
(page generated 2021-09-28 23:02 UTC)