[HN Gopher] Hacker increased chemical level at Oldsmar's city wa...
       ___________________________________________________________________
        
       Hacker increased chemical level at Oldsmar's city water system,
       sheriff says
        
       Author : bschne
       Score  : 69 points
       Date   : 2021-02-08 21:16 UTC (1 hours ago)
        
 (HTM) web link (www.wtsp.com)
 (TXT) w3m dump (www.wtsp.com)
        
       | dialamac wrote:
       | Love that it was a "hacker" rather than a bored teenager or some
       | casual idiot, or even a disgruntled former or current employee.
       | Invoking the hacker term I guess is supposed to make it absolve
       | gross negligence in even the most basic security practices. It
       | makes me sad to think this spin actually works.
        
         | amenghra wrote:
         | I thought we had all agreed that "Chinese hackers" was the
         | correct term to use /s.
        
           | nix23 wrote:
           | You forgot Russians/N.Koreans or Iranians...it's a flavor
           | question.
        
         | ogre_codes wrote:
         | This was my gut instinct too. Some high school kid or just a
         | random hacker who stumbled across it is just as likely as
         | anything else.
        
           | gowld wrote:
           | A random hacker is a hacker.
        
         | justaman wrote:
         | If it calls attention to vulnerable infrastructure in the US,
         | good. We need to put a lot more tax dollars into securing these
         | systems.
        
           | toomuchtodo wrote:
           | Only if those dollars are used effectively.
           | 
           | If you're just checking a checkbox and not actually securing
           | the system, you're no better off.
        
           | brendoelfrendo wrote:
           | Right, but the concern is that if we focus too much on the
           | "hacker" and not enough on the vulnerable infrastructure, we
           | may spend all our tax dollars chasing computer criminals
           | instead of preventing computer crime. As a general rule, I
           | don't like blaming the victims of a crime for falling
           | victim... but the real victims here are the people downstream
           | of the water supply, and we shouldn't absolve industry or
           | infrastructure operators of negligence because some scary
           | hacker attacked them.
        
         | neolog wrote:
         | What does the age or casualness of the hacker have to do with
         | anything?
        
       | _Microft wrote:
       | The press conference can be found here:
       | 
       | https://www.youtube.com/watch?v=MkXDSOgLQ6M
        
       | welder wrote:
       | I have an under-sink Reverse Osmosis water filter. If this wasn't
       | caught I wonder if the RO filter would have removed the sodium
       | hydroxide or not.
        
         | leesalminen wrote:
         | Efficacy of RO systems is highly dependent on incoming water
         | pressure and temperature. I got a professional water test done
         | on our RO system and was surprised at how much remained in our
         | (hard well) water. It turned out that our incoming water temp
         | was far too low for the system to reach peak efficiency.
        
           | welder wrote:
           | The built-in TDS sensor shows 004 after filtering, but the
           | unfiltered tap water is good here so that probably helps.
        
           | Cerium wrote:
           | Another aspect is TDS creep. The membrane only reaches the
           | specified rejection rate after a few minutes of use. If they
           | under sink RO system is used frequently for small amounts of
           | water it can cause frequent cycling of the RO system which
           | will reduce the water quality produced.
           | 
           | My RO system can take the 450 TDS tap water down to about 30
           | under my normal use. If I close off the tank and run the
           | water for about 10 minutes it will get down to about 20.
        
             | welder wrote:
             | Mine's tankless (Waterdrop brand), is TDS creep still an
             | issue?
        
       | hoppla wrote:
       | Access Denied
       | 
       | You don't have permission to access
       | "http://www.wtsp.com/article/news/local/pinellascounty/pinell..."
       | on this server. Reference #18.8700561.1612822078.194a07f6
        
         | llacb47 wrote:
         | That's because Tegna is lazy and geoblocks EU visitors from
         | their sites to avoid GDPR compliance.
        
         | codetrotter wrote:
         | Maybe the hacker deleted the article too :^)
        
         | amenghra wrote:
         | https://twitter.com/zackwhittaker/status/1358867424171425799
         | has a link to a Youtube video if you want to watch the press
         | conference.
        
         | niea_11 wrote:
         | Probably because you're accessing the website from the EU and
         | they don't want to bother with the GDPR.
        
         | dariusj18 wrote:
         | That looks like an Akamai error
        
         | MandieD wrote:
         | This link works for me (EU):
         | https://www.tampabay.com/news/pinellas/2021/02/08/someone-tr...
        
       | macawfish wrote:
       | One thought that comes to mind: how do we know this same
       | individual hasn't successfully gotten away with this elsewhere?
       | How long would it take for people to report symptoms?
        
       | amenghra wrote:
       | from
       | https://twitter.com/zackwhittaker/status/1358868187656388611:
       | I can't immediately verify the veracity of the claims made by the
       | sheriff but,         the fact that the authorities *set up* a
       | public-facing and/or remotely         accessible system that
       | allowed someone to change the water chemical levels is by
       | far the bigger issue here.
        
         | jcranmer wrote:
         | I worked at a water treatment facility for a few summers, and
         | the SCADA system there was on a physically separated network.
         | Actually, there were two SCADA networks, one for each of the
         | plants, with the distribution system (the water towers and
         | pumping stations randomly scattered throughout the service
         | area) attached to one of those networks. I don't know how
         | secure those remote links were, but I suspect they were the
         | easiest ingress into the network.
         | 
         | A couple computers did bridge the two networks, but (IIRC) they
         | were simple embedded systems doing read-only access (for
         | compiling reports). I know when they did a pen-test, the pen-
         | tester could compromise most of the corporate network
         | (including service accounts), but they couldn't punch through
         | to the SCADA systems.
        
         | colechristensen wrote:
         | There has been for quite a while a big concern that industrial
         | control systems are accessible, often poorly hardened (and by
         | that I mean to the extent of having default passwords), and
         | quite vulnerable to attack.
         | 
         | The only thing surprising about this is that we don't hear
         | about it tenfold more.
        
           | laurowyn wrote:
           | Absolutely.
           | 
           | Meanwhile, we live in a world where VPNs are sold to the
           | casual user while critical systems are left on internet
           | facing networks.
           | 
           | I've never understood why, if these critical systems need
           | remote access, it's not all done through a VPN of some sort.
           | VPNs are not infallible, but it significantly increases the
           | bar for entry from script kiddie to nation state real quick
           | (depending on choice of crypt), while choosing a well
           | supported implementation ensures long term bug fixes and
           | security patches.
        
           | JPKab wrote:
           | After meeting enough SAP consultants in the ICS space, all I
           | can say is I'm shocked it doesn't happen every day.
        
           | 2bitencryption wrote:
           | > The only thing surprising about this is that we don't hear
           | about it tenfold more.
           | 
           | If you're someone who stands to gain from disrupting a
           | nation's infrastructure... you don't tip your hand until it
           | most benefits you.
           | 
           | If it really is the case that large parts of the
           | infrastructure are very unsecure, expect to hear about it all
           | at once, instead of little by little.
        
             | munk-a wrote:
             | Water seems like a really weird system to sabotage though -
             | power can bring businesses offline in a serious way but a
             | city reservoir likely isn't supplying any businesses with a
             | real need of water for any sorts of industrial needs...
             | It's more of an inconvenience. Messing with chemical
             | balances in particular seems like a prank or someone really
             | twisted trying to give a bunch of folks long term health
             | complications.
        
               | devonkim wrote:
               | Sometimes attacks are probes and discoveries meant to
               | determine or validate efficacy of a set of attack vectors
               | including but not limited to human assets. Other uses are
               | for distractions from other efforts. And yeah, sometimes
               | they're pranks. It's not clear with the given facts
               | what's really going on.
        
         | snypher wrote:
         | Seems to me that the real issue is lack of security, not the
         | fact this system exists at all. Eg Every cell tower has remote
         | access protocol and we rarely hear about those being hacked.
        
           | amenghra wrote:
           | There's probably 100x more cell towers than there are water
           | plants. The impact of hacking a cell tower isn't direct loss
           | of human life (granted, knowing off a large number of cell
           | towers would be very disruptive). The answer to the question
           | "should it be online" and "how much $$$ should we spend
           | securing it" is going to be different in these two cases.
        
             | chongli wrote:
             | _impact of hacking a cell tower isn 't direct loss of human
             | life_
             | 
             | Not a direct loss, but plenty of opportunity for indirect
             | loss. Disrupting emergency systems is the first that comes
             | to mind. Covert hacking and surveillance could also be used
             | for assassination plots.
        
             | munk-a wrote:
             | I think there's also a fair question of "ownership of
             | damages" here - cities get sold water treatment management
             | systems and want them online as cheaply as possible - city
             | councils end up owning the mistakes in misconfiguration but
             | companies selling the systems are incentivized to make
             | those default bad configurations possible - even while, in
             | bold lettering, mentioning that you should not use the
             | default authentication.
             | 
             | Cell towers are a really integral part of carrier's
             | business - I'm not certain whether most are owned by
             | providers or other companies, but either way the folks that
             | put the tower up owe the customer (be it a phone user, a
             | phone provider or some subcontactor of the provider) an
             | explanation and pay the costs of bad configuration... I'd
             | also assume that making sure these towers stay up is
             | someone's fulltime job (likely multiple people) - while
             | there won't be an employee constantly monitoring city water
             | systems since it would take so little of a single person's
             | time.
        
               | Tyrek wrote:
               | I'm not sure I agree that this is /wrong/ per se - the
               | issue arises from the city council's disinterest / lack
               | of expertise (which itself comes from disinterest) in
               | these systems. If the issues are disclosed clearly, and
               | the city council continues to sign off on the
               | implementation (due to disinterest, cost pressure,
               | whatever) without consulting knowledgeable third parties,
               | then it's only realistic that the blame falls on the
               | ultimate decision-maker (in this case, the city council).
        
       | beervirus wrote:
       | If my vague memories of high school chemistry serve me correctly,
       | then 11,100 ppm is 0.2775M, which would have a pH of about 13.4.
       | That's definitely not something I'd want to drink.
        
         | gowld wrote:
         | 11K ppm is 1% of the entire water supply. I doubt the plant had
         | that much lye in stock.
        
           | beervirus wrote:
           | But they probably had enough lye in stock to _start_
           | producing water at that concentration.
        
       | achillean wrote:
       | Internet-accessble industrial control systems have been a problem
       | for many years now. It's a documented issue but it's difficult to
       | fix for a variety of reasons:
       | 
       | 1. Difficult to identify the owner: a lot of the devices are on
       | mobile networks that don't point to an obvious owner.
       | 
       | 2. Unknown criticality: is it a demo system or something used in
       | production?
       | 
       | 3. Security budget: lots of smaller utilities don't have a budget
       | for buying cyber security products.
       | 
       | 4. Uneducated vendor: sometimes the vendors of the device give
       | very bad advice (https://blog.shodan.io/why-control-systems-are-
       | on-the-intern...)
       | 
       | That being said, based on the numbers in Shodan the situation has
       | improved over the past decade. And there's been a large
       | resurgence of startups in the ICS space. Here's a current view of
       | exposed industrial devices on the Internet:
       | 
       | https://beta.shodan.io/search/report?query=tag%3Aics&title=I...
       | 
       | I've written/ presented on the issue a few times:
       | 
       | https://blog.shodan.io/taking-things-offline-is-hard/
       | 
       | https://blog.shodan.io/trends-in-internet-exposure/
       | 
       | https://exposure.shodan.io/#/
        
         | the_only_law wrote:
         | There's probably a dumb reason I'm not thinking of, but why
         | does the US have such a higher count than other large,
         | industrialized nations?
        
           | rootsudo wrote:
           | Underpaid IT/Infosec. People conflate IT and Infosec, once
           | it's on an Govt payroll for billing purposes, no one touches
           | the system if it's on a network provider, and not internal.
           | If not internal, it won't show up on audits, most IT
           | departments deal with a Windows Domain/Network and that's
           | most locked down, but if it doesn't share a true connection
           | physically, it's exempted from most audits.
           | 
           | The question is, why are the telecom providers allowing this,
           | but there's also alot of legacy stuff they don't want to
           | touch as it may violate the terms/contract and the bandwidth
           | isn't the issue, so telecoms largely ignore it as they're
           | just a bridge/
        
           | achillean wrote:
           | Some mobile networks in the US will give you a public IP
           | whereas in most other countries they do Carrier-NAT. You can
           | get a better sense of it when looking at the IP space owners
           | for the devices:
           | 
           | https://beta.shodan.io/search/facet?query=tag%3Aics+country%.
           | ..
        
         | giantg2 wrote:
         | Even non-connected systems can be a problem. Stuxnet was an
         | example. But I think the main point is that owners of those
         | systems think they are protected just by being disconnected.
        
       | ph4 wrote:
       | Why would 11,100 be recognized as a valid value to begin with?
        
         | welder wrote:
         | Yea, with the Mac key repeating issue that value could get
         | input by accident.
         | 
         | https://apple.stackexchange.com/questions/293523/single-keyp...
         | 
         | https://www.theverge.com/2020/5/4/21246223/macbook-keyboard-...
        
         | rootsudo wrote:
         | Vs, why was this remote facing, and why don't they have a
         | definitive answer on if it's an USA or non USA ip address.
         | 
         | Sounds like no logs, probably showed up on shodan and someone
         | wanted to have fun/many people did.
        
           | mediocregopher wrote:
           | The geolocation of the IP isn't all that useful, it could be
           | a VPN or an owned machine.
        
             | rootsudo wrote:
             | Disagree, if it was USA, it is easily possible to enforce a
             | warrant and maybe you're lucky it's residential.
             | 
             | If it was an VPN, you know it's a more competent person,
             | org, and most VPN's also, keep logs.
        
         | nix23 wrote:
         | Maybe to have a Cleaning Cycle once a year? But yeah there is a
         | lack of security there.
        
       ___________________________________________________________________
       (page generated 2021-02-08 23:00 UTC)