[HN Gopher] SolarWind, enough with the password already
___________________________________________________________________
SolarWind, enough with the password already
Author : troydavis
Score : 193 points
Date : 2021-02-27 18:31 UTC (4 hours ago)
(HTM) web link (gru.gq)
(TXT) w3m dump (gru.gq)
| OptionX wrote:
| Claming the hackers would have been able to compromise SolarWinds
| even with good security pratices does not absolve the company of
| having actual good security.
|
| I know there are people that can pick a lock. Still gonna have
| them on my doors anyway.
| jakelazaroff wrote:
| I think the point is that if you accidentally leave your door
| unlocked, you can't truthfully say "well, if I didn't do that
| the burglars couldn't have gotten in" while you also have
| ground floor windows.
| joe_the_user wrote:
| If we're using the door analogies... it seems like the solar
| winds attack came because someone figured out how to sneak-in
| the "doggie door" (hacked the auto-update function, maybe
| call it the "delivery door" like homes had 50 years).
|
| And problem wasn't so much that this happened (cause indeed,
| shit happen). The problem was all these _key_ enterprises
| (Microsoft, government agencies, etc, etc) had "doggies
| doors" when really they should have only had the most secure
| doors themselves.
|
| And sadly, unlike the retail situation, where a bank can
| decide they want only a secure door to their vault, today's
| enterprises have basically decided the benefits of giving
| multiple access to other enterprises trumps the security
| costs, since they never pay the costs of bad security anyway.
| tptacek wrote:
| The article notes that none its analysis absolves SolarWinds,
| and repeatedly goes out of its way to knife them.
| grugq wrote:
| My problem is not with SolarWind but with the analysis that
| keeps raising the password as the one and only problem. "If
| only they had good password hygiene then the company would've
| been totally safe against the Russian intelligence services!"
|
| That is just not how things work.
|
| To go further. The password was on GitHub from 2017 to
| November 2019. The first test build to see if they could
| backdoor things was in October 2019. If the password was the
| problem, why wasn't SolarWind hacked in 2017 or 18?
|
| The only explanation is that it wasn't an operation that
| existed back then. It wasn't a target for the SVR at that
| point in time, or they weren't able to service it with their
| operational capacity. But regardless, the critical factor
| here is that the RIS started this operation, not that the
| password was bad or available on GitHub. (Or whatever the
| issue is with the password.)
|
| Let's discuss whether the operational concept (CONOP) of
| hacking a civilian target to get into the supply chain and
| hit other targets is acceptable in cyber espionage. It seems
| to be acceptable because that is a methodology that everyone
| uses.
|
| My point has not been that you just can't win against Ho Chi
| Minh, or that SolarWind was particularly negligent (or not,
| their security posture was abysmal but also irrelevant)... my
| point is that we should focus on what actually matters -- the
| CONOP. Because if the US sanctions Russia for this operation
| then the US is locking itself and it's allies into a position
| when this CONOP is off the table. If that is what everyone
| agrees with, fine. But it's the real discussion to have. Not
| what sort of security SolarWind did or (realistically) did
| not have.
| wyldfire wrote:
| > There is no rule that would prohibit the SolarWinds espionage
| campaign which the US would be willing to abide by itself.
|
| Of course!
|
| > misunderstanding of the hack in the public sphere
|
| Hopefully that part is not misunderstood, right?
| mmaunder wrote:
| Grugq makes a few good points here:
|
| Sophisticated hacks employ a kill chain - think of it as what
| aviation calls a "cascade of failures". There's no single cause
| for the awful outcome, but instead a series of events where
| intercepting any of them could have mitigated the crash or in
| this case, the hack. For example, sure they got in, but they also
| remained undetected. If they didn't get in or if they were
| detected, the whole thing may have been mitigated.
|
| I also like how he's breaking away from just labeling the thing
| 'APT' and instead he describes the entity behind the attack, who
| they are, where they come from, what motivates them and how they
| are goal oriented rather than opportunistic. In other words, they
| didn't pick the target because of a weak password, they picked
| the target for strategic reasons.
|
| And finally the point of how well resourced and experienced these
| operatives are - or to use his phrase, they're pretty fucking
| metal. To unpack this a bit, the operatives targeting these kinds
| of attacks are well funded, experienced, patient, persistent,
| have large teams and once they've picked you, it's really hard to
| consider the odds stacked in your favor unless you truly
| understand what you're up against and have prepared accordingly.
| tmpz22 wrote:
| Reminds me of events like Pearl Harbor or even DDay where
| multiple signals existed to warn of the attacks and by some
| obnoxious sequence of ignorance they all went ignored.
| mmaunder wrote:
| That's perhaps a better analogy than my aviation one above.
| Exactly this. Break a link in the kill chain and it's
| prevented. So that's a slight advantage the defender has, but
| the trouble is most businesses (and individuals) have such a
| large attack surface that a determined attacker can usually
| find an alternative chain.
| grugq wrote:
| Thanks
| nominated1 wrote:
| > Could SolarWind have been too difficult for the KGB to use them
| in an enablement operation? Yes, it is possible to achieve that
| level of security. Creating a strong fast detection capability
| with rapid remediation and incident response will make it hard
| for attackers to dwell for any length of time, or persist on the
| system after they gain access. It requires vigilance and some
| effort, but it can be done. Of course, SolarWind wasn't close to
| reaching that level.
|
| Who is responsible for vetting these partners? What's the process
| look like? Surely it's more than "trust us, we gotz great
| securties".
| swiley wrote:
| "Everyone has been doing it this way, don't break stuff."
| chokeartist wrote:
| Password blame aside, any CEO who blames an individual
| contributor just failed a major career test.
|
| They get paid the big bucks not only for the good times, but the
| bad times too.
|
| Total weasel move.
| simias wrote:
| There's a lot of baseless conjecture in this as far as I can
| tell.
|
| >The SolarWind backdoor was deeply integrated into the code, it
| was injected during their build process, and there is no way that
| the server having a weak password was the pivotal factor. As if
| Russian Intelligence would just give up if there were a strong
| password instead!
|
| If this was a wikipedia article there'd be a [citation needed]
| every other word.
|
| I also wonder if they're not overplaying the skill of Russian
| cybersecurity agents. I'm sure they're good, I'm sure some of
| them are very, very good but the idea that basically they don't
| care about passwords is going too far IMO. The main advantage of
| being a state-sponsored hacker is that you have access to
| resources most other black hats couldn't dream of (like sending a
| team of burglars to ransack somebody's house, or physically
| threaten an employee) but that doesn't mean that they can stop
| obeying the laws of physics and algorithmics.
|
| >There is practically no chance that the server's password was in
| anyway relevant to the hack overall.
|
| Source: my behind.
|
| I think the author has a good point that it's probably best to
| have a holistic approach about these hacks instead of focusing
| exclusively on some details, but the details do matter. After
| all, the big picture is nothing but a long series of details,
| isn't it?
|
| >Close does not count in security. In offensive security you're
| either successful or not. When you're dealing with access then
| the only possible states are: did it work? Yes or no. Whether you
| need 5 minutes or 5 weeks to get a shell, once you have that
| shell, it is the same level of game over. That's what we're
| talking about here. The technique used to gain access is a minor
| issue.
|
| Reductio ad absurdum. So what does the author want us to do then?
| Set our passwords to qwerty1234 and just give up?
|
| It's especially weird when the author a few paragraphs earlier
| states:
|
| >Could SolarWind have been too difficult for the KGB to use them
| in an enablement operation? Yes, it is possible to achieve that
| level of security. Creating a strong fast detection capability
| with rapid remediation and incident response will make it hard
| for attackers to dwell for any length of time, or persist on the
| system after they gain access.
|
| So it turns out that 5 minutes or 5 weeks does matter after all?
|
| I find very little of substance in this entire rant. Also the KGB
| doesn't exist anymore, I don't know if the author doesn't know
| that or decides to keep using it for stylistic reasons but real
| life is not an 80s American B movie.
| grugq wrote:
| I call them the SVR, an agency formed from the FCD of the KGB.
| The first chief directorate was responsible for international
| espionage. The internal security responsibilities, along with
| everything else, was made into the FSB. There are some cross
| overs which reflect official Russian policy, such as the use of
| FSB officers to conduct espionage in the near abroad (such as
| Estonia) because the state believes they're oblasts not
| sovereign nations.
|
| That said, KGB is KGB. They call themselves Chekists, we call
| them KGB.
| chokeartist wrote:
| > Also the KGB doesn't exist anymore
|
| Come on, don't be pedantic. The KGB successors are the SVR and
| FSB. They are still active in every way imaginable from the KGB
| era.
| fortran77 wrote:
| > _I find very little of substance in this entire rant. Also
| the KGB doesn 't exist anymore, I don't know if the author
| doesn't know that or decides to keep using it for stylistic
| reasons but real life is not an 80s American B movie._
|
| I agree with you; not sure why the HN community finds this
| interesting.
| MikeDelta wrote:
| Obviously the FSB now.
|
| Social Engineering is a part of hacking and let's not
| underestimate the skills of spying agencies in this field.
| tptacek wrote:
| It's not clear what you're really trying to argue here.
| Obviously, he's not suggesting we use "qwerty1234" as our
| passwords; it's you who's reducing arguments to absurdity here.
|
| I'm pretty sure Grugq is aware of the current names for the
| Russian IC agencies.
| slt2021 wrote:
| did we ever see post mortem from Solarwinds? How did attackers
| enter the network? Even if build server used admin:admin, this
| server was in the intranet. How did they get inside the network?
|
| we need post mortem and to understand the entire attack chain,
| rather than sit and speculate about the abilities of KGB/SVR.
|
| Truth is KGB/SVR employees are very dumb and routinely leave
| traces. Their best hackers are actually civilians with commercial
| interests who do black hat campaigns for them in exchange for
| cover/protection on russian soil, but are not officially
| employed/enlisted.
|
| I have a hard time believing these were russian state sponsored
| hackers, unless somebody provides the hard evidence
| gist wrote:
| > did we ever see post mortem from Solarwinds?
|
| Other than to provide misinformation (to lead people astray)
| what is the advantage to solar winds to do a post mortem? Why
| educate people? Upside vs. downside?
|
| A businesses purpose is to act like a business. Not to educate
| (for lack of a better way to put it) the 'peanut' gallery,
| pundits, news outlets, bloggers or to improve security for
| others. Or to seem like 'a good company'. Nobody will deal with
| or not deal with Solar Winds based on what they say afterwords
| in a public and open forum. Privately and maybe under NDA sure
| but why broadcast this to everyone? (Answer is not 'well that's
| what you do').
| fphhotchips wrote:
| Simple: credibility. Right now, the default position is that
| they're insecure and they don't know what they're doing. A
| post-mortem with a solid RCA shifts that to "we've identified
| this, fixed it, and put in place systems to ensure it can't
| happen again."
|
| It's the aviation industry playbook; air travel is perceived
| as safe (partially) because of the big song and dance they
| put on about safety analysis after an incident.
| samstave wrote:
| A post-mortem allows for the company to perform an analysis
| of what exactly went wrong and what needs to be fixed etc...
|
| It also shows that they have a CSO role and they are trying
| to instill faith in their customers...
| bscphil wrote:
| Plenty of companies seem to think that seeming like "a good
| company" is a sufficient business reason to do a post-mortem.
| That's the thing about expectations: even if you can't come
| up with a rationale for doing something, other peoples'
| reactions to not doing the thing can be a sufficient
| justification for doing it.
|
| > well that's what you do
|
| I assume what this phrase is supposed to get at the sense
| that doing port mortems is the (morally) right thing to do
| and part of our duties as engineers to each other and to the
| public that has an interest in security. If that's the case
| then I have to disagree with you, that's an excellent reason
| to do it. (The fact that something would be the right thing
| to do means that you have a good reason to do the thing:
| namely, that it would be the right thing to do.) You can be
| cynical and say that "doing the right thing" is not going to
| be a good enough motivation to convince business X to do it,
| and that's fine, but it doesn't sound like that's what you're
| saying here.
| slt2021 wrote:
| doing post mortem will be responsible thing to do as it
| will allow other companies to strenghten their defenses
| against this vector. if it was rogue employee who used his
| credentials - well that's one vector. if it was cheap CIO
| who kept security team understaffed and underpaid =>
| underskilled -> well that's another lesson to learn for all
| other CIOs/CISOs around the world. Try to cut costs on your
| IT people, try to outsource talent - this is what you get.
| It will earn a goodwill for Solarwinds and will help
| everybody else to be aware of the attack vector
| genmud wrote:
| You are absolutely wrong on your approach to this. If there
| is no transparency into what happened, there is nothing but
| the companies word that it won't happen again. In the case of
| SolarWinds, their word means less than nothing.
|
| When you have a breach of this magnitude, people need to
| understand how the attack happened and what technical or
| process controls you have put in place to prevent it going
| forward.
|
| The biggest issue with SolarWinds breach is they have done
| nothing but try to obfuscate what happened. When they did
| press release, they said it was a "Security Vulnerability".
|
| _WRONG!!!!_ There was a backdoor intentionally placed in
| their product and sent out as update to tens of thousands of
| customers. At the very best their response has been
| uninformed, but knowing what we know about solarwinds as a
| company it seems intentional. To date they have not corrected
| that release and still often times refer to it as a
| vulnerability.
| AnimalMuppet wrote:
| > If there is no transparency into what happened, there is
| nothing but the companies word that it won't happen again.
| In the case of SolarWinds, their word means less than
| nothing.
|
| True, the silence is damaging. But what if the answers
| would be _more_ damaging than the silence?
| troydavis wrote:
| tl;dr: thegrugq argues that it's possible for SolarWinds'
| security to have been inadequate, yet that security posture to
| have made no difference in whether they were hacked by the SVR.
|
| thegrugq argues it's very likely:
|
| > I'm perfectly willing to believe that their build servers were
| using "admin:admin" and that's how the Russians gained access to
| inject their code... but, this was a clandestine intelligence
| operation. They did not succeed merely because SolarWind had poor
| password hygiene.
| dilyevsky wrote:
| Nonsense argument. What's the point of securing anything if THE
| KGB can cast a hack spell and get root on my server?
| sydd wrote:
| No, read the article. What can they do if someone from KGB
| will apply to a junior dev position and get the job? They
| will hand them all their keys.
| tptacek wrote:
| And there _are_ teams, a very few of them large, where
| internal segmentation is strong enough to survive a
| compromised developer machine. But that 's an
| extraordinarily high bar to clear and very few companies,
| even those with strong security teams, really manage to
| clear it.
| dilyevsky wrote:
| I've been hearing this argument for over 10 years now about
| nsa, idf, china and now the kgb. The truth is internal
| audits are quite effective at catching these. If nsa could
| just place a plant why did they spend so much effort to tap
| into companies' inter-dc fibers?
| ukj wrote:
| It's not an XOR.
| a1369209993 wrote:
| Not quite. Rather, they argue that it's _theoretically
| possible_ (though not necessarily realistic) to have good
| enough security to resist targeted hacking by the KGB, but this
| requires _vastly_ more than just better password hygiene.
| Sebb767 wrote:
| > it's theoretically possible (though not necessarily
| realistic) to have good enough security to resist targeted
| hacking by the KGB
|
| It is possible to have a security level where the cost
| outweighs the benefit. If the KGB _really_ wants to go all
| out, they could buy employees, burn zero days or even hold
| sysadmin families hostage - but that would be extremely
| expensive and risky and they 'd really need a big reason to
| go that far. If your password is "admin" or "solarwinds123",
| on the other hand, the biggest expense is probably the
| employee time spent laughing in the coffee room.
|
| I agree that it is vastly expensive to have a security level
| high enough that the KGB realistically _can not_ take over
| your network, but it 's far cheaper and more realistic to
| have a security level where it's _not worth_ the expense.
| a1369209993 wrote:
| > I agree that it is vastly expensive to have a security
| level high enough that the KGB realistically _can not_ take
| over your network, but it 's far cheaper and more realistic
| to have a security level where it's _not worth_ the
| expense.
|
| Yep. The former is possible in theory but not likely to
| happen. The latter is somewhat difficult, but ought to be
| table stakes for any company dealing with security-
| sensitive anything.
| blincoln wrote:
| I have mixed feelings about this. I agree with him that if the
| SVR specifically targeted a particular organization with a
| specific goal in mind, it wouldn't really matter if they had
| weak passwords or not.
|
| OTOH, I think it's not out of the question that one or more
| organizations (intelligence agencies, criminals, etc.) found
| the password and took advantage of it more as a "let's see
| where this thread leads" type of opportunistic attack, and all
| of the downstream consequences only happened because of that.
|
| I've never worked for an intelligence agency, but I've been a
| professional penetration tester for about a decade, and when I
| go after an organization, that's typically my approach: find
| the weakest links and start following them to see where they
| go. In a complex environment, usually that leads to control
| over everything sooner or later.
|
| Edit: just to clarify that last paragraph, what I'm getting at
| is that if I imagine myself in the shoes of an intelligence
| agency, the "organizations" I'd be going after would be foreign
| countries. I'm sure in some cases it would make sense to go
| after specific businesses of interest, but in the absence of
| legal restrictions, I'd be looking for the weakest links in
| entire industries that supported those countries in some way,
| not necessarily picking a specific business and targeting them.
|
| There are quite a few companies out there that make systems
| monitoring and administration software that would provide
| similar levels of access to a wide range of organizations if
| their build chains were compromised. The one that _was_
| compromised was the one that also had a publicly-exposed update
| server with a password that could have been obtained in at
| least two different ways.[1] Coincidence? Perhaps, but I don 't
| think it's fair to just take it off the table.
|
| [1] Accidentally exposed in a public GitHub repo for many
| months, as well as being easily guessed. Either alone would
| have been enough. Both being true seems to me to make it more
| likely.
| troydavis wrote:
| I think that's a good summary. Also, it's easy to imagine the
| SVR poking at all companies which sell high-trust
| applications[1] to many government agencies, and running with
| the ones[2] that worked.
|
| Regarding whether the password thing was a coincidence, I
| wouldn't be surprised that, if other large enterprise
| software companies were severely hacked, similar stories
| surfaced. That doesn't mean it's a coincidence, of course,
| but may mean that this is average among enterprise software
| companies. One takeaway here is that software companies
| shipping trusted software to third-party networks have an
| exposure more like Google or the Federal Reserve, not like
| other software companies. That's not how (a lot of) the
| software industry has acted.
|
| [1]: NMS, systems management, facilities management, possibly
| CRM
|
| [2]: Where one was detected (because the attackers chose the
| wrong target in FireEye), others have probably occurred
| and/or are active now. While this was very sophisticated, it
| wasn't Stuxnet
| (https://blog.erratasec.com/2021/02/no-1000-engineers-were-
| no...).
| dang wrote:
| Related ongoing thread:
|
| _SolarWinds CEO blames intern for password leak_ -
| https://news.ycombinator.com/item?id=26284782
| dj_mc_merlin wrote:
| The underestimation of the offense is done to ridiculous extents.
| The NSA pwned most of the world and exfiltrated data for years
| before it ever came to light (from one of their own!), yet all
| non-technical and even some technical people talk about security
| as if it's a bit you turn on or off. Unless you intend to spend
| as much money and resources on defense as USA/Russia/China does
| on offense, it's an uphill battle you will only seldom win. And
| you only have to lose once to lose almost everything.
| Sebb767 wrote:
| But that doesn't absolve you from trying! I totally agree with
| the point that a TLA has the resources to get into the network
| if they throw everything they have at you. A strong password
| might have changed nothing. But it is _still_ a total failure
| on basic security on SolarWinds side. And honestly, the fact
| that they 're now blaming an intern shows that this was not the
| _one small weak_ point the attacker found, but a cultural
| problem.
| jaredsohn wrote:
| I think I remember seeing solarwinds123 as a password around 2011
| (perhaps as a default password within Orion?) but couldn't find a
| web search for it.
|
| But I did find this example which I find amusing:
|
| "For example if your account name is 'orion@mycompany.com' and
| the password is SolarWinds123, that's what you put in for the
| authentication." https://thwack.solarwinds.com/product-
| forums/network-perform...
| jaredsohn wrote:
| Did some more searching; Google's filter by date is broken
| since companies update old urls to include recent news
| headlines.
|
| https://www.google.com/search?q=solarwinds123&tbs=cdr:1,cd_m...
| afrcnc wrote:
| Thanks Reuters for propagating that bug bounty hunter's
| speculation. Cause this is where all of this goes back to.
| walrus01 wrote:
| What I find really funny about Solarwinds - is that in the medium
| to large ISP sector it's _always_ been an absolute joke. Nobody
| of any consequence or real size has ever used it for network
| monitoring.
|
| Once you get to the scale of ISPs that have 50,000+ customers, or
| are supporting more than that through other smaller ISPs that are
| downstream of them - the monitoring and network automation tools
| are almost entirely open source, and some combination of
| GPL/LGPL/BSD/Apache/MIT license. Combined with custom things
| written in house to tie together different tools for a company's
| specific business needs.
|
| What you'll have typically is a collection of network equipment
| that may have closed-source operating systems (cisco, juniper
| routers and switches and similar, optical transport platforms
| from vendors like Infinera, Ciena), but everything managing and
| monitoring them is open source and runs on a *nix platform.
|
| If you have the in-house Linux/BSD knowledge to run the world's
| most powerful and popular open source networking tools, there is
| no need to ever touch solarwinds.
|
| My job interacts on a regular basis with all of the different
| pieces of the puzzle that make up solutions which are, in my
| opinion, vastly superior to Solarwinds.
|
| In the serious ISP business, if you ask the persons who admin the
| monitoring tools what they think of solarwinds, the answer you'll
| almost universally get back is "Windows GUI button pushing tools
| for enterprise end users who don't have the knowledge or
| motivation to really understand what's going on under the hood of
| their network".
| Spooky23 wrote:
| I don't know what you consider a "serious ISP", or what serious
| people do, but I'm confident that 10/10 of the big ISPs have
| Solarwinds on their corporate network.
|
| The customer facing stuff may be different, but once you own
| the LAN, you own the company.
|
| Their play was a cheaper, easier, multi-vendor toolset for
| enterprise networks. You'd pay half of whatever the Cisco dreck
| costs, and not need an army of consultants to tend it.
|
| I'd argue that the vast majority of network people do not
| demonstrate strong Unix skill sets. Windows tools FTW in most
| enterprises, as dumb as that may be.
| walrus01 wrote:
| > I don't know what you consider a "serious ISP"
|
| Something that's big enough and has a wide enough enough
| reach that other network operators with presence at major IX
| points know its AS number by sight - the same way people will
| recognize AS174 as Cogent or AS1299 as Telia, for instance.
| Or an ISP that is big enough that its wholly-owned/controlled
| fiber network spans most of a state, or several states, and
| has other major ISPs riding on it (whether as lit 10/100G
| customers, or dark fiber IRUs, or whatever).
|
| Something big enough to have a whole team of guys with bucket
| trucks and fiber equipment running around building the
| physical internet, while at the same time there's an
| office/work-from-home environment with 4 or 5 people whose
| job title has some form of "network engineer" in it, building
| the network at OSI layers 2/3.
|
| Or for an ISP that is not middle-mile/last-mile focused, and
| is rather a hosting/colocation company, something with
| significant datacenter presence at or near major IX points,
| as measured in square feet of space leased, kW of electrical
| power and cooling.
|
| > I'm confident that 10/10 of the big ISPs
|
| Which ASes would those be? If you can find a documented
| instance of a top-50 (by CAIDA ASRANK size) ISP using
| solarwinds to run its core stuff, please provide a reference
| to it...
|
| https://asrank.caida.org/asns
| vitus wrote:
| The most likely source of said claims would be something
| like SolarWinds's customer list, which they took offline in
| December.
|
| http://web.archive.org/web/20201214030038/https://www.solar
| w...
|
| "Our customer list includes: ... All ten of the top ten US
| telecommunications companies"
|
| I see... AT&T, Sprint, Comcast, Level 3 (now CenturyLink,
| still AS3356) for US-based ISPs. Telecom Italia made the
| shortlist, too.
|
| (And an honorable mention for Cisco, which was also
| apparently explicitly targeted: https://tools.cisco.com/sec
| urity/center/resources/solarwinds... "While Cisco does not
| generally use SolarWinds for its enterprise network
| management or monitoring, we have isolated and removed the
| Orion installations from a small number of Cisco assets.")
|
| That said, it doesn't say anything about _how_ the ISPs
| were using SolarWinds, just that they were in some
| capacity. But with any infiltration, it doesn't matter if
| it's widely used, so long as it's used somewhere that can
| be used as a launching pad for a follow-up attack.
|
| edit: apparently CenturyLink rebranded as Lumen last year.
| They're still AS3356 (and its subsidiary networks) to me.
| lucb1e wrote:
| > I don't know what you consider a "serious ISP"
|
| I understood this as referring to their definition higher up
| in their answer:
|
| > > ISPs that have 50,000+ customers, or are supporting more
| than that through other smaller ISPs that are downstream of
| them
| Sylamore wrote:
| >Nobody of any consequence or real size has ever used it for
| network monitoring.
|
| At least one of the big 3 telcos uses it very extensively for
| network monitoring, inventory, device configuration enforcement
| and alert generation.
| walrus01 wrote:
| I am much less surprised to hear that in the context of a
| company that is a "baby bell" / ILEC such as Verizon,
| Frontier, Centurylink (Former embarq/uswest/qwest/whatever),
| than I would be if I heard that they were using solarwinds
| inside NTT or Telia.
|
| Without going into a whole lot of personal opinion and
| detail, the business practices and management methodologies
| in a ILEC are very different from other ISPs.
| nikisweeting wrote:
| Out of curiosity, can you give some examples of open-source
| monitoring tools that large-scale ISPs use?
| gerdesj wrote:
| I'm not a large scale ISP but I do have to monitor quite a
| lot of stuff.
|
| You'd be amazed at how much you can monitor with
| Nagios/Icinga(1,2). They are written in C but call a lot of
| external stuff written in whatever you fancy and that's the
| power, right there. Bodge upon bodge! There's no single
| technology in these beasties. The interface between the
| system and the plugins is very basic to say the least, so you
| can throw whatever at it as you require. I'm a sysadmin not a
| programmer and need to get jobs done.
|
| We currently use Icinga 1 with a dash of Netdisco and I
| intend to migrate to Icinga 2 with DIrector etc.
|
| That said, I have dallied with OpenNMS many times ever since
| the project began - it's too good to ignore. Zabbix also
| turns my head quite often.
| walrus01 wrote:
| opennms is simultaneously
|
| arcane
|
| weirdly laid out
|
| a massive java memory hog (thankfully, RAM is cheap, giving
| an opennms VM 16GB of memory isn't a big deal anymore)
|
| extremely powerful
|
| something that has 450+ pages of documentation
|
| totally open source
|
| extensible to support monitoring of massive international-
| scale networks
| walrus01 wrote:
| the first and most important thing is to have the correct
| _network architecture and engineering_ to make effective use
| of the tools, and not have a network that needs a lot of
| babysitting in the first place. after that 's taken care of
| as an over-arching and continual business process:
|
| there is no one single _god box_ piece of software that is
| the be-all and do-all of network management /monitoring for
| an ISP. Some things come close, such as LibreNMS when used as
| the sole tool for a small ISP. But most often it is a
| patchwork quilt of many different things, each used for a
| discrete purpose.
|
| in no particular order:
|
| opennms
|
| a combination of (influxdb + telegraf + grafana)
|
| librenms
|
| provisioning and automation tools like ansible
|
| various in house things built on traditional RRA files and
| rrdtool
|
| tools like netbox for keeping track of datacenter
| customers/hosting environments
|
| phpipam or nipap for IP address management
|
| various self-hostable wiki software packages for internal
| documentation
|
| various types of self-hostable ticketing systems, monitoring
| systems that integrate with a customized asterisk system for
| NOC phone workflow
|
| 4 or 5 different tools that fill the same role as smokeping
|
| wireshark
|
| lots of different things for analyzing netflow data
| (Elastiflow or other)
|
| ELK stack stuff, elasticsearch/logstash/kibana, customized as
| needed.
|
| in house setups for openstreetmap tile servers and map
| presentation, to pull data from back-end mariadb databases
| and present them on monitoring displays.
|
| GIS software like QGIS and a PostGIS backend
|
| lots of different possible things done with custom code and
| postgresql, mysql/mariadb, or similar
|
| if you go through the PDF slideshows for the powerpoint decks
| at the last 4-5 years of the NANOG, RIPE and APNIC
| conferences you'll see discussion of some of the most popular
| network automation and monitoring tools.
| slt2021 wrote:
| a lot of CIOs are very cheap and prefer to keep their IT teams
| "lean" (understaffed, underpaid, underskilled) as they see them
| as cost center, and rather hand over couple mills to a vendor
| like SolarWinds to install their "automation/AI" pixie dust.
|
| So instead of investing into own employees who have the best
| interest of a company in mind (because you know job
| satisfaction and job security) -> they prefer investing into
| third party vendor whose interest is only to keep renewing
| multi-mill contract year after year while keep delivering
| barely above what's required to keep things afloat
| [deleted]
| detaro wrote:
| That's true with pretty much any enterprise-aimed tool, isn't
| it? With enterprise IT generally expected to run a very mixed
| pile of different stuff, without having the resources to
| specialize much on each of these things. Whereas an ISP, SaaS,
| specialized hosting company, ... puts much more emphasis on
| specific stacks and mastery of them, and often more recognition
| that investing in these things is not just a cost center.
| walrus01 wrote:
| Yes - though a big enough ISP with 40, 50 staff or more also
| needs a very wide range of common enterprise software tools,
| which the people running must have full mastery of.
|
| You've got stuff going on like billing/accounting systems,
| call centres, GIS systems for outside plant fiber
| construction and aerial+underground utilities work, HR
| software, VoIP systems, IDS and NAC systems. Lots of things
| that support the ordinary office-worker environment of the
| ISP in addition to all of the tools that automate and monitor
| the network.
| bob33212 wrote:
| People want an easy fix. It is easier to blame a single person
| for now follow a single rule, than to set the standard that
| executives should put controls in place to ensure that policies
| are being followed.
| Judgmentality wrote:
| I understand the spirit of your comment, but I want to point
| out
|
| > It is easier to blame a single person
|
| There is only one person to blame. The CEO.
| crazygringo wrote:
| Huh? But the board appointed the CEO. So why isn't it the
| board?
|
| Or wait, the board was appointed by shareholders. So why
| isn't it shareholders?
|
| And so on...
| Judgmentality wrote:
| The board can't be expected to know as much as the CEO, the
| shareholders can't be expected to know as much as the
| board, and so on.
|
| As far as I'm concerned, the #1 responsibility of the CEO
| is to take blame for fuckups.
|
| Yes, I consider that a higher priority than making profits.
| Because if the CEO is unable to make profits, then the CEO
| has to own the fuckup of not making profits.
| crazygringo wrote:
| But the CEO can't be expected to know as much as each VP,
| just like each VP doesn't know everything each manager
| knows, etc.
|
| Also, when profits aren't made, it's not the CEO who
| suffers. They already got their salary. Its shareholders
| who suffer.
|
| Sorry if it's not clear but my overall point is that
| accountability has to exist at all levels. The CEO isn't
| the position where all accountability emanates from or
| where it all stops. The CEO is held accountable to the
| board; VP's are held accountable to the CEO. The CEO is
| just one cog in the chain.
| burnthrow wrote:
| People focus on the password because it's the only part of the
| story they can relate to or understand. Orange County Rep. Katie
| Porter:
|
| > "I've got a stronger password than 'solarwinds123' to stop my
| kids from watching too much YouTube on their iPad ... You and
| your company were supposed to be preventing the Russians from
| reading Defense Department emails!"
|
| Words fail.
| owenmarshall wrote:
| Is she _that wrong_? I don't think so.
|
| Do I think most private companies could defend against Double
| Dragon or Lazarus or Fancy Bear? No, if a state level adversary
| is attacking you and the payoff is that good, you are going to
| get popped.
|
| But a strong posture makes it harder, which means they throw
| more at you and you have a chance of picking up on the attack.
| Best case, anyways. Worst case, you get to testify to Congress
| that your security measures were top notch and industry
| leading. That sounds a shit ton better than "we left a screen
| door open and didn't notice for months."
| burnthrow wrote:
| She's wrong to imply that if only SolarWinds had followed her
| iPad password policy, the attack would have been stopped. And
| she's mistaken about Orion's use case, which has nothing to
| do with email security.
|
| And while Russia conducted this attack, I'm tired of the
| Russian scarecrow: SolarWinds' job here has nothing to do
| with Russia.
|
| But mostly I'm jaded by ambitious SoCal pols neglecting their
| districts to score easy points on national issues.
| cobythedog wrote:
| > She's wrong to imply that if only SolarWinds had followed
| her iPad password policy, the attack would have been
| stopped.
|
| I don't think she was implying that at all. She was
| highlighting that if they couldn't even do a basic thing
| like employing stronger, more complex passwords - how could
| they defend against Russians reading DoD emails.
| throwawayboise wrote:
| > if a state level adversary is attacking you and the payoff
| is that good, you are going to get popped
|
| So we should assume Windows, Linux, every CDN, every major
| firewall, switch and router, etc. are all owned by Russia?
| owenmarshall wrote:
| Depends on how you want to slice that.
|
| _My_ laptop? _My_ OpenBSD router? Very unlikely anyone has
| attacked it. I've had boring jobs and have boring
| interests.
|
| Do I think the Russians, Iranians, or any major foreign
| adversary have a 0-day they could use against my systems if
| I suddenly got a top secret clearance and clocked in as
| more interesting? _Absolutely._
| disgruntledphd2 wrote:
| And by China, and by the US and probably a bunch of other
| actors.
|
| I mean, software is far too complicated in our current rube
| goldberg tower of abstractions, and the asymmetry favours
| the attacker (only have to be lucky once, etc).
|
| Until a few generations have grown up with software, I'm
| not sure this is going to improve (although in that case,
| we've probably solved climate change, so that would be
| good).
| tester756 wrote:
| Of course,
|
| because hacking other countries in Russia is "legal" and they do
| it.
|
| The trade off is that cannot really go on good vacations with all
| that stolen money, because they might have a unexpected visit
| slim wrote:
| this guy makes a lot of unsubstantiated assumptions about how kgb
| works
| mmaunder wrote:
| Grugq is well known in the infosec industry and more
| experienced in the area than you realize. He's been writing
| about opsec for years, among other areas.
| qbasic_forever wrote:
| I would wager anyone with intimate knowledge of how the KGB
| works, and has the evidence to prove it probably isn't going to
| be writing publicly about it for very long...
| closeparen wrote:
| Forensics and attribution are big parts of the infosec world.
| Researchers study attacks, tools, payloads, etc. and get an
| idea of different threat actors, their levels of
| sophistication, and who might have done which one. I'm sure
| it's never ironclad - the conclusion that it was a particular
| intelligence agency is just an educated guess, and
| sophisticated attackers might intentionally ape the
| signatures of others. But it's not completely hopeless.
| Veserv wrote:
| The article has good points about the nature of attacks by a
| determined adversary. The concept of "The One Critical TTP" that
| companies tout to divert blame and that observers use to justify
| why they are not vulnerable to the same thing is utter nonsense.
| If somebody shoots a bullet at a bulletproof vest and it goes
| through you should not conclude that they just so happened to hit
| the only weak spot and if you just fix it everything is okay. You
| should instead assume that the bulletproof vest might have lots
| of problems at least against the gun you were using to test it.
| Successful breaches and attacks do not show you where your only
| weak points are, they show you the level of quality your process
| provides. To actually fix the problems the process, not the
| product, needs to be improved so that it is able to deliver
| higher quality outcomes.
|
| In the case of SolarWinds, we now know that the level of quality
| their process provides is insufficient to stop whoever attacked
| them. If we assume that it was a targeted attack by a nation-
| state actor, then we now know that they can not protect their
| customers against an actual adversary who had reason to attack
| them, willingness to attack them, and the ability to attack them.
| They are completely unable to defend against actual threats who
| will actually attack them. Lots of people will say: "Of course if
| a nation-state wants to attack me then there is nothing I can do,
| but why would they attack me?" Well, in this case, that is an
| actual threat. To provide an actual solution they do, in fact,
| actually need to be able to stop a nation-state.
|
| So, how does SolarWinds fare against a nation-state? They are not
| even on the same continent. Everybody thinks it is completely and
| utterly laughable that they would have had any hope of stopping
| them. Not just that, it is a forgone conclusion that if a nation-
| state wants to attack _any_ commercial system they can with utter
| ease. It is not even viewed as a possibility for any currently
| deployed system to stop any nation-state from getting what they
| want.
|
| How far are these systems from stopping a nation-state? Well
| first we need to figure out what a nation-state can do. How
| valuable do you think the specs for the $1.5 trillion F-35
| project would be to a peer adversary [1]? $100B? $10B? $1B? At
| the very least I would state that if a peer adversary could get
| the specs for the F-35 they would be willing to spend at least
| $1B on that project. So, to stop a nation-state you need a system
| that can protect against an attack funded to the $1B level.
| Assuming $500k/engineer-yr that is an attack with 2,000 engineer-
| years of development on it. There is no organization in the world
| who would even dare to claim that a team of 400 engineers working
| for 5 years could not completely and utterly compromise their
| systems. Even at 1/10th that nobody would dare to claim they
| could stop 40 engineers with 5 years. Even at 1/100th you would
| be hard pressed to actually find anybody who would claim they can
| stop 4 engineers for 5 years and you could probably count on one
| hand somebody who could actually deliver. The systems that are
| being deployed that are actually attacked by and must protect
| against nation-states need to improve by at least a factor of
| 100x before they can actually do their job. So, these systems are
| multiple orders of magnitudes away from achieving the minimum
| standard of functionality.
|
| What can be done about this state of affairs? Either we must do
| 100x better than the best deployed systems, or if that can not be
| done, then we must assume that these systems can be 100%
| guaranteed compromised and act accordingly. Either we must
| disconnect these systems since they can not be defended, or the
| benefits of their use must be greater than the worst-case outcome
| of failure.
|
| [1] https://www.idga.org/archived-content/news/pentagon-
| admits-f...
| qbasic_forever wrote:
| I do think a lot of folks are missing the main point that no
| matter what security theater is in place, a state actor with
| enough motivation is going to breach it. They're not going to
| send someone after the password protected parts, they're going to
| send your recruiter the most irresistible candidate--the perfect
| background, right out of your favorite school and with expertise
| in exactly your tech stack. You'll get pages and pages of glowing
| recommendations from people inside and around the industry.
| They'll ace your interview loops, be loved by all your engineers
| and managers, and they won't bat an eye at the lowball first
| offer you give them. They'll move up the corporate ranks with
| ease and be everyone's friend... and then the best security in
| the world doesn't matter one bit.
| katzgrau wrote:
| An engineer with a great pedigree and work background and
| social finesse who passes all interviews and interaction
| without detection... Willing to take on a high risk role... All
| for the good of the homeland?
|
| I think you're describing a work of fiction. It's practically a
| James Bond or Bruce Wayne-like character. This particular
| person would be extremely hard to find/hire/compel by some
| competing nation, if they even exist.
| xienze wrote:
| > I think you're describing a work of fiction. It's
| practically a James Bond or Bruce Wayne-like character.
|
| Diane Feinstein, the _Chair of the Senate Intelligence
| Committee_, had a Chinese spy as her personal driver _for 20
| years_. So yes, slipping engineers into tech companies is not
| that complicated and in fact happens all the time.
| walrus01 wrote:
| You may be underestimating the recruiting efforts and budgets
| of major nation-state intelligence agencies - Russia and
| China are doing exactly the same thing the US has been doing
| for a long time. Spending some tens of millions of dollars to
| accomplish inserting an advanced persistent threat inside of
| software used by some huge percentage of the US federal
| government is a tiny drop in the bucket of the budget of such
| agencies.
|
| The main difference is that the US has historically
| accomplished it through a role in many cases as a vendor, or
| a supplier of essential high tech stuff (CIA/NSA and Crypto
| AG for instance).
| devoutsalsa wrote:
| Found the person who trains spies!
| eej71 wrote:
| Meet Jack Barsky.
|
| https://en.wikipedia.org/wiki/Jack_Barsky
| katzgrau wrote:
| Doesn't quite fit the image of perfection described above
| slt2021 wrote:
| care to add examples? has there ever been a case when state
| actor sent a rogue employee and succeeded? because in serious
| organizations there are robust defenses even against
| potentially rogue employees
| qbasic_forever wrote:
| It is _far_ more common than you think. A few years after I
| started at MS this guy was caught: https://www.theatlantic.co
| m/international/archive/2010/07/wh... I knew a lot of folks
| internally that worked with him, knew him, etc. and had _no
| idea_ he was a spy or agent. He was the model of a perfect
| employee or hire for MS at the time.
| slt2021 wrote:
| wow, so this guy worked as senior developer for infosec
| vendor NeoBIT (neobit.ru/partners openly cites FSB as their
| #1 client) and then gets a QA tester job at MSFT, no wonder
| DHS tracked him since he applied for visa.
|
| see, these people are not that smart, actually
| qbasic_forever wrote:
| Sure, but those are the people that have been caught. The
| ones actually making a difference will likely never be
| known.
| BCM43 wrote:
| https://www.nytimes.com/2019/11/06/technology/twitter-
| saudi-...
| spc476 wrote:
| While this happened to me in 2004, I can still see it working
| even today: http://boston.conman.org/2004/09/19.1
| freeone3000 wrote:
| Stuxnet is the one that comes to mind first, where a Polish
| mole was able to be hired at an Iranian nuclear enrichment
| facility.
| boogies wrote:
| Could you cite that? Why would they need to infect flash
| drives all around the world if they had a mole directly
| inside the facility?
|
| Edit: grepping Wikipedia for "mole" it does appear that
| there was an Iranian one working with the Dutch government.
|
| https://en.wikipedia.org/wiki/Stuxnet#cite_ref-156
| gcampos wrote:
| Not exactly what OP mentioned, but something like that did
| happen in the past:
|
| https://www.theguardian.com/technology/2010/jul/14/russian-s.
| ..
| [deleted]
| notsureaboutpg wrote:
| What's more common is that states can influence or "turn out"
| a model employee / excellent achiever into an informant /
| turncoat. Benedict Arnold is a classic historical example.
| mycall wrote:
| I thought China does this all the time.
|
| https://www.fbi.gov/news/speeches/responding-effectively-
| to-...
| slt2021 wrote:
| economic espionage is different thing and yes it exists.
| for example Huawei equipment - is cheap and low quality
| Cisco ripoff. But it is no different than employees in
| Silicon Valley changing companies and taking their
| knowledge to the competitor. In the former, people relocate
| from one company in the valley to China, in the latter,
| people go from one SV company to another. Same thing for me
|
| Has Lucid Motors's founder (former Tesla employee)
| conducted economic espionage or is it simply capitalism
| working as intended?
| nsxwolf wrote:
| Are there? I've done some hiring and I don't recall any
| procedures in place for the detection of foreign intelligence
| assets. I suppose it is possible that I have never worked for
| a serious organization.
| qbasic_forever wrote:
| Security clearances would help, but obviously that's not
| something that's going to be tenable for all companies.
|
| I think the main way to prevent issues is to just assume at
| any time you could be infiltrated. Don't mistrust all your
| employees, but don't live with lax security policies that
| allow a person to get away with something undetected.
| vitus wrote:
| Google learned this the hard way, between the Snowden
| revelations and China's cyberattacks.
|
| Audit trails are important, as is, um, not giving read
| access to user data without really really good
| justifications. Even beyond espionage, employees could
| stalk personal contacts (as happened at Uber in 2016 and
| Facebook in 2018).
| sanderjd wrote:
| My reading of the comment was that the protections are in
| the form of limiting what insiders can do / auditing what
| they do, rather than in detecting them during the hiring
| process.
| lucb1e wrote:
| I have yet to come across such an organisation, actually, and
| I'm a security consultant. The scenario is usually if an
| employee's systems get hacked, very few "serious
| organisations" seem to have procedures for rogue employees,
| at least seen from an EU perspective (perhaps the culture is
| different in the USA or in poorer parts of the world).
| slt2021 wrote:
| it is hard to catch expert rogue employees, but there are
| systems like conducting security clearance, DLP, UEBA,
| airgapped system separation (low security, medium security,
| high security), and ton of other security layers/controls
| that together can give some assurance
| kordlessagain wrote:
| > Security theater is the practice of taking security measures
| that are intended to provide the feeling of improved security
| while doing little or nothing to achieve it.
|
| https://en.wikipedia.org/wiki/Security_theater
| karmakaze wrote:
| That's an excessively long game--much easier to compromise
| someone already in a position.
| rantwasp wrote:
| why not both?
| qbasic_forever wrote:
| States and powerful, well-funded adversaries have all the
| time in the world. Look at the 9/11 attacks, the hijackers
| were taught how to fly planes by US flight schools!
| joncrane wrote:
| Keep in mind that there's a ton of institutional shorting on $SWI
| and that's where a lot of these attack articles are coming from.
|
| Having said that, SolarWinds is garbage software even without the
| security vulnerabilities and I hope it goes the way of the dodo.
| Source: I've had the misfortune of using it on multiple
| contracts.
| genmud wrote:
| Because anyone who is smart or has competent security teams
| have been looking to drop them like a sack of potatoes. I work
| with a ton of customers that have/had SolarWinds and the
| general sentiment with them is that they are ripping it out and
| replacing with alternatives.
|
| I would be willing to bet that the majority of SolarWinds sales
| are relationship based, not based on technical wins.
|
| It is extremely dangerous for a company that is based on
| relationship sales to require their economic buyers spend a
| non-insignificant amount of political capital defending bad
| practices. TBH most buyers aren't going to stick their neck out
| for them unless they have a _really_ good reason. Even then,
| security teams and legal teams might poo-poo the purchase and
| for most people in IT it is easier to find an alternative.
|
| Being reliant on relationship sales rather than technical wins
| is not by itself a bad sales or growth strategy... But as a
| company who has taken that approach, you have to ensure you
| don't do stupid stuff, or the stupid stuff isn't something that
| requires people to stick their neck out for you on.
|
| I imagine _this_ is why institutional investors are shorting
| $SWI more so than anything. Their customer churn is going to
| probably not be pretty and they are going to have to work
| really hard and hope that in 2-3 quarters people have forgotten
| about it.
| alfiedotwtf wrote:
| FIFY: I would be willing to bet that the majority of
| Enterprise sales are relationship based, not based on
| technical wins.
| karmakaze wrote:
| I accidentally clicked on a link in the article and the headline
| made me laugh.
|
| > Former SolarWinds CEO blames intern for 'solarwinds123'
| password leak
___________________________________________________________________
(page generated 2021-02-27 23:01 UTC)