[HN Gopher] British Library cyber incident review [pdf]
___________________________________________________________________
British Library cyber incident review [pdf]
Author : cnorthwood
Score : 100 points
Date : 2024-03-08 11:45 UTC (11 hours ago)
(HTM) web link (www.bl.uk)
(TXT) w3m dump (www.bl.uk)
| wara23arish wrote:
| I happened to be there while this attack was in progress
| (ocotober 23). And all there systems were really offline, POS
| didnt work, wifi didn't work, literally anything connected to a
| computer didnt work.
|
| What's unfortunate is that they flagged this vulnerability in
| 2022 and planned to review it in 2024 ???
|
| Does it usually take this long to identify impact of users? They
| mentioned they paid for identity protection for their staff & ex-
| staff as well.
| alexriddle wrote:
| I work in a related field (cyber insurance response) -
| typically takes a few months to identify exfiltrated data and
| then analyse it to understand what is in it. This might seem
| simple but there are usually in the region of hundreds of
| thousands to millions of files, and that may contain
| spreadsheets with tens of thousands of rows. This all has to be
| analysed, filtered and reduced to the point you have a list of
| PII which has been impacted, and can decide on what to do.
|
| Credit monitoring is usually offered as standard when a breach
| occurs, the UK is much less litigation friendly than the US so
| in the absence of any actual harm, that would discharge most of
| their obligations to protect you following an incident.
| ooterness wrote:
| Who decided credit monitoring was an adequate remedy for
| these breeches? I think I've accumulated three or four
| lifetimes of it by now, but it's never done anything but spew
| false alarms.
| jefc1111 wrote:
| "The Library utilises numerous trusted partners for software
| development, IT maintenance, and other forms of consultancy" ...
| "this terminal server was protected by firewalls and virus
| software, but access was not subject to Multi-Factor
| Authentication (MFA)"
|
| -\\_(tsu)_/-
| the8472 wrote:
| Occasionally malware groups do patch vulnerabilities to
| maintain exclusive control over the victim machines. But that
| wouldn't be my default expectation, so relying on virus
| software to provide security does not seem like a great idea.
| yard2010 wrote:
| There are many attack vectors to bypass MFA, especially sms
| based MFA
| jefc1111 wrote:
| True, but if you don't have it enabled / required then you're
| giving off signals of negligence which may extend into other
| vulnerabilities.
| nonrandomstring wrote:
| Good report. Well written incident summary useful for cyber-
| students to follow and learn.
|
| > The Library utilises numerous trusted partners for software
| development, IT maintenance, and other forms of consultancy
|
| > increasing complexity of managing their access was flagged as a
| risk.
|
| > first detected unauthorised access to our network was
| identified at the Terminal Services server. This terminal server
| had been installed in February 2020 to facilitate efficient
| access for trusted external partners
|
| Sadly their response seems to be using _more_ cloud
| infrastructure and outsourcing more.
|
| trusted != trustworthy
|
| The essential lesson - that good IT and security people _within_
| your company cost money. It is worth paying for vigilance,
| loyalty and care - has not been heeded.
| graemep wrote:
| > Sadly their response seems to be using more cloud
| infrastructure and outsourcing more.
|
| CYA - it stops being their management's fault if its
| outsourced,
| everfrustrated wrote:
| This report is a joke.
|
| No root cause. On other forums it is understood they were running
| very old and unpatched VMware os. Which is simply embarrassing
| and everybody within their IT team should be fired immediately
| for gross negligence.
|
| They can't inform people whos data has been compromised because
| they refuse to pay the ransom and have no other way to tell what
| was stolen. Farcical.
|
| Their ability to rebuild in a timely manner was hampered by not
| having any spare servers and presumably because all their server
| hardware was compromised and couldnt be used for restore.
| tokai wrote:
| >They can't inform people whos data has been compromised
| because they refuse to pay the ransom and have no other way to
| tell what was stolen.
|
| That doesn't fit their claims on page 7 about reviewing the
| lost data and contacting affected users.
| toyg wrote:
| They reviewed what the criminals later dumped on the dark
| web. They have no way to determine if the criminals kept more
| for themselves.
| clwg wrote:
| I suspect they don't have the forensic evidence to determine
| the root cause. Chances are there are probably too many ways it
| could have happened, and the evidence was encrypted or simply
| wasn't being captured.
|
| At least they seem to have a plan moving forward that seems
| considered, though I think a lot of what they want to do is
| easier said than done effectively. I wish them the best of
| luck.
| nonrandomstring wrote:
| > I suspect they don't have the forensic evidence to
| determine the root cause.
|
| It said that. The terminal server entry point was completely
| scorched in the attack. Offsite rlogd would have helped.
| toyg wrote:
| _> they refuse to pay the ransom and have no other way to tell
| what was stolen. Farcical._
|
| It's bad that they don't know what was taken, but as for paying
| the ransom, I wouldn't do it either: first, because it's
| _danegeld_ ; second, because you're just exposing yourself to
| even further risk by accepting files from criminals; third,
| because as others said, it would be UK tax money.
| timthorn wrote:
| > they refuse to pay the ransom
|
| As an organisation forming part of the UK State, they're not
| allowed to. Rightly, in my opinion.
| wara23arish wrote:
| If i was user/staff, I would sure prefer if they paid the
| ransom...
|
| Since I dont trust the library to actually assess my impact,
| or track records of companies getting hacked often drag their
| feet making it up to victims. (equifax)
| mschuster91 wrote:
| > No root cause. On other forums it is understood they were
| running very old and unpatched VMware os. Which is simply
| embarrassing and everybody within their IT team should be fired
| immediately for gross negligence.
|
| The IT team most likely begged _for years_ for funds to upgrade
| their infrastructure, but did not receive any of it. Public
| institutions are already short on money, but _education_ has it
| even worse.
|
| If anyone is to blame, it is the last British governments, who
| have focused their attention on Brexit and Ruanda crap instead
| of providing services for the citizens.
| hardlianotion wrote:
| It's a government with huge civil service infrastructure. The
| people involved with Brexit and Rwanda miles away from this
| stuff. Willing to bet that in your counterfactual world
| lacking Brexit and Rwanda (and let's throw in, say, a Labour
| government), this would still not have been financed.
| nonrandomstring wrote:
| > everybody within their IT team should be fired immediately
| for gross negligence.
|
| That may be true, but by that standard about 90% of every
| sysadmin, IT managers and even CISOs would be out of a job next
| week.
|
| Most companies are just "getting by" and hoping it won't be
| them next.
|
| We have a multi-national cybersecurity crisis due to decades of
| kicking the can down the road, excusing poor software
| engineering to allow unfettered commercial development, and
| destroying our education and training sectors.
| everfrustrated wrote:
| Not keeping on top of basic IT security is the equivalent of
| driving drunk.
| nonrandomstring wrote:
| Good analogy. It is. People's livelihoods and even people's
| lives are at risk.
|
| But we've utterly normalised digital ignorance and built
| what Edward Snowden very rightly calls an "Insecurity
| Industry".
|
| I'd go further, we've turned a celebration of ignorance
| around cybersecurity and dismissive attitudes into virtuous
| slogans. "Don't make me think" - Krug
| "Move fast and break things" - Mark Zuckerberg
| "If you've nothing to hide you've nothing to fear" - J
| Random Idiot
|
| And those who are charged with advising and protecting are
| deeply conflicted - because they want backdoor access or at
| least insecure products.
|
| What it boils down to is that presently there's more money
| and power in insecurity than there is in security. Our
| industry has multiple principal agent, Shirky Principle and
| Pournelle's Law problems, see [0].
|
| We allow ransomware and stalkerware companies, and outfits
| like NSO (which I only mention because they are most well
| recognised) to operate as legitimate.
|
| We flood markets with defective IoT crap and reduce
| consumers expectations to the level of accepting vendor
| malware and backdoors installed out of the box.
|
| And then we turn around and complain that "stuff ain't
| secure".
|
| This whole ship is DUI.
|
| [0] https://cybershow.uk/blog/posts/love/
| objclxt wrote:
| > I'd go further, we've turned a celebration of ignorance
| around cybersecurity and dismissive attitudes into
| virtuous slogans.
|
| > "Don't make me think" - Krug
|
| That quote has nothing to do with cybersecurity, it's the
| title of a book by Steve Krug about web usability.
|
| I am unfortunately old enough to have read that book when
| it first came out, and it's exclusively around how to
| design front-end UIs on websites to reduce user
| complexity. There is no mention of infrastructure or
| security at all.
|
| You're making a quote around how we should make websites
| more usable and understandable to users - so they can use
| them without thinking - into something it isn't.
| nonrandomstring wrote:
| > That quote has nothing to do with cybersecurity
|
| It has everything to do with it.
|
| I know exactly what the book is and I read it. It's
| actually an excellent book on UX and I expect Steve Krug
| picked the title because it sounds cool.
|
| No disrespect to that author intended, but it (maybe
| unwittingly) expresses a sentiment that has grave
| implications about the position of technology in human
| affairs. To understand why, please look deeper into what
| we used to call Human Computer Interaction (HCI) or
| "Cognitive Ergonomics".
|
| I think I recently mentioned it in this online chat [0]
|
| Explicit cognition is the "thinking slow" part of our
| brains that uses so-called left-brain linear reasoning
| and logic. It sits high in the cognitive stack. But as
| people use devices today, in what McLuhan [4] or Innes
| [5] would call an "acoustic" (nothing much to do with
| actual sound) way, we drop down a cognitive level to a
| faster, visual-haptic loop that bypasses explicit
| reasoning.
|
| Designing applications that bypass this has major effects
| on security. The work of B J Fogg will show you more
| about this [1].
|
| Tristan Harris also has lots on it [2,3].
|
| One of the disastrous effects of this "distracted" level
| of HCI is that people use more emotional cues, rote,
| colour, word association, implicit trust and other models
| that make them easy prey for phishing and other kinds
| magic and trickery.
|
| If you're interested in a much broader understanding of
| cybersecurity I give you a sincere invitation to check us
| out here [6].
|
| [0] https://www.youtube.com/watch?v=hYnOf4PWGpA
|
| [1] https://behaviordesign.stanford.edu/people/bj-fogg
|
| [2] https://www.youtube.com/watch?v=LUNErhONqCY
|
| [3] https://www.wired.com/story/our-minds-have-been-
| hijacked-by-...
|
| [4] https://en.wikipedia.org/wiki/Marshall_McLuhan
|
| [5] https://en.wikipedia.org/wiki/Harold_Innis
|
| [6] https://cybershow.uk/
| Veserv wrote:
| If 90% of them qualify as grossly negligent, then they should
| be fired. That is kind of what grossly negligent means.
|
| You do not really worry about what would happen if all the
| grossly negligent doctors get fired. Who will do those
| procedures with a total disregard for safety, said no one
| ever.
| nonrandomstring wrote:
| > You do not really worry about what would happen if all
| the grossly negligent...
|
| But I do. I care about them as people. People who have
| families and need a job. I'd rather help them to _not_ be
| grossly negligent than see them fired (and probably worse
| idiots take their place since we are in a major skills
| crisis right now).
|
| The world is getting complex faster than anyone can track.
| Tomorrow it could be you, or I who is getting called on
| gross negligence because we can't follow it. So I choose to
| be a teacher even though telling people the truth is
| getting REALLY F**ING HARD these days - cos no one wants to
| hear it.
| Veserv wrote:
| No, they should not continue to be in a position where
| they can continue committing grossly negligent actions
| and harm others.
|
| You can train them once they are removed and reinstate
| them when they can do the job right, but supporting their
| continued harm of others so they can "support themselves"
| is detrimental, counterproductive, misguided, and
| extremely selfish.
|
| You are literally better off paying them to do nothing.
| Please at least do that instead of paying for harm.
| nonrandomstring wrote:
| > they should not continue to be in a position
|
| "should" is doing a lot of work there. Im so many ways
| we're in agreement. But I do this in the real world, and
| experience has shown me we must deal with the world as it
| is and not merely as we wish it to be.
| KineticLensman wrote:
| > because they refuse to pay the ransom
|
| They were following explicit government guidance, as
| promulgated by the National Cyber Security Centre (NCSC), which
| is the civvie offshoot of GCHQ.
| CyberEldrich wrote:
| @everfrustrated: There is nothing in your piece that can be
| refuted. Therefore it must be modded into invisibility.
|
| > This report is a joke.
|
| > No root cause. On other forums it is understood they were
| running very old and unpatched VMware os. Which is simply
| embarrassing and everybody within their IT team should be fired
| immediately for gross negligence.
|
| > They can't inform people whos data has been compromised
| because they refuse to pay the ransom and have no other way to
| tell what was stolen. Farcical.
|
| > Their ability to rebuild in a timely manner was hampered by
| not having any spare servers and presumably because all their
| server hardware was compromised and couldnt be used for
| restore.
| emmelaich wrote:
| I object to the word "utilises" instead of just plain "uses",
| especially from a library.
| jgrahamc wrote:
| Yes. Horrible word. I actually banned utilise/utilize on the
| Cloudflare blog because use says the same thing (mostly), is
| easier to say, and shorter.
| aiiotnoodle wrote:
| A lot of this sounds like they were under-resourced and the
| business increasingly adopted new technology with no ongoing
| support for their IT infrastructure.
|
| > These legacy systems will in many cases need to be migrated to
| new versions, substantially modified, or even rebuilt from the
| ground up, either because they are unsupported and therefore
| cannot be repurchased or restored, or because they simply will
| not operate on modern servers or with modern security controls.
|
| > There is a clear lesson in ensuring the attack vector is
| reduced as much as possible by keeping infrastructure and
| applications current, with increased levels of lifecycle
| investment in technology infrastructure and security.
|
| > Our reliance on legacy infrastructure is the primary
| contributor to the length of time that the Library will require
| to recover from the attack.
|
| A lot of lines like the following, also indicate to me IT was
| increasingly were involved in fighting fires and maintining
| operational systems ("keeping the lights on") rather than
| deploying new infrastructure and automation, updating software
| etc.
|
| > Some of our older applications rely substantially on manual
| extract (...) which in a modern data management and reporting
| infrastructure would be encapsulated in secure, automated end-to
| end workflows.
|
| Modern business is IT, I know that I am preaching to the chior
| but this sounds a lot like their IT was seen as a cost.
| toyg wrote:
| The British Library is closer to academia than business. Their
| IT provider is a state-adjacent entity:
| https://en.wikipedia.org/wiki/Jisc .
| clwg wrote:
| I've known people who have worked in IT in national museum
| settings, and from what I heard it sounded like a mix of
| traditional IT support--ensuring the lights stayed on, printers
| could print, emails and phones worked, and a very simple
| website stayed online.
|
| Some aspects sounded quite interesting, but these weren't
| places pushing the envelope in any aspect of technology. I'm
| sure they were running outdated software and configurations on
| everything, but IT was closing their tickets and meeting their
| SLAs. And with no disrespect, these people weren't necessarily
| disruptors looking to shake up and modernize the museums'
| infrastructure and take it into the future either, they just
| did their job to the best of their ability and went home at the
| end of the day.
|
| To generalize I find that this usually holds true in a lot of
| non-tech industries, and IT is generally seen as a burdensome
| cost as opposed to enabler of business.
| graemep wrote:
| The British library has pretty complex systems because of the
| vast size of teh collection. Some pretty interesting stuff:
|
| https://www.youtube.com/watch?v=ZNVuIU6UUiM
| nazgulsenpai wrote:
| > However, the first detected unauthorised access to our
| network was identified at the Terminal Services server. This
| terminal server had been installed in February 2020 to
| facilitate efficient access for trusted external partners and
| internal IT administrators, as a replacement for the previous
| remote access system, which had been assessed as being
| insufficiently secure. Remote usage expanded during the
| subsequent Covid-19 pandemic because of the greatly increased
| requirement for remote working and the range of IT projects
| being undertaken with third party support.
|
| While I'm certain they are underfunded and overworked, this
| sounds like they had an internet accessible terminal server.
| I'd like to imagine IT screaming this is a bad idea but a suit
| somewhere saying they needed easy access for partners. I can
| only imagine how insecure the solution they replaced with this
| one was.
| KineticLensman wrote:
| I think it's part of a general trend where UK govt institutions
| have notoriously poor IT, usually consisting of semi-obsolete
| infrastructure, multiple legacy systems, sticking-plaster
| upgrades, one or two new state-of-the-art bits where budget is
| available, etc. Consider the NHS, the MOD, DVLA, etc.
| domh wrote:
| I would be fully supportive of the GDS
| (https://www.gov.uk/government/organisations/government-
| digit...) taking on additional responsibilities and providing
| support and assistance to other government agencies. gov.uk
| is almost universally praised by the general public and tech
| people.
| KineticLensman wrote:
| Agree, but they can't really do very much about the massive
| number of legacy systems in departments that can't or won't
| spend money to modernise. My favourite example to hate is
| the Driver and Vehicle Licensing Agency which tracks
| different things in multiple systems, and still requires
| snail mail interactions (!!!) for some services, such as
| reclaiming a license after a medical suspension (personal
| experience). To DVLA, people like me are a pure cost, as
| are the systems that record my data.
| kimixa wrote:
| Having experienced both the DVLA and (California) DMV,
| the DVLA feels miles ahead, like it's living in the
| future.
|
| Things like finding out the status of a renewal involved
| finding a _fax machine_ , everything but the most trivial
| renewal (say, renewing if you're on a work visa) seems to
| be done in person with handwritten paperwork, and the
| amount of busywork that seems to be done by hand by the
| DMV agent is quite easy to blame for the impressive wait
| times, multiple hours even if you have an appointment.
|
| My DVLA renewal was trivial comparatively, they could
| even use my passport for an updated ID photo. But maybe
| if you're not a UK citizen they also make you jump
| through weird hoops?
|
| I'm not saying that the DVLA is _good_ , just that it
| could be even worse.
| KineticLensman wrote:
| > I'm not saying that the DVLA is good, just that it
| could be even worse
|
| Some things they do reasonably well, yes. But edge cases
| like mine are the pits. To get a licence back after a
| medical suspension involves DVLA and the NHS posting
| physical letters to each other! It took 5 months for this
| purely admin process to complete, after I was medically
| fit. Grrrr. That is a long time to be denied the right to
| drive.
| toyg wrote:
| _> When alerted by the Library following discovery of the attack,
| Jisc (who provide the Library's internet access and monitor
| movement of data across their networks) identified that an
| unusually high volume of data traffic (440GB) had left the
| Library's estate at 1.30am on 28 October._
|
| "Jisc is the UK digital, data and technology agency focused on
| tertiary education, research and innovation."
|
| State-owned quango asleep at the wheel. Unsurprising.
| _tk_ wrote:
| Could you elaborate why them being state owned was a
| contributing factor? We've seen countless similar incidents
| with private MSSPs as well.
| toyg wrote:
| Because the state (eh) of State-owned or state-adjacent
| _anything_ , in modern Britain, is simply terrible. The
| dominant Thatcherite ideology ensures that state-provided
| services are almost invariably second-rate, thanks to
| systemic under-funding.
|
| In this case, it looks like Jisc was basically turned into a
| charity in 2011, so technically they're not even state-owned
| anymore.
| nonrandomstring wrote:
| > State-owned quango asleep at the qwheel. Unsurprising.
|
| This used to be what we called JANET. Back in the day this was
| top banana and prestigious to work for like GCHQ etc.
|
| I expect they've died from a thousand cuts under the Tories.
| Every university I've been in the past 10 years have their ICT
| run by Microsoft, and is absolute rubbish.
| ta1243 wrote:
| The library's ISP said "yes, our monitoring shows you shifted
| an unusual amount of traffic" at that time.
|
| My ISP could do the same thing. How is that being asleep at the
| wheel?
| physicsguy wrote:
| > The increasing use of third-party providers within our network,
| some of which has been due to capacity and capability constraints
| within Technology and elsewhere in the Library, was noted by the
| Library's Corporate Information Governance Group (CIGG) in late
| 2022, and the increasing complexity of managing their access was
| flagged as a risk. A review of security provisions relating to
| the management of third parties was planned for 2024; and the
| tightening of access provisions that would be enabled by
| improvements to underlying computer and storage infrastructure
| and the migration of storage to the cloud, which is currently
| being implemented. Unfortunately, the attack occurred before
| these necessary pre-requisites for this work were completed.
|
| Price of everything and value of nothing. Outsource everything,
| underfund everything from systems renewal to staff salaries.
| pheatherlite wrote:
| So Tom, Dick and Harry all have Terminal rdp access into the core
| infrastructure and they slept well knowing that they had - what
| was it? Ah, yes, - prevented clipboard copying as a hardening
| measure. That'll stop them pirates in their tracks. Nicely
| written post mortem. Though I can't help but notice the amount of
| committees and acronyms. Is it a British thing?
| KineticLensman wrote:
| > the amount of committees and acronyms. Is it a British thing?
|
| Take a look at the US DoD, NASA, etc. They _love_ acronyms,
| complicated internal organisation structures, just as much as
| the Brits do.
| b800h wrote:
| "Our major software systems cannot be brought back in their pre-
| attack form, either because they are no longer supported by the
| vendor or because they will not function on the new secure
| infrastructure that is currently being rolled out."
|
| Ouch.
| herodotus wrote:
| I have to applaud the library for releasing this report. In
| Canada, the most likely response to cyberattacks is mealy mouthed
| platitudes like "Please be assured that we take your privacy very
| seriously and are doing everything possible to recover the data
| and ensure that something like this does not happen again." and
| on and on.
|
| So refreshing.
| suyash wrote:
| Nice job on publishing this detailed report, I wish after every
| attack all organizations disclosed in such detail so we can
| create future defence and counter measures in an open source way.
| pbhjpbhj wrote:
| A few naive questions:
|
| I see a few comments indicating that connecting Microsoft (? not
| mentioned anywhere in the report??) t Terminal Services to the
| internet was a wholly bad idea.
|
| Aside: is the report using "Terminal Services" generically, or do
| they mean that the server hasn't been updated since before 2009
| (? when it seems Terminal Services became Remote Desktop Services
| (RDS))?
|
| Is there something inherently insecure about remote desktops, or
| is MS software here known to be particularly insecure, or ...?
| RDP is default enabled on MS Windows installs (I always disable
| it), is that more of a problem than one might imagine?
|
| Do they say anywhere where the access was from (maybe only GCHQ
| know that). Presumably the firewall would only allow known
| connections - did they report on analysis of all the remote
| clients?
| EvanAnderson wrote:
| > Is there something inherently insecure about remote desktops,
| or is MS software here known to be particularly insecure...
|
| Exposing RDP to the Internet directly has been frowned-upon
| because of the attack surface being presented, there's no two
| factor "story" out-of-the-box, and you're opened up to brute
| force attempts on cruddy user passwords.
|
| Older versions of the Microsoft Remote Desktop Protocol had a
| much larger attack surface than current versions. The current
| versions with Network Level Authentication (starting in Windows
| Vista/Server 2008) present a smaller attacks surface. Older
| versions used "homegrown" Microsoft crypto, whereas current
| versions use TLS.
|
| Disclosure: I made a FLOSS fail2ban-like tool for RDP many
| years ago[0]. I had a situation where I was forced to expose
| RDP to the Internet and I didn't like having it open w/o some
| protection against brute force attacks. This tool happens to
| still work in Server 2022 and will slow the velocity of brute
| force attacks. I still highly recommend not exposing RDP
| directly to the Internet anyway.
|
| (The ts_block tool is missing some fairly essential
| functionality that I never got around to implementing. It works
| fine and is really easy to install but some things are sub-
| optimal.)
|
| [0] https://github.com/EvanAnderson/ts_block
| penguin_booze wrote:
| > This paper provides an overview of the cyber-attack on the
| British Library that took place in October 2023 and examines its
| implications for the Library's operations, future infrastructure,
| risk assessment and lessons learned.
|
| For a report from British--and a library, no less--the lack of
| Oxford comma cocnerns me.
| seabass-labrax wrote:
| Despite its name, use of the Oxford comma is more frequently
| promoted in the USA than it is in Britain. As a British person
| myself, I generally avoid it. N=1, but I wouldn't expect the
| London-based British Library to use a construction named after
| an Oxford University Press style guide.
| gatvol wrote:
| Herein lies the kicker:
|
| > In common with other on-premise servers, this terminal server
| was protected by firewalls and virus software, but access was not
| subject to Multi-Factor Authentication (MFA).
___________________________________________________________________
(page generated 2024-03-08 23:01 UTC)