[HN Gopher] Fire destroys S. Korean government's cloud storage s...
___________________________________________________________________
Fire destroys S. Korean government's cloud storage system, no
backups available
https://www.chosun.com/english/national-en/2025/10/02/FPWGFS...
Author : ksec
Score : 714 points
Date : 2025-10-05 17:20 UTC (5 hours ago)
(HTM) web link (koreajoongangdaily.joins.com)
(TXT) w3m dump (koreajoongangdaily.joins.com)
| benoau wrote:
| > However, due to the system's large-capacity, low-performance
| storage structure, no external backups were maintained -- meaning
| all data has been permanently lost.
|
| Yikes. You'd think they would at least have _one_ redundant copy
| of it all.
|
| > erasing work files saved individually by some 750,000 civil
| servants
|
| > 30 gigabytes of storage per person
|
| That's 22,500 terabytes, about 50 Backblaze storage pods.
|
| Or even just mirrored locally.
| yongjik wrote:
| It's even worse. According to other articles [1], the total
| data of "G drive" was 858 TB.
|
| It's almost farcical to calculate, but AWS S3 has pricing of
| about $0.023/GB/month, which means the South Korean government
| could have reliable multi-storage backup of the whole data at
| about $20k/month. Or about $900/month if they opted for
| "Glacier deep archive" tier ($0.00099/GB/month).
|
| They did have backup of the data ... in the same server room
| that burned down [2].
|
| [1] https://www.hankyung.com/article/2025100115651
|
| [2] https://www.hani.co.kr/arti/area/area_general/1221873.html
|
| (both in Korean)
| paleotrope wrote:
| That's unfortunate.
| poly2it wrote:
| It's incompetent really.
| lukan wrote:
| No. Fortuna had nothing to do with this, this is called bad
| planning.
| BolexNOLA wrote:
| Couldn't even be bothered to do a basic 3-2-1! Wow
| sneak wrote:
| Did you expect government IT in a hierarchical respect-
| your-superiors-even-when-wrong society to be competent?
| BolexNOLA wrote:
| I mean...I feel you but holy hell dude. _Nothing_?
| Boggles the mind.
|
| Edit: my bad backups in the room is something, somehow
| just forgot about that part
| sneak wrote:
| It wasn't nothing. They had backups, according to yongjik
| above.
| username332211 wrote:
| South Korea isn't some sort of backwards nation and I'm
| sure it's chaebols share the same culture.
|
| Having had unfortunate encounters with government IT in
| other countries I can bet that the root cause wasn't the
| national culture. It was the internal culture of "I want
| to do the same exact same thing I've always done until
| the day I retire."
|
| Absent outside pressure, civil services across the word
| tend advance scientifically - one funeral (or retirement)
| at a time.
| rvba wrote:
| How does this even make sense business wise for AWS?
|
| Is their cost per unit so low?
| Ekaros wrote:
| When you start to do math, hard drive are cheap when you go
| for capacity and not performance.
|
| 0.00099*1000 is 0.99. So about 12$ a year. Now extrapolate
| something like 5 year period or 10 year period. And you get
| to 60 to 120$ for TB. Even at 3 to 5x redundancy those
| numbers start to add up.
| vbezhenar wrote:
| S3 does not spend 3x drives to provide redundancy.
| Probably 20% more drives or something like that. They
| split data to chunks and use erasure coding to store them
| in multiple drives with little overhead.
| sudo_and_pray wrote:
| This is just the storage cost. That is they will keep your
| data on their servers, nothing more.
|
| Now if you want to do something with the data, that's where
| you need to hold your wallet. Either you get their compute
| ($$$ for Amazon) or you send it to your data centre (egress
| means $$$ for Amazon).
| npteljes wrote:
| They charge little for storage and upload, but download, so
| getting your data back, is pricey.
| lucb1e wrote:
| Do the math. It's expensive. Or see e.g. this graph from a
| recent HN submission: https://si.inc/posts/the-heap/#the-
| cost-breakdown-cloud-alte...
| mastax wrote:
| I made an 840TB storage server last month for $15,000.
| baobabKoodaa wrote:
| You're assuming average worker utilized the full 30G of
| storage. More likely average was at like 0.3G.
| PeterStuer wrote:
| I'm sure they had dozens of process heavy cybersecurity
| committees producing hundreds if not thousands of powerpoints and
| word documents outlining procedures and best practices over the
| last decade.
|
| There is this weird divide between the certified class of non-
| technical consultants and actual overworked and pushed to corner
| cut techs.
| zaphar wrote:
| Ironically many of those documents for procedures probably
| lived on that drive...
| ksec wrote:
| I dont know why but cant stop laughing. And the great thing
| is that they will get paid again to write the same thing.
| comprev wrote:
| You jest, but I once had a client who's IaC provisioning code
| was - you guessed it - stored on the very infrastructure
| which got destroyed.
| perihelions wrote:
| Here's a 2024 incident:
|
| > _" The outage also hit servers that host procedures meant
| to overcome such an outage... Company officials had no paper
| copies of backup procedures, one of the people added, leaving
| them unable to respond until power was restored."_
|
| https://www.reuters.com/technology/space/power-failed-
| spacex...
| toast0 wrote:
| The data seems secure. No _cyberthreat actors_ can access it
| now. Effective access control: check.
| miniBill wrote:
| Ironically, see the phrack article someone linked above
| Titan2189 wrote:
| Surely there must be something that's missing in translation?
| This feels like it simply can't be right.
| mrbluecoat wrote:
| I agree. No automated fire suppression system for critical
| infrastructure with no backup?
| fredoralive wrote:
| That may not be a perfect answer. One issue with fire
| suppression systems and spinning rust drives is that the
| pressure change etc. from the system can also 'suppress' the
| glass platters in drives as well.
| privatelypublic wrote:
| I'd be interested in if you can even use dry fire
| suppression on the 5th floor of a building.
| magicalhippo wrote:
| Reminds me of the classic video[1] showing how shouting at
| the harddrives make them go slower.
|
| [1]: https://www.youtube.com/watch?v=tDacjrSCeq4
| perlgeek wrote:
| That's why the top-security DCs that my employer operates
| have large quantities of Nitrogen stored, and use that
| slightly lower the O2 saturation of the air in the case of
| fire.
|
| Yes, it's fucking expensive, that's one of the reason you
| pay more for a VM (or colocation) than at Hetzner or OVH.
| But I'm also pretty confident that single fire wouldn't
| destroy all hard drives in that IT space.
| rini17 wrote:
| Battery fire is impossible to suppress.
| oceansky wrote:
| Much harder, but not impossible.
| perlgeek wrote:
| That's why in high-quality DCs, battery backup is in a
| separate room with good fire isolation from the IT space.
|
| Yes, the servers still have some small batteries on their
| mainboards etc, but it's not too bad.
| layer8 wrote:
| It's accurate: https://www.chosun.com/english/national-
| en/2025/10/02/FPWGFS...
| nextworddev wrote:
| Because it was arson, not an accident
| pengaru wrote:
| Arson? Sounds increasingly like espionage.
| BrandoElFollito wrote:
| > all documents be stored exclusively on G-Drive
|
| Does G-Drive mean Google Drive, or "the drive you see as G:"?
|
| If this is Google Drive, what they had locally were just pointers
| (for native Google Drive docs), or synchronized documents.
|
| If this means the letter a network disk storage system was mapped
| to, this is a weird way of presenting the problem (I am typing on
| the black keyboard and the wooden table, so that you know)
| lysace wrote:
| _The name G-Drive is said to be derived from the word
| 'government'._
| indy wrote:
| It's now derived from the word 'gone'
| ncr100 wrote:
| 'Gone' up in smoke
| kristianc wrote:
| It's an X-Drive now
| prmph wrote:
| G-drive was simply the name of the storage system
| bryanhogan wrote:
| Saw a few days ago that the application site for the GKS, the
| most important scholarship for international students in Korea,
| went offline for multiple days, surprising to hear that they
| really lost all of the data though. Great opportunity to build a
| better website now?
|
| But yeah it's a big problem in Korea right now, lots of important
| information just vanished, many are talking about it.
| Zacharias030 wrote:
| Must have been a program without much trickle down into gov
| tech
| m3047 wrote:
| Mindblowing. Took a walk. All I can say is that if business
| continues "as usual" and the economy and public services continue
| largely unaffected then either there were local copies of
| critical documents, or you can fire a lot of those workers;
| either one of those ways the "stress test" was a success.
| layer8 wrote:
| "Final reports and official records submitted to the government
| are also stored in OnNara, so this is not a total loss".
| dghlsakjg wrote:
| How do you come to the conclusion that because things work
| without certain documents that you can start laying off
| workers?
| RaptorJ wrote:
| Surely having human-resource backups will also help with
| disaster recovery
| MiddleEndian wrote:
| >or you can fire a lot of those workers
|
| Sometimes things can seem to run smoothly for years when
| neglected... until they suddenly no longer run smoothly!
| npteljes wrote:
| Long term damage, and risk are two things that don't show up
| with a test like this. Also, often why things go forward is
| just momentum, built from the past.
| danparsonson wrote:
| Yeah you can do the same with your car too - just gradually
| remove parts and see what's _really_ necessary. Seatbelts,
| horn, rear doors? Gone. Think of the efficiency!
| aio2 wrote:
| Funny, because the same thing happened in Nepal a few weeks ago.
| Protestors/rioters burned some government buildings, along with
| the tech infrastructure within them, so now almost all electronic
| data is gone.
| dottjt wrote:
| Would this have been any different if these documents were
| stored non-electronically though? I understand that the whole
| point of electronic data is that it can be backed up, but if
| the alternative were simply an analog system then it would have
| fared no better.
| seunosewa wrote:
| It would have been better if storage was distributed.
| Muromec wrote:
| Paper records are usually distributed both by agency and by
| locality.
| rvba wrote:
| Happened in Bladerunner too
| senordevnyc wrote:
| And Fight Club
| serioussecurity wrote:
| Anti authoritarian patriots?
| 727564797069706 wrote:
| Meanwhile, Estonia has a "data embassy" in Luxembourg:
| https://e-estonia.com/solutions/e-governance/data-embassy/
|
| TL;DR: Estonia operates a Tier 4 (highest security) data center
| in Luxembourg with diplomatic immunity. Can actively run critical
| government services in real-time, not just backups.
| lostmsu wrote:
| This comment is in some way more interesting than the topic of
| the article.
| _joel wrote:
| Totally, backup disasters are a regular occurence (maybe not
| to the degree of negligence) but the Estonia DR is wild.
| lucb1e wrote:
| Definitely. Especially when considering that there were 95
| other systems in this datacentre which do have backups and
|
| > The actual number of users is about 17% of all central
| government officials
|
| Far from all, and they're not sure what's recoverable yet
| (""It's difficult to determine exactly what data has been
| lost."")
|
| Which is not to say that it's not big news ("the damage to
| small business owners who have entered amounts to 12.6
| billion Korean won." The 'National Happiness Card,' used for
| paying childcare fees, etc., is still 'non-functional.'"),
| but to put it a bit in perspective and not just "all was
| lost" as the original submission basically stated
|
| Quotes from https://www.chosun.com/english/national-
| en/2025/10/02/FPWGFS... as linked by u/layer8 elsewhere in
| this thread
| hkt wrote:
| That is absolutely delightful. Estonia is just _good_ at this
| stuff. Admirable.
| lukeqsee wrote:
| This is because everything is in digital form. Essentially all
| government systems are digital-first, and for the citizen,
| often digital-only. If the data is lost, there may be no paper
| records to restore everything from land registry, business
| registry (operating agreements, ownership records), etc.
|
| Without an out-of-country backup, a reversion to previous
| statuses means the country is lost (Estonia has been occupied a
| lot). With it, much of the government can continue to function,
| as an expat government until freedom and independence is
| restored.
| chpatrick wrote:
| "secured against cyberattacks or crisis situations with KSI
| Blockchain technology"
|
| hmmmm
| tamimio wrote:
| > Estonia follows the "once-only" principle: citizens provide
| their data just once, and government agencies re-use it
| securely. The next step is proactive services--where the
| government initiates service delivery based on existing data,
| without waiting for a citizen's request.
|
| I wish the same concept was in Canada as well. You absolutely
| have to resubmit all your information every time you do a
| request. On top of that, federal government agencies still mail
| each other the information, so what usually can be done in 1
| day takes a whole month to process, assuming the mail post
| isn't on strike (spoiler: they are now).
|
| I think Canada is one of the worst countries in efficiency and
| useless bureaucracy among 1st world countries.
| layer8 wrote:
| Some more details in this article:
| https://www.chosun.com/english/national-en/2025/10/02/FPWGFS...
| dang wrote:
| Thanks! we've added that link to the toptext as well
| lucb1e wrote:
| > The stored data amounts to 858TB (terabytes), equivalent to
| 449.5 billion A4 sheets.
|
| This attempt at putting it in perspective makes me wonder what
| _would_ put it in perspective. "100M sets of harry potter
| novels" would be one step in the right direction, but nobody
| can imagine 100M of anything either. Something like "a million
| movies" wouldn't work because they are very different from text
| media in terms of how much information is in one, even if the
| bulk of the data is likely media. It's an interesting problem
| even if this article's attempt is so bad it's almost funny
|
| Good article otherwise though, indeed a lot more detail than
| the OP. It should probably replace the submission. Edit: dang
| was 1 minute faster than me :)
| jopsen wrote:
| > The Interior Ministry explained that while most systems at the
| Daejeon data center are backed up daily to separate equipment
| within the same center and to a physically remote backup
| facility, the G-Drive's structure did not allow for external
| backups.
|
| This is why I don't really want to run my own cloud :)
|
| Actually testing the backups is boring.
|
| That said, ones the flames are out, they might actually be able
| to recover some of it.
| whartung wrote:
| Testing backups is boring. If you want exciting, test restores!
| MangoCoffee wrote:
| what's the point of a storage system with no back up?
| WJW wrote:
| It works fine as long as it doesn't break, and it's cheaper to
| buy than an equivalently sized system that does have back ups.
| lucb1e wrote:
| Isn't that self-evident? Do you have two microwaves from
| different batches, regularly tested, solely for the eventuality
| that one breaks? Systems work fine until some (unlikely) risk
| manifests...
|
| Idk if this sounds like I'm against backups, I'm not, I'm just
| surprised by the question
| MangoCoffee wrote:
| It's hard to believe this happened. South Korea has tech giants
| like Samsung, and yet this is how the government runs? Is the US
| government any better?
| userbinator wrote:
| The US government still relies heavily on physical records.
| nebula8804 wrote:
| Didn't Elon shut that down?
|
| [0]: https://www.cnbc.com/2025/02/13/company-ripped-by-elon-
| musk-...
| AshamedCaptain wrote:
| Why is there a "still" in there?
| ashirviskas wrote:
| South Korean IT seemed to be stuck in 2007 just not too long
| ago, would be surprised if it has changed much in the last few
| years. Do the websites still require you to use internet
| explorer?
| r_lee wrote:
| Software and information technology in Korea just sucks.
|
| buttons are jpegs/gifs, everything is on Java EE and on
| vulnerable old webservers etc... A lot of government stuff
| supports only Internet Explorer even though it's long dead
| creakingstairs wrote:
| Remember Log4j vulnerability? A lot of the Korea governmental
| sites weren't affected because the Java version was too old
| :)
|
| Don't even get me started on ActiveX.
| carrychains wrote:
| The first thing that comes to mind when I think of the South
| Korean government is the storied tradition of physical
| confrontation in their parliament along with more than a few
| viral videos of brawls and such over the years. It used to be
| better in the US, but with the intensity of discord in our
| government lately, I don't think anyone really knows anymore.
| eagleislandsong wrote:
| > The first thing that comes to mind when I think of the
| South Korean government is the storied tradition of physical
| confrontation in their parliament along with more than a few
| viral videos of brawls and such over the years
|
| You're thinking of Taiwan, not South Korea.
| creakingstairs wrote:
| No South Korea has the same thing. It doesn't happen yearly
| but has happened quite a bit. We lovingly call it
| parliament siege raid.
|
| https://m.blog.naver.com/gard7251/221339784832 (a random
| blog with gifs)
| logicchains wrote:
| Samsung's software is generally terrible; they're decent at
| hardware, not software.
| moduspol wrote:
| Our incompetence in the US is much more distributed. It
| wouldn't surprise me if the same kind of data isn't backed up,
| but at least it's dozens of separate federal agencies not-
| backing up their data in different physical places.
| foofoo12 wrote:
| Well, Elon has a recent copy of everything at least.
| jml78 wrote:
| Yes. The US government requires offsite backups .
|
| They also require routine testing distaster recovery plans.
|
| I participated in so many different programs over the years
| with those tests.
|
| Tests that would roll over to facilities across the country
| aorloff wrote:
| Theoretically, they still have the primary copies (on each
| individual person's "cloud-enabled" device).
| cthalupa wrote:
| > The Ministry of the Interior and Safety also issued
| guidelines to each ministry stating, "All work materials should
| not be stored on office PCs but should be stored on the
| G-Drive."
|
| They very well might have only been saving to this storage
| system. It was probably mapped as a drive or shared folder on
| the PC.
| crazygringo wrote:
| Do they? It's not clear if this was two-way sync or access on-
| demand.
|
| Like, I use Google Drive for Desktop but it only downloads the
| files I access. If I don't touch a file for a few days it's
| removed from my local cache.
| jn78 wrote:
| https://phrack.org/issues/72/7_md#article
| msbhvn wrote:
| Woah, read the timeline at the top of this. The fire happened
| the very day the government ordered onsite inspection was
| supposed to start due to Chinese/NK hacking.
| yieldcrv wrote:
| So, someone figured out how to do backups
| AnimalMuppet wrote:
| Yeah, this whole thing smells.
|
| Who has the incentive to do this, though? China/North Korea?
| Or someone in South Korea trying to cover up how bad they
| messed up? Does adding this additional mess on top mean they
| looked like they messed up _less_? (And for that to be true,
| how horrifically bad does the hack have to be?)
| gtirloni wrote:
| wow https://x.com/koryodynasty/status/1973956091638890499
|
| _> A senior government official overseeing recovery efforts
| for South Korea 's national network crisis has reportedly died
| by suicide in Sejong._
| covercash wrote:
| If the US government and corporate executives had even half
| this level of shame, we'd have nobody left in those
| positions!
| j3th9n wrote:
| Figures.
| jddj wrote:
| Silver lining: it's likely that technically there is a backup
| (section 1.3).
|
| It's just in NK or china.
|
| Yikes.
| tibbon wrote:
| I don't backup my phone. The NSA does it for me!
| 63stack wrote:
| This is the first time I see this site, who/what is phrack? A
| hacker group?
| fiatpandas wrote:
| It's a zine. Been around since the 80's. Hackers / security
| industry types read and publish to it.
| AnimalMuppet wrote:
| https://en.wikipedia.org/wiki/Phrack
| NKosmatos wrote:
| Thanks for this, it gives a lot of extra info and content
| compared to the original article.
| neilv wrote:
| When you see a chronology like that, you don't keep trying to
| speak truth to power.
|
| You delete your data, trash your gear, and hop on a bus, to
| start over in some other city, in a different line of work.
| AnimalMuppet wrote:
| s/city/country/
| baobun wrote:
| > 27th of September 2025, The fire is believed to have been
| caused while replacing Lithium-ion batteries. The batteries
| were manufactured by LG, the parent company of LG Uplus (the
| one that got hacked by the APT).
|
| Compromised batteries or battery controllers?
| dang wrote:
| [stub for offtopicness]
| mouse_ wrote:
| We will learn nothing
| pr337h4m wrote:
| Now imagine they had a CBDC.
| glitchc wrote:
| I thought most liberal governments gave up on those.
| blueflow wrote:
| [flagged]
| dvh wrote:
| Technically the data is still in the cloud
| pestaa wrote:
| I've been putting off a cloud to cloud migration, but
| apparently it can be done in hours?
| zigzag312 wrote:
| You can use accelerants to speed up migration
| VeninVidiaVicii wrote:
| The egress cost is gonna be a doozie though!
| datadrivenangel wrote:
| one of many fires to fight in such a fast scenario
| zigzag312 wrote:
| The cloud has materialized
| anonu wrote:
| Lossy upload though
| _zoltan_ wrote:
| Lossy download, no?
| layer8 wrote:
| No information is lost: https://en.wikipedia.org/wiki/No-
| hiding_theorem#:~:text=info...
| pjc50 wrote:
| Cloud of smoke, amirite.
| higginsniggins wrote:
| Unfortunately, the algorithm to unhash it is written in smoke
| signals
| gnfargbl wrote:
| https://mastodon.social/@nixCraft/113524310004145896
| cs702 wrote:
| Brilliant.
|
| This deserves its own HN submission. I submitted it but it
| was flagged due to the title.
|
| Thank you for sharing it on HN.
| kyrra wrote:
| Copy/paste:
|
| 7 things all kids need to hear
|
| 1 I love you
|
| 2 I'm proud of you
|
| 3 I'm sorry
|
| 4 I forgive you
|
| 5 I'm listening
|
| 6 RAID is not backup. Make offsite backups. Verify backup.
| Find out restore time. Otherwise, you got what we call
| Schrodinger backup
|
| 7 You've got what it takes
| zer00eyz wrote:
| This is the reason the 3, 2, 1 rule for backing up exists.
| dardeaup wrote:
| They might be singing this song now. (To the tune of
| 'Yesterday' from the Beatles). Yesterday,
| All those backups seemed a waste of pay. Now my
| database has gone away. Oh I believe in yesterday.
| Suddenly, There's not half the files there used to be,
| And there's a deadline hanging over me. The
| system crashed so suddenly. I pushed something
| wrong What it was I could not say. Now my
| data's gone and I long for yesterday-ay-ay-ay.
| Yesterday, The need for back-ups seemed so far away.
| Thought all my data was here to stay, Now I believe in
| yesterday.
| GLdRH wrote:
| For the German enjoyers among us I recommend also this old
| song: https://www.youtube.com/watch?v=jN5mICXIG9M
| Zacharias030 wrote:
| Thanks! mmd.
| cramcgrab wrote:
| Well that works out doesn't it? Saves them from discovery.
| rolph wrote:
| repeat after me:
|
| multiple copies; multiple locations; multiple formats.
| miohtama wrote:
| I thought clouds could not burn (:
| wartywhoa23 wrote:
| They are clouds of smoke to begin with. The smoke from the
| joints of those who believed that storing their data
| somewhere out of their control was a good idea!
| johnnienaked wrote:
| Good example of a Technology trap
| BurningFrog wrote:
| "The day the cloud went up in smoke"
| abujazar wrote:
| LOL
| Havoc wrote:
| >the G-Drive's structure did not allow for external backups.
|
| ah the so called schrodingers drive. It's there unless you try
| to copy it
| ahmgeek wrote:
| nice
| nntwozz wrote:
| The Egyptians send their condolences.
| gardnr wrote:
| Has there been a more recent event, or are you referring to
| Alexandria?
| Zacharias030 wrote:
| I think Alexandria.
| em-bee wrote:
| https://en.wikipedia.org/wiki/Library_of_Alexandria#Burning
| _...
|
| the destruction of the library of alexandria is under
| dispute.
| Zacharias030 wrote:
| touche
| shadowgovt wrote:
| Yikes. That is a nightmare scenario.
| thepill wrote:
| Watching Mr. Robot and seeing the burned batteries the same
| time...
| dagaci wrote:
| No problem -- I'm sure their Supremely nice Leader up north
| kept a backup. He's thoughtful like that...
| presentation wrote:
| Should have given him back his stapler
| Titan2189 wrote:
| I don't get it. Can you please explain the reference?
| elcapitan wrote:
| That's an "Office Space" reference, in which a grumpy
| employee burns down the IT company building.
| Terr_ wrote:
| Perhaps extra-relevant to a story about data-loss, Milton
| was an employee who fell through the cracks in a broken
| corporate bureaucracy.
|
| His was supposedly laid off years ago, but nobody
| actually stopped his paycheck, so he kept coming in to
| work assuming he was still employed, getting shuffled
| into increasingly-abusive working environments by
| callously indifferent managers who assume he's somebody
| else's problem.
| latexr wrote:
| It's a reference to the movie Office Space and the Milton
| character.
|
| https://en.wikipedia.org/wiki/Office_Space
| mekoka wrote:
| Or a piece of cake.
| zb3 wrote:
| Well, now they'll have to negotiate with North Korea to get
| these backups..
| pengaru wrote:
| I seem to have misplaced my tiny violin
| mirekrusin wrote:
| They should ask if North has a backup.
| GPerson wrote:
| Hope this happens to Altman's data centers.
| smlacy wrote:
| Someone found the literal HCF instruction.
| r011erba11 wrote:
| Too bad this can't happen everywhere.
| ivape wrote:
| I mean ... was making backups on the backlog at least? Can they
| at least point to the work item that was _going to get done
| soonish_?
| redditor98654 wrote:
| May be a "fast follow"? Right after launch of the "MVP"?
| odie5533 wrote:
| It got pushed a couple sprints and we've got it on the plan for
| next quarter as long as no new features come in before then.
| FinnKuhn wrote:
| If it wasn't it most certainly is now
| system2 wrote:
| I wonder how many IT professionals were begging some incompetent
| upper management official to do this the right way, but were
| ignored daily. You'd think there would be concrete policies to
| prevent these things...
| zulban wrote:
| If I worked there I'd have had a hard time believing there were
| really no backups. Governments can be very nebulous.
| kristianc wrote:
| The government official who insisted that commercial
| AWS/GCP/Azure couldn't possibly be trusted with keeping the
| information will be keeping their head low for a few days then...
|
| "The Interior Ministry explained that while most systems at the
| Daejeon data center are backed up daily to separate equipment
| within the same center and to a physically remote backup
| facility, the G-Drive's structure did not allow for external
| backups."
|
| This is absolutely wild.
| zwnow wrote:
| Rightfully did not trust these companies. Sure what happened is
| a disaster for them, but you cant simply trust Amazon &
| Microsoft.
| oceansky wrote:
| For sure the only error here is zero redundancy.
| kingnothing wrote:
| Why not? You can easily encrypt your data before sending it
| for storage on on S3, for example.
| zhouzhao wrote:
| You can encrypt them at rest, but data that lies encrypted
| and is never touched, is useless data. You need to decrypt
| them as well. Also, plenty of incompetent devops around,
| and writing a decryption toolchain can be difficult.
| kspacewalk2 wrote:
| Am I missing something? If you ever need to use this
| data, obviously you transfer it back to your premises and
| then decrypt it. Whether it's stored at Amazon or North
| Korean Government Cloud makes no difference whatsoever if
| you encrypt before and decrypt after transfer.
| DarkmSparks wrote:
| Encryption only protects data for an unknown period of
| time, not indefinately.
| mikehotel wrote:
| If your threat model includes the TLA types, then backup
| to a physical server you control in a location
| geographically isolated from your main location. Or to a
| local set of drives that you physically rotate to remote
| locations.
| oceansky wrote:
| They can take the data hostage, the foreign nation would
| have no recourse.
| icedchai wrote:
| Why write one when there are tools like "restic"?
| mikehotel wrote:
| Decryption is not usually an issue if you encrypt
| locally.
|
| Tools like Kopia, Borg and Restic handle this and also
| include deduplication and other advanced features.
|
| Really no excuse for large orgs or even small businesses
| and somewhat tech literate public.
| AshamedCaptain wrote:
| Is encryption, almost any form, really reliable protection
| for a countries' government entire data? I mean, this is
| _the_ ultimate playground for "state level actors" -- if
| someday there's a hole and it turns out it takes only 20
| years to decrypt the data with a country-sized
| supercomputer, you can bet _this_ is what multiple alien
| countries will try to decrypt first.
| lucb1e wrote:
| You're assuming that this needs to protect...
|
| > ... a countries' government entire data?
|
| But the bulk of the data is "boring": important to
| individuals, but not state security ("sorry Jiyeong, the
| computer doesn't know if you are a government employee.
| Apologies if you have rent to make this month!")
|
| There likely exists data where the risk calculation ends
| up differently, so that you wouldn't store it in this
| system. For example, for nuke launch codes, they might
| rather lose than loose them. Better to risk having to
| reset and re-arm them than to have them hijacked
|
| > Is encryption, [in?] any form, really reliable
| protection
|
| There's always residual risk. E.g.: can you guarantee
| that every set of guards that you have watching national
| datacenters is immune from being bribed?
|
| Copying data around on your own territory thus also
| carries risks, but you cannot get around it if you want
| backups for (parts of) the data
|
| People in this thread are discussing specific
| cryptographic primitives that they think are trustworthy,
| which I think goes a bit deeper than makes sense here.
| Readily evident is that there are ciphers trusted by
| different governments around the world for their
| communication and storage, and that you can layer them
| such that all need to be broken before arriving at the
| plain, original data. There is also evidence in the
| Snowden archives that (iirc) e.g. PGP could not be broken
| by the NSA at the time. Several ciphers held up for the
| last 25+ years and are not expected to be broken by
| quantum computers either. All of these sources can be
| drawn upon to arrive at a solid choice for an encryption
| scheme
| kazinator wrote:
| You and I can encrypt our data before saving it into the
| cloud, because we have nothing of value or interest to
| someone with the resources of a state.
|
| Sometimes sensitive data at the government level has a
| pretty long shelf life; you may want it to remain secret in
| 30, 50, 70 years.
| Den_VR wrote:
| On the Microsoft side CVE-2025-55241 is still pretty recent.
|
| https://news.ycombinator.com/item?id=45282497
| politelemon wrote:
| S3 features have saved our bacon a number of times. Perhaps
| your experience and usage is different. They are worth
| trusting with business critical data as long as you're
| following their guidance. GCP though have not proven it,
| their data loss news is still fresh in my mind.
| hosh wrote:
| Were you talking about this incidence?
| https://arstechnica.com/gadgets/2024/05/google-cloud-
| acciden...
|
| I am currently evaluating between GCP and AWS right now.
| fabian2k wrote:
| Using the cloud would have been the easiest way to achieve the
| necessary redundancy, but by far not the only one. This is just
| a flawed concept from the start, with no real redundancy.
| DarkmSparks wrote:
| But not security. And for governmental data security is a far
| more important consideration.
|
| not losing data and keeping untrusted parties out of your
| data is a hard problem, that "cloud" aka "stored somewhere
| that is accessible by agents of a foreign nation" does not
| solve.
| freehorse wrote:
| As OP says, cloud is not the only solution, just the
| easiest. They should probably have had a second backup in a
| different building. It would probably require a bit more
| involvement, but def doable.
| DrewADesign wrote:
| It's the government of South Korea, which has a nearly 2
| trillion dollar GDP. Surely they could have built a few
| more data centers connected with their own fiber if they
| were that paranoid about it.
| miken123 wrote:
| Because these companies never lose data, like during some
| lightning strikes, oh wait:
| https://www.bbc.com/news/technology-33989384
|
| As a government you should not be putting your stuff in an
| environment under control of some other nation, period. That is
| a completely different issue and does not really relate to
| making backups.
| firesteelrain wrote:
| For this reason, Microsoft has Azure US Government, Azure
| China etc
| whatevaa wrote:
| Yeah, I heard that consumer clouds are only locally redundant
| and there aren't even backups. So big DC damage could result
| in data loss.
| lima wrote:
| What do you mean by "consumer clouds"?
| alwa wrote:
| I mean... at the risk of misinterpreting sarcasm--
|
| Except for the backup strategy said consumers apply to
| their data themselves, right?
|
| If I use a service called "it is stored in a datacenter in
| Virginia" then I will not be surprised when the meteor that
| hits Virginia destroys my data. For that reason I might
| also store copies of important things using the "it is
| stored in a datacenter in Oregon" service or something.
| Johnny555 wrote:
| By default, Amazon S3 stores data across at least separate
| datacenters that are in the same region, but are physically
| separate from each other:
|
| _Amazon S3 provides a highly durable storage
| infrastructure designed for mission-critical and primary
| data storage. S3 Standard, S3 Intelligent-Tiering, S3
| Standard-IA, S3 Glacier Instant Retrieval, S3 Glacier
| Flexible Retrieval, and S3 Glacier Deep Archive redundantly
| store objects on multiple devices across a minimum of three
| Availability Zones in an AWS Region. An Availability Zone
| is one or more discrete data centers with redundant power,
| networking, and connectivity in an AWS Region. Availability
| Zones are physically separated by a meaningful distance,
| many kilometers, from any other Availability Zone, although
| all are within 100 km (60 miles) of each other._
|
| You can save a little money by giving up that redundancy
| and having your data i a single AZ:
|
| _The S3 One Zone-IA storage class stores data redundantly
| across multiple devices within a single Availability Zone_
|
| For further redundancy you can set up replication to
| another region, but if I needed that level of redundancy,
| I'd probably store another copy of data with a different
| cloud provider so an AWS global failure (or more likely, a
| billing issue) doesn't leave my data trapped in one
| vendor).
|
| I believe Google and Azure have similar levels of
| redundancy levels in their cloud storage.
| ncruces wrote:
| _"The BBC understands that customers, through various backup
| technologies, external, were able to recover all lost data."_
|
| You backup stuff. To other regions.
| littlestymaar wrote:
| But the Korean government didn't backup, that's the problem
| in the first place here...
| ncruces wrote:
| Sure. Using _a_ cloud _can_ make that more convenient.
| But obviously not so if you then keep all your data in
| the same region, or even "availability-zone" (which seems
| to be the case for the all "lost to lightening strikes"
| data here).
| lima wrote:
| ...on a single-zone persistent disk: https://status.cloud.goo
| gle.com/incident/compute/15056#57195...
|
| > _GCE instances and Persistent Disks within a zone exist in
| a single Google datacenter and are therefore unavoidably
| vulnerable to datacenter-scale disasters._
|
| Of course, it's perfectly possible to have proper distributed
| storage without using a cloud provider. It happens to be hard
| to implement correctly, so apparently, the SK government team
| in question just decided... not to?
| kspacewalk2 wrote:
| >As a government you should not be putting your stuff in an
| environment under control of some other nation, period.
|
| Why? If you encrypt it yourself before transfer, the only
| possible control some_other_nation will have over you or your
| data is availability.
| littlestymaar wrote:
| First of all, you cannot do much if you keep all the data
| encrypted on the cloud (basically just backing things up,
| and hope you don't have to fetch it given the egress cost).
| Also, availability is exactly the kind of issue that a fire
| cause...
| creddit wrote:
| Yeah backups would've been totally useless in this case.
| All South Korea could've done is restore their data from
| the backups and avoid data loss.
| shakna wrote:
| You're forgetting that you're talking nation states, here.
| Breaking encryption is in fact the role of the people you
| are giving access.
|
| Sovereign delivery makes sense for _nations_.
| bombcar wrote:
| You can use and abuse encrypted one time pads and
| multiple countries to guarantee it's not retrievable.
| firesteelrain wrote:
| I know there is legit hate for VMWare/Broadcom but there is a
| legit case to be made for VCF with an equivalent DR setup where
| you have replication enabled by Superna and Dell PowerProtect
| Data Domain protecting both local and remote with Thales Luna
| K160 KMIP for the data at rest encryption for the vSAN.
|
| To add, use F710s, H710s and then add ObjectScale storage for
| your Kubernetes workloads.
|
| This setup repatriates your data and gives you a Cloud like
| experience. Pair it with like EKS-A and you have a really good
| on premises Private Cloud that is resilient.
| eCa wrote:
| Agree completely that it's absolute wild to run such a system
| without backups. But at this point no government should keep
| critical data on foreign cloud storage.
| stogot wrote:
| Why not? If the region is in country, encrypted, and with
| proven security attestations validated by third parties, a
| backup to a cloud storage would be incredibly wise. Otherwise
| we might end up reading an article about a fire burning down
| a single data center
| g-b-r wrote:
| And which organization has every file, from each of their
| applications using the cloud, encrypted *before* it is sent
| to the cloud?
| exe34 wrote:
| They're talking about backups. you can absolutely send an
| updated copy every night.
| g-b-r wrote:
| True, the user I was replying to only mentioned backups.
|
| For those there's sure no problem
| crazygringo wrote:
| Exactly.
|
| Like, don't store it in the cloud of an _enemy country_ of
| course.
|
| But if it's encrypted and you're keeping a live backup in a
| second country with a second company, ideally with a
| different geopolitical alignment, I don't see the problem.
| OvbiousError wrote:
| Enemy country in the current geopolitical climate is an
| interesting take. Doesn't sound like a great idea to me
| tbh.
| deaddodo wrote:
| There are a lot of gray relations out there, but there's
| almost no way you could morph the current US/SK relations
| to one of hostility; beyond a negligible minority of
| citizens in either being super vocal for some perceived
| slights.
| 9dev wrote:
| Trump will find a way, just as he did with Canada for
| example (i mean, _Canada_ of all places). Things are way
| more in flux than they used to be. There's no stability
| anymore.
| shantara wrote:
| A year ago, I would have easily claimed the same thing
| about Denmark.
| gitremote wrote:
| You think when ICE arrested over 300 South Korean
| citizens who were setting up a Georgia Hyundai plant and
| subjected them to alleged human rights abuses, it was
| only a perceived slight?
|
| https://www.huffpost.com/entry/south-korea-human-rights-
| inve...
|
| How Trump's ICE Raid Triggered Nationwide Outrage in
| South Korea
|
| https://www.newsweek.com/trump-ice-raid-hyundai-outrage-
| sout...
|
| 'The raid "will do lasting damage to America's
| credibility," John Delury, a senior fellow at the Asia
| Society think tank, told Bloomberg. "How can a government
| that treats Koreans this way be relied upon as an
| 'ironclad' ally in a crisis?"'
| kergonath wrote:
| One could have said the exact same thing about US-EU
| relations just a couple of years ago. And yet, here we
| are.
| t-3 wrote:
| From the perspective of securing your data, what's the
| practical difference between a second country and an
| enemy country? None. Even if it's encrypted data, all
| encryption can be broken, and so we must assume it _will_
| be broken. Sensitive data shouldn 't touch outside
| systems, period, no matter what encryption.
| Avamander wrote:
| Any even remotely proper symmetric encryption scheme "can
| be broken" but only if you have a theoretical adversary
| with nearly infinite power and time, which is in practice
| absolutely utterly impossible.
|
| I'm sure cryptographers would _love_ to know what makes
| it possible for you to assume that say AES-256 or AES-512
| can be broken in practice for you to include it in your
| risk assessment.
| 9dev wrote:
| You're assuming we don't get better at building faster
| computers and decryption techniques. If an adversary gets
| hold of your encrypted data now, they can just shelf it
| until cracking becomes eventually possible in a few
| decades. And as we're talking about literal state secrets
| here, they may very well still be valuable by then.
| stavros wrote:
| Barring any theoretical breakthroughs, AES can't be
| broken any time soon even if you turned every atom in the
| universe into a computer and had them all cracking all
| the time. There was a paper that does the math.
| Avamander wrote:
| You make an incorrect assumption about my assumptions.
| Faster computers or decryption techniques will never
| fundamentally "break" symmetric encryption. There's no
| discrete logarithm or factorization problem to speed up.
| Someone might find ways to make for example AES key
| recovery somewhat faster, but the margin of safety in
| those cases is still incredibly vast. In the end there's
| such an unfathomably vast key space to search through.
| XorNot wrote:
| The risk that the key leaks through an implementation bug
| or a human intelligence source.
|
| Exfiltrating terabytes of data is difficult, exfiltrating
| 32 bytes is much less so.
| Avamander wrote:
| That's very far from the encryption itself being broken
| though. If that were the claim, I would have had no
| complaints.
| crazygringo wrote:
| > _From the perspective of securing your data, what 's
| the practical difference between a second country and an
| enemy country? None._
|
| Huh? An enemy country will shut off your access. Friendly
| countries don't.
|
| > _Even if it 's encrypted data, all encryption can be
| broken, and so we must assume it will be broken._
|
| This is a very, very hot take.
| VirusNewbie wrote:
| A statement like "all encryption can be broken" is about
| as useful as "all systems can be hacked" in which case,
| not putting data in the cloud isn't really a useful
| argument.
| manquer wrote:
| The problem is money,
|
| you are seeing the local storage decision under the lens
| of security, that is not the real reason for this type of
| decision.
|
| While it may have been sold that way, reality is more
| likely the local DC companies just lobbied for it to be
| kept local and cut as many corners as they needed. Both
| the fire and architecture show they did cut deeply.
|
| Now why would a local company voluntary cut down its
| share of the pie by suggesting to backup store in a
| foreign country. They are going to suggest keep in
| country or worse as was done here literally the same
| facility and save/make even more !
|
| The civil service would also prefer everything local
| either for nationalistic /economic reasons or if corrupt
| then for all kick backs each step of the way, first for
| the contract, next for the building permits, utilities
| and so on.
| vkou wrote:
| A country can become an adversary faster than a
| government can migrate away from it.
| crazygringo wrote:
| Hence a backup country. I already covered that.
|
| But while countries go from unfriendly to attacking you
| overnight, they don't generally go from friendly to
| attacking you overnight.
| vkou wrote:
| Overnight, Canada went from being an ally of the US to
| being _threatened by annexation_ (and target #1 of an
| economic war).
|
| If the US wants its state-puppet corporations to be used
| for integral infrastructure by foreign governments, it's
| going to need to provide some better legal assurances
| than 'trust me bro'.
|
| (Some laws on the books, and a congress and a SCOTUS that
| has demonstrated a willingness to enforce those laws
| against a rogue executive would be a good start.)
| shakna wrote:
| Microsoft has already testified that the American
| government maintains access to their data centres, in all
| regions. It likely applies to all American cloud companies.
|
| America is not a stable ally, and has a history of spying
| on friends.
|
| So unless the whole of your backup is encrypted offline,
| and you trust the NSA to never break the encryption you
| chose, its a national security risk.
| bink wrote:
| Not only does the NSA break encryption but they actually
| sabotage algorithms to make them easier to break when
| used.
| edoceo wrote:
| Can the NSA break the Ed25519 stuff? Like the crypto_box
| from libsodium?
| immibis wrote:
| ed25519 (and ec25519) are generally understood not to be
| backdoored by the NSA, or weak in any known sense.
|
| The lack of a backdoor can be proven by choosing
| parameters according to straightforward reasons that do
| not allow the possibility for the chooser to insert a
| backdoor. The curve25519 parameters have good reasons why
| they are chosen. By contrast, Dual_EC_DRBG contains two
| random-looking numbers, which the NSA pinky-swears were
| completely random, but actually they generated them using
| a private key that only the NSA knows. Since the NSA got
| to choose any numbers to fit there, they could do that.
| When something is, like, "the greatest prime number less
| than 2^255" you can't just insert the public key of your
| private key into that slot because the chance the NSA can
| generate a private key whose public key just happens to
| match the greatest prime number less than 2^255 is zero.
| These are called "nothing up my sleeve numbers".
|
| This doesn't prove the algorithm isn't just plain old
| weak, but nobody's been able to break it, either. Or find
| any reason why it would be breakable. Elliptic curves
| being unbreakable rests on the discrete logarithm of a
| random-looking permutation being impossible to
| efficiently solve, in a similar way to how RSA being
| unbreakable relies on nobody being able to efficiently
| factorize very big numbers. The best known algorithms for
| solving discrete logarithm require O(sqrt(n)) time, so
| you get half the bits of security as the length of the
| numbers involved; a 256-bit curve offers 128 bits of
| security, which is generally considered sufficient.
|
| (Unlike RSA, you can't just arbitrarily increase the bit
| length but have to choose a completely new curve for each
| bit length, unfortunately. ed25519 will always be 255
| bits, and if a different length is needed, it'll be
| similar but called something else. On the other hand,
| that makes it very easy to standardize.)
| jacquesm wrote:
| > but nobody's been able to break it, either.
|
| Absence of evidence is not evidence of absence. It could
| well be that someone has been able to break it but that
| they or that organization did not publish.
| edoceo wrote:
| How could you not!? Think of the bragging rights. Or,
| perhaps the havoc. That persons could sit on this secret
| for long periods of time seem... difficult to maintain.
| If you know it's broken and you've discovered it; surely
| someone else could too. And they've also kept the secret?
|
| I agree on the evidence/absence of conjecture. However,
| the impact of the secret feels impossible to keep.
|
| Time will, of course, tell; it wouldn't be the first
| occasion where that has embarrassed me.
| jacquesm wrote:
| There are a large number mathematicians gainfully
| employed in breaking such things without talking about
| it.
| Avamander wrote:
| Large amounts of data, like backups, are encrypted using
| a symmetric algorithm. Which makes the strength of
| Ed25519 somewhat unimportant in this context.
| JumpCrisscross wrote:
| > _America is not a stable ally, and has a history of
| spying on friends_
|
| America is a shitty ally for many reasons. But spying on
| allies isn't one of them. Allies spy on allies to verify
| they're still allies. This has been done throughout
| history and is basic competency in statecraft.
| 9dev wrote:
| That doesn't capture the full truth. Since Snowden, we
| have hard evidence the NSA has been snooping on foreign
| governments and citizens alike with the purpose of
| harvesting data and gathering intelligence, not just to
| verify their loyalty.
|
| No nation should trust the USA, especially not with their
| state secrets, if they can help it. Not that other
| countries are inherently more trustworthy, but the US is
| a known bad actor.
| JumpCrisscross wrote:
| > _Since Snowden, we have hard evidence the NSA has been
| snooping on foreign governments and citizens alike_
|
| We also know this is also true for Russia, China and
| India. Being spied on is part of the cost of relying on
| external security guarantees.
|
| > _Not that other countries are inherently more
| trustworthy, but the US is a known bad actor_
|
| All regional and global powers are known bad actors. That
| said, Seoul is already in bed with Washington. Sending
| encrypted back-ups to an American company probably
| doesn't increase its threat cross section materially.
| dralley wrote:
| > France spies on the US just as the US spies on France,
| the former head of France's counter-espionage and
| counter-terrorism agency said Friday, commenting on
| reports that the US National Security Agency (NSA)
| recorded millions of French telephone calls.
|
| > Bernard Squarcini, head of the Direction Centrale du
| Renseignement Interieur (DCRI) intelligence service until
| last year, told French daily Le Figaro he was
| "astonished" when Prime Minister Jean-Marc Ayrault said
| he was "deeply shocked" by the claims.
|
| > "I am amazed by such disconcerting naivete," he said in
| the interview. "You'd almost think our politicians don't
| bother to read the reports they get from the intelligence
| services."
|
| > "The French intelligence services know full well that
| all countries, whether or not they are allies in the
| fight against terrorism, spy on each other all the time,"
| he said.
|
| > "The Americans spy on French commercial and industrial
| interests, and we do the same to them because it's in the
| national interest to protect our companies."
|
| > "There was nothing of any real surprise in this
| report," he added. "No one is fooled."
| ants_everywhere wrote:
| France has had a reputation for being especially active
| in industrial espionage since at least the 1990s. Here's
| an article from 2011
| https://www.france24.com/en/20110104-france-industrial-
| espio...
|
| I always thought it was a little unusual that the state
| of France owns over 25% of the defense and cyber security
| company Thales.
| kergonath wrote:
| > I always thought it was a little unusual that the state
| of France owns over 25% of the defense and cyber security
| company Thales.
|
| Unusual from an American perspective, maybe. The French
| state has stakes in many companies, particularly in
| critical markets that affect national sovereignty and
| security, such as defence or energy. There is a
| government agency to manage this: https://en.wikipedia.or
| g/wiki/Agence_des_participations_de_l... .
| neom wrote:
| Good thing Korea has cloud providers, apparently Kakao has
| even gone...beyond the cloud!
|
| https://kakaocloud.com/ https://www.nhncloud.com/
| https://cloud.kt.com/
|
| To name a few.
| alephnerd wrote:
| They are overwhelmingly whitelabeled providers. For
| example, Samsung SDI Cloud (the largest "Korean" cloud) is
| an AWS white label.
|
| Korea is great at a lot of engineering disciplines. Sadly,
| software is not one of them, though it's slowly changing.
| There was a similar issue a couple years ago where the
| government's internal intranet was down a couple days
| because someone deployed a switch in front of outbound
| connections without anyone noticing.
|
| It's not a talent problem but a management problem -
| similar to Japan's issues, which is unsurprising as Korean
| institutions and organizations are heavily based on
| Japanese ones from back in the JETRO era.
| skissane wrote:
| I spent a week of my life at a major insurance company in
| Seoul once, and the military style security, the
| obsession with corporate espionage, when all they were
| working on was an internal corporate portal for an
| insurance company... The developers had to use machines
| with no Internet access, I wasn't allowed to bring my
| laptop with me lest I use it to steal their precious
| code. A South Korean colleague told me it was this way
| because South Korean corporate management is stuffed full
| of ex-military officers who take the attitudes they get
| from defending against the North with them into the
| corporate world; no wonder the project was having so many
| technical problems-but I couldn't really solve them,
| because ultimately the problems weren't really technical
| msla wrote:
| I see they even amputated your periods, and forced you to
| use ellipses exclusively.
| ahartmetz wrote:
| I've done some work for a large SK company and the
| security was manageable. Certainly higher than anything
| I've seen before or after and with security theater
| aspects, but ultimately it didn't seriously get in the
| way of getting work done.
| vgivanovic wrote:
| I am very happy with the software that powers my Hyundai
| Tuscon hybrid. (It's a massive system that runs the gas
| and electric engines, recharging, shifting gears,
| braking, object detection, and a host of information and
| entertainment systems.) After 2 years, 0 crashes and no
| observable errors. Of course, nothing is perfect: maps
| suck. The navigation is fine; it's the display that is at
| least 2 decades behind the times.
| dralley wrote:
| Samsung owns Joyent
| ciupicri wrote:
| Nevertheless isn't Joyent registered in the US?
| dtech wrote:
| Encrypted backups would have saved a lot of pain here
| edoceo wrote:
| Any backup would do at this point. I think the most best
| is: encrypted, off-site & tested monthly.
| CamouflagedKiwi wrote:
| And yet here is an example where keeping critical data off
| public cloud storage has been significantly worse for them in
| the short term.
|
| Not that they should just go all in on it, but an encrypted
| copy on S3 or GCS would seem really useful right about now.
| vladms wrote:
| You can do a bad job with public or private cloud. What if
| they would have had the backup and lost the encryption key?
|
| Cost wise probably having even a Korean different data
| center backup would not have been huge effort, but not
| doing it exposed them to a huge risk.
| hinkley wrote:
| We've had Byzantine crypto key solutions since at least
| 2007 when I was evaluating one for code signing for
| commercial airplanes. You could put an access key on k:n
| smart cards, so that you could extract it from one piece of
| hardware to put on another, or you could put the actual key
| on the cards so burning down the data center only lost you
| the key if you locked half the card holders in before
| setting it on fire.
| n5NOJwkc7kRC wrote:
| SSS is from 1979.
| https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing
| JumpCrisscross wrote:
| > _no government should keep critical data on foreign cloud
| storage_
|
| Primary? No. Back-up?
|
| These guys couldn't provision a back-up for their on-site
| data. Why do you think it was competently encrypted?
| jacquesm wrote:
| They fucked up, that much is clear but the should not have
| kept that data on foreign cloud storage regardless. It's
| not like there are only two choices here.
| JumpCrisscross wrote:
| > _the should not have kept that data on foreign cloud
| storage regardless. It 's not like there are only two
| choices here_
|
| Doesn't have to be an American provider (Though anyone
| else probably increases Seoul's security cross section.
| America is already its security guarantor, with tens of
| thousands of troops stationed in Korea.)
|
| And doesn't have to be permanent. Ship encrypted copies
| to S3 while you get your hardenede-bunker domestic option
| constructed. Still beats the mess that's about to come
| for South Korea's population.
| jacquesm wrote:
| I'm aware of a big cloud services provider (I won't name
| any names but it was IBM) that lost a fairly large amount
| of data. Permanently. So that too isn't a guarantee. They
| simply should have made local and off-line backups,
| that's the gold standard, and to ensure that those
| backups are complete and can be used to restore from
| scratch to a complete working service.
| nicolas_17 wrote:
| DigitalOcean lost some of my files in their object
| storage too:
| https://status.digitalocean.com/incidents/tmnyhddpkyvf
|
| Using a commercial provider is not a guarantee.
| lukevp wrote:
| DO Spaces, for at least a year after launch, had no
| durability guarantees whatsoever. Perhaps they do now,
| but I wouldn't compare DO in any meaningful way to S3,
| which has crazy high durability guarantees as well as
| competent engineering effort expended on designing and
| validating that durability.
| xoa wrote:
| > _I 'm aware of a big cloud services provider (I won't
| name any names but it was IBM) that lost a fairly large
| amount of data. Permanently. So that too isn't a
| guarantee._
|
| Permanently losing data at a given store point isn't
| relevant to losing data overall. Data store failures are
| assumed or else there'd be no point in backups. What
| matters is whether failures in multiple points happen _at
| the same time_ , which means a major issue is whether
| "independent" repositories are actually truly independent
| or whether (and to what extent) they have some degree of
| correlation. Using one or more completely unique systems
| done by someone else entirely is a pretty darn good way
| to bury accidental correlations with your own stuff,
| including human factors like the same tech people making
| the same sorts of mistakes or reusing the same components
| (software, hardware or both). For government that also
| includes political factors (like any push towards using
| purely domestic components).
|
| > _They simply should have made local and off-line
| backups_
|
| FWIW there's no "simply" about that though at large
| scale. I'm not saying it's undoable at all but it's not
| trivial. As is literally the subject here.
| bombcar wrote:
| If you can't encrypt your backups such that you could store
| them tatooed on Putin's ass, you need to learn about backups
| more.
| charlieyu1 wrote:
| You don't need cloud when you have the data centre, just
| backups in physical locations somewhere else
| pico303 wrote:
| What a lame excuse. "The G-Drive's structure did not allow for
| backups" is a blatant lie. It's code for, "I don't value other
| employees' time and efforts enough to figure out a reliable
| backup system; I have better things to do."
|
| Whoever made this excuse should be demoted to a journeyman ops
| engineer. Firing would be too good for them.
| CoastalCoder wrote:
| You could be right, but it could also be a bad summary or bad
| translation.
|
| We shouldn't rush to judgement.
| MBCook wrote:
| It could be accurate. Let's say, for whatever reason, it is.
|
| Ok.
|
| Then it wasn't a workable design.
|
| The idea of "backup sites" has existed forever. The fact you
| use the word "cloud" to describe your personal collection of
| servers doesn't suddenly mean you don't need backups in a
| separate physical site.
|
| If the government mandates its use, it should have a hot site
| at a minimum. Even without that a physical backup in a
| separate physical location in case of
| fire/attack/tsunami/large band of hungry squirrels is a total
| must-have.
|
| However it was decided that not having that was OK, that
| decision was negligence.
| littlestymaar wrote:
| > The government official who insisted that commercial
| AWS/GCP/Azure couldn't possibly be trusted with keeping the
| information
|
| They were still right though: it's absolutely clear without an
| ounce of doubt that whatever you put on an US cloud is being
| accessible by the US government, who can also decide to
| sanction you and deprive you from your ability to access the
| data yourself.
|
| Not having backups is entirely retarded, but also completely
| orthogonal.
| otterley wrote:
| The U.S. Government can't decrypt data for which it does not
| possess the key (assuming the encryption used is good).
| dboreham wrote:
| In theory. I'm very much happier to have my encrypted data
| _also_ not be available to adversaries.
| alwa wrote:
| Not sure "sane backup strategy" and "park your whole government
| in a private company under American jurisdiction" are mutually
| exclusive. I feel like I can think of a bunch of things that a
| nation would be sad to lose, but would be even sadder to have
| adversaries rifling through at will. Or, for that matter,
| extort favors under threat of cutting off your access.
|
| At least in this case you can track down said officials in
| their foxholes and give them a good talking-to. Good luck
| holding AWS/GCP/Azure accountable...
| atoav wrote:
| Well it is just malpractise. Even when I was an first semester
| art student I knew about the concept of off-site backups.
| Nux wrote:
| He may or may not have been right, but it's besides the point.
|
| The 3-2-1 backup rule is basic.
| StopDisinfo910 wrote:
| The issue here is not refusing to use a foreign third party.
| That makes sense.
|
| The issue is mandating the use of remote storage and not
| backing it up. That's insane. It's like the most basic amount
| of preparation you do. It's recommended to even the smallest of
| companies specifically because a fire is a risk.
|
| That's gross mismanagement.
| VirusNewbie wrote:
| why? I can upload backups to 20 third parties and no one has
| a prayer of getting access to those files. Are you under an
| impression that's challenging?
| lukevp wrote:
| How's that? Using encryption, which is known to have
| backdoors and is vulnerable to nation state cracking?
| vbezhenar wrote:
| It is much more likely and cheaper, that US marines will
| desant and capture your backup facility, than someone
| would break AES-128.
| senko wrote:
| Sending troops would be an act of war, and definitely not
| cheap.
|
| Stealing some encryption keys, just another Wednesday.
| xoa wrote:
| > _Using encryption, which is known to have backdoors and
| is vulnerable to nation state cracking?_
|
| WTF are you talking about? There are absolutely zero
| backdoors of any kind known to be in any standard open
| source encryption systems, and symmetric cryptography
| 256-bits or more is not subject to cracking by anyone or
| anything, not even if general purpose quantum computers
| are doable and prove scalable. Shor's algorithm applies
| to public-key not symmetric, where the best that can be
| done is Grover's quantum search for a square-root speed
| up. You seem to be crossing a number of streams here in
| your information.
| speedgoose wrote:
| Yeah let's fax all government data to the Trump administration.
| PunchyHamster wrote:
| cloud will also not back up your stuff if you configure it
| wrong so not sure how's that related
| redwood wrote:
| S. Korea has the most backward infosec requirements. It's wild
| Frost1x wrote:
| Having just visited South Korea last year, one thing that sort
| of caught me off guard was the lack of Google Maps or other
| major direction system. I wasn't aware but turns out anything
| considered "detailed mapping" infrastructure has to be ran
| stored and on South Korean soil, probably lots of other
| requirements. So you're stuck with some shotty local mapping
| systems that are just bad.
|
| There may be a point in time it made sense but high resolution
| detailed satellite imagery is plenty accessible and someone
| could put a road and basically planning structure atop it,
| especially a foreign nation wishing to invade or whatever
| they're protecting against.
|
| Some argument may be made that it would be a heavy lift for
| North Korea but I don't buy it, incredibly inconvenient for
| tourists for no obvious reason.
| WhyNotHugo wrote:
| Several other countries have similar requirements with
| regards to storing and serving maps locally.
|
| If you take a moment to think about it, what's weird is that
| so many countries have simply resorted to relying on Google
| Maps for everyday mapping and navigation needs. This has
| become such a necessity nowadays that relying on a foreign
| private corporation for it sounds like a liability.
| bmandale wrote:
| OSM is competitive with google maps in most places. Even if
| a person uses google maps, its inaccurate to say they
| "rely" on it when they could fail over to osm if google
| maps went down.
| Avamander wrote:
| Local mapping efforts and allowing Google Maps to operate
| aren't mutually exclusive though. I don't see how it's
| weird that people can choose which map app they use.
| jhasse wrote:
| In my experience Open Street Maps was very good there.
| luispauloml wrote:
| >So you're stuck with some shotty local mapping systems that
| are just bad.
|
| What made you think of them as bad? Could you be more
| specific? I use them almost daily and I find them very good.
| guillem_lefait wrote:
| I was there few months ago and I found them to be quite
| good too, both in coverage (shops, bus/metro networks) and
| accuracy. Obviously, not the apps I'm used to so & the
| language but otherwise, it was okay.
| tedk-42 wrote:
| A management / risk issue and NOT and engineering one.
| ookblah wrote:
| after the kakao fire incident and now this i struggle to
| understand how they got so advanced in other areas. this is like
| amateur hour level shit.
| Theodores wrote:
| I was smirking at this until I remembered that I have just one
| USB stick as my 'backup'. And that was made a long time ago.
|
| Recently I have been thinking about whether we actually need
| governments, nation states and all of the hubris that goes with
| it such as new media. Technically this means 'anarchism' with
| everyone running riot and chaos. But, that is just a big fear,
| however, the more I think through the 'no government' idea, the
| less ludicrous it sounds. Much can be devolved to local
| government, and so much else isn't really needed.
|
| South Korea's government have kind-of deleted themselves and my
| suspicion is that, although a bad day for some, life will go on
| and everything will be just fine. In time some might even be
| relieved that they don't have this vast data store any more.
| Regardless, it is an interesting story regarding my thoughts
| regarding the benefits of no government.
| poncho_romero wrote:
| Government is whatever has a monopoly on violence in the area
| you happen to live. Maybe it's the South Korean government.
| Maybe it's a guy down the street. Whatever the case, it'll be
| there.
| forinti wrote:
| What structure could possibly preclude backups? I've never seen
| anything that couldn't be copied elsewhere.
|
| Maybe it was just convenient to have the possibility of losing
| everything.
| Johnny555 wrote:
| I think that alluded to that earlier in the article:
|
| >However, due to the system's large-capacity, low-performance
| storage structure, no external backups were maintained --
| meaning all data has been permanently lost.
|
| I think they decided that their storage was too slow to allow
| backups?
|
| Seems hard to believe that they couldn't manage _any_
| backups... other sources said they had around 900TB of storage.
| An LTO-9 tape drive holds ~20TB uncompressed, so they could
| have backed up the entire system with 45 tapes. At 300MB /sec
| with a single drive, it would take them a month to complete a
| full backup, so seems like even a slow storage system should be
| able to keep up with that rate. They'd have a backup that's
| always a month out of date, but that seems better than no
| backup at all.
| themafia wrote:
| Too slow to allow batched backups. Which means you should
| just make redundant copies at the time of the initial save.
| Encrypt a copy and send it offsite immediately.
|
| If your storage performance is low then you don't need fat
| pipes to your external provider either.
|
| They either built this too quickly or there was too much
| industry corruption perverting the process and the government
| bought an off the shelf solution that was inadequate for
| their actual needs.
| r0ckarong wrote:
| My guess is someone somewhere is very satisfied that this data is
| now unrecoverable.
| gritzko wrote:
| In a world where data centers burn and cables get severed
| physically, the entire landscape of tradeoffs is different.
| fijiaarone wrote:
| What info needed to be destroyed and who did it implicate?
| crmd wrote:
| I would love to know how a fire of this magnitude could happen in
| a modern data center.
| AnimalMuppet wrote:
| Allegedly from replacing batteries.
| esskay wrote:
| Often poor planning or just lithium based batteries far too
| close to the physical servers.
|
| OVH's massive fire a couple of years ago in one of the most
| modern DC's at the time was a prime example of just how wrong
| it can go.
| nullable_bool wrote:
| I like to think that at least one worker was loafing on a project
| that was due the next day and there was no way it was going to
| get done. Their job was riding on it. They got drunk to embrace
| the doom that faces them, only to wake up with this news. Free to
| loaf another day!
| kupopuffs wrote:
| just his luck
| mekoka wrote:
| This is wild. Wilder would be to see that the government runs the
| same after the loss.
| WiggleGuy wrote:
| I was in Korea during the Kakao fire incident and thought it was
| astounding that they had no failovers. However, I thought it'd be
| a wake up call!
|
| I guess not.
| HeavyStorm wrote:
| Well I'll be. Backup is a discipline to not be taken lightly by
| any organization, _specially_ a government. Fire? This is backup
| 101: files should be backed up and copies should be physically
| apart to avoid losing data.
|
| There are some in this threading pointing out that this would be
| handled by cloud providers. That bad - you can't hope for
| transparent backup, you need to actively have a discipline over
| it.
|
| My fear is that our profession has become very amateurish over
| the past decade and a lot of people are vulnerable to this kind
| of threat.
| creakingstairs wrote:
| One of the workers jumped off a building. [1] They say the person
| was not being investigated for the incident. But I can't help but
| think he was a put under intense pressure to be scapegoat for how
| fucked up Korea can be in situations like this.
|
| To be some context on Korea IT scene, you get pretty good pay and
| benefits if you work for a big product company, but will be
| treated like dogshit inside subcontracting hell if you work
| anywhere else.
|
| [1]
| https://www.hani.co.kr/arti/society/society_general/1222145....
| jiggawatts wrote:
| I was the principal consultant at a subcontractor to a contractor
| for a large state government IT consolidation project, working on
| (among other things) the data centre design. This included the
| storage system.
|
| I noticed that someone had daisy-chained petabytes of disk
| through relatively slow ports _and_ hadn't enabled the site-to-
| site replication that they had the hardware for! They had the
| dark fibre, the long-range SFPs, they even licensed the HA
| replication feature from the storage array vendor.
|
| I figured that in a disaster just like this, the time to recover
| from the tape backups -- assuming they were rotated off site,
| which might not have been the case -- would have been six to
| eight weeks _minimum_ , during which a huge chunk of the
| government would be down. A war might be less disruptive.
|
| I raised a stink and insisted that the drives be rearranged with
| higher bandwidth and that the site-to-site replication be turned
| on.
|
| I was a screamed at. I was called unprofessional. "Not a team
| player." Several people tried to get me fired.
|
| At one point this all culminated in a meeting where the lead
| architect stood up in front of dozens of people and calmly told
| everyone to understand one critical aspect of his beautiful
| design: _No hardware replication!!!_
|
| (Remember: they had paid for hardware replication! The kit had
| arrived! The licenses were installed!)
|
| I was younger and brave enough to put my hand up and ask "why?"
|
| The screeched reply was: The on-prem architecture must be "cloud
| compatible". To clarify: He thought that hardware-replicated data
| couldn't be replicated to the cloud in the future.
|
| This was some of the dumbest shit I had ever heard in my life,
| but there you go: decision made.
|
| This. This is how disasters like the one in South Korea happen.
|
| In private organisations you get competent shouty people at the
| top insisting on a job done right. In government you get
| incompetent shouty people insisting that the job gets done wrong.
| Swenrekcah wrote:
| > In private organisations you get competent shouty people at
| the top insisting on a job done right. In government you get
| incompetent shouty people insisting that the job gets done
| wrong.
|
| Great post and story but this conclusion is questionable. These
| kinds of incompetences or misaligned incentives absolutely
| happen in private organisations as well.
| jiggawatts wrote:
| Much more rarely in my experience, having been at both kinds
| of organisations.
|
| There's a sort-of "gradient descent" optimisation in private
| organisations, established by the profit motive and the
| competitors nipping at their heels. There's no such gradient
| in government, it's just "flat". Promotions hence have a much
| weaker correlation with competence and a stronger correlation
| with nepotism, political skill, and willingness to
| participate in corruption.
|
| I've worked with may senior leaders in all kinds of
| organisations, but only in government will you find someone
| who is functionally illiterate and innumerate in a position
| of significant power.
|
| Obviously this is just a statistical bias, so there's overlap
| and outliers. Large, established monopoly corporations can be
| nigh indistinguishable from a government agency.
| foofoo12 wrote:
| This is extraordinarily loony shit. Someone designed a system
| like this without backups? Someone authorized it's use? Someone
| didn't scream and yell that this was bat and apeshit wacky level
| crazy? Since 2018? Christ almighty.
| hopelite wrote:
| Does anyone have an understanding of what the impact will be of
| this, i.e., what kind of government impact scale and type of data
| are we talking about here?
|
| Is this going to have a real impact in the near term? What kind
| of data are we're talking about being permanently lost?
| spawarotti wrote:
| There are two types of people: those who do backups, and those
| who will do backups.
| mmaunder wrote:
| Are we talking about actual portable thunderbolt 3 connected RAID
| 5 G-drive arrays with between 70 and 160TB of storage per array?
| We use that for film shoots to dump TB of raw footage. That
| G-Drive?? The math checks at 30GB for around 3000 users on a
| single RAID5 array. This would be truly hilarious if true.
| ChuckMcM wrote:
| Article comments aside, it is entirely unclear to me whether or
| not there was _no_ backups. Certainly no "external" backups, but
| potentially "internal" backups. My thinking is that not actually
| allowing backups and forcing all data there creates a prime
| target for the PRK folks right? I've been in low level national
| defense meetings about security where things like "you cannot
| backup off site" are discussed but there are often fire vaults[1]
| on site which are designed to withstand destruction of the
| facility by explosive force (aka a bomb) or fire or flood Etc.
|
| That said, people do make bad calls, and this would be an
| epically bad one, if they really don't have any form of backup.
|
| [1] These days creating such a facility for archiving an exabyte
| of essentially write mostly data are quite feasible. See this
| paper from nearly 20 years ago:
| https://research.ibm.com/publications/ibm-intelligent-bricks...
| nowittyusername wrote:
| I must say, at least for me personally when I hear about such
| levels of incompetence it rings alarm bells in my head making me
| think that maybe intentional malice was involved. Like someone
| higher up had set up the whole thing to happen in such a matter
| because there was a benefit to this happening we are unaware of.
| I think this belief maybe stems from lack of imagination on how
| really stupid humans can get.
| quantumsequoia wrote:
| Most people overestimate the prevalence of malice, und
| underestimate the prevalence of incompetence
| vayup wrote:
| A lot of folks are arguing that the real problem is that they
| refused to use US cloud providers. No, that's not the issue. It's
| a perfectly reasonable choice to build your own storage
| infrastructure if it is needed.
|
| But the problem is they sacrificed "Availability" in pursuit of
| security and privacy. Losing your data to natural and man-made
| disasters is one of the biggest risks facing any storage
| infrastructure. Any system that cannot protect your data against
| those should never be deployed.
|
| "The Interior Ministry explained that while most systems at the
| Daejeon data center are backed up daily to separate equipment
| within the same center and to a physically remote backup
| facility, the G-Drive's structure did not allow for external
| backups."
|
| This is not a surprise to them. They had knowingly accepted the
| risk of infrastructure being destroyed by natural and man-made
| disasters. I mean, WTF!
| john-tells-all wrote:
| This is literally comic. The plot of the live action comic book
| movie "Danger: Diabolik" [1] has a segment where the a country's
| tax records are destroyed, thus making it impossible for the
| government to collect taxes from its citizens.
|
| [1] https://en.wikipedia.org/wiki/Danger:_Diabolik
___________________________________________________________________
(page generated 2025-10-05 23:00 UTC)