[HN Gopher] Iron Mountain: It's Time to Talk About Hard Drives
___________________________________________________________________
Iron Mountain: It's Time to Talk About Hard Drives
Author : severine
Score : 73 points
Date : 2024-09-10 19:04 UTC (3 hours ago)
(HTM) web link (www.mixonline.com)
(TXT) w3m dump (www.mixonline.com)
| alchemist1e9 wrote:
| Does it mean LTO tape for the win then?
|
| We're about to start a project to build an LTO-9 based in-house
| backup system. Any suggestions for DIY Linux based operation
| doing it "correctly" would be appreciated. Preliminary planning
| is to have one drive system on in our primary data center and
| another offsite at an office center where tapes are verified
| before storage in locked fireproof storage cabinet. Tips on good
| small business suppliers and gear models would be great help.
| antisthenes wrote:
| What's your budget?
| alchemist1e9 wrote:
| $50K if needed but it doesn't look to need that. 2PB initial
| data. predicted 1PB/year with around 10-30%/year rate of
| growth of rate of growth (acceleration?)
| akira2501 wrote:
| > fireproof storage cabinet.
|
| Nothing is fire proof. Is the cabinet "fire suppression system
| liquid" proof?
|
| > Tips on good small business suppliers and gear models would
| be great help.
|
| Hire an auditor would be my advice. Every business is
| different.
|
| I am, just now, having flashbacks of when I was in a SOX
| environment and had to regularly contract with them... and
| while the experience can be somewhat unpleasant I've often
| found good auditors to be extremely knowledgeable about
| solutions and their practical implementation considerations.
| hinkley wrote:
| Pretty sure the pyramids at Giza count as fireproof boxes.
| But that's out of most people's price range.
| burnished wrote:
| Not if the fire is big enough.
| hinkley wrote:
| When a catastrophic event is catastrophic enough, your
| other problems cease to be your problem.
| akira2501 wrote:
| They're not fully sealed. There are two shafts which
| connect the King's chamber to the exterior of the pyramid.
| The lower ritual congregation area is not fully sealed off
| from the upper chambers either. Which means bats are a
| constant problem in pyramids.
|
| You could easily get a fire going inside one.
| hinkley wrote:
| The advice I got long ago from an IT guy was: if you wait long
| enough, tape will be on top again.
|
| That was a long time ago but I've peeked in at backup systems
| in the intervening years and it does seem to hold true over
| time.
|
| But it really depends how much data you have. My ex dropped a
| single HDD in a safety deposit box at CoB, N times per week and
| fetched back the oldest disk. I don't think she ever said how
| many were in there but I doubt it was more than three. I think
| the CTO took one home with him once per week.
|
| The silly thing about most of this set up is that the office,
| the bank, and the data center were all within half a kilometer
| of each other. If something bad happened to that part of town
| they only had the infrequent offsite backup.
| katbyte wrote:
| 20tb and under is easy. 500tb is hard
| exe34 wrote:
| the sort of bad that would render all three unreadable would
| probably melt the rest of the infrastructure the company
| relies on for business anyway.
| marcosdumay wrote:
| Every time I've looked, tapes were on top "again" for large
| scale archival. And I've been looking for ~20 years by now.
|
| I don't get where people get the impression that X was at the
| top right before tapes got that last innovation (where X here
| is most often HDDs, but not always). But that's always the
| impression, and tapes are always on top.
|
| People also have been working with 3D phase change drives
| since the 90s. Those always promise to replace tapes. But
| nobody ever got them robust enough to leave the labs.
| WaitWaitWha wrote:
| Q: Why not archival M-Disc?
| actionfromafar wrote:
| A: Are there any "real" M-Discs for purchase anymore? (I.e.
| not just rebranded regular dye-discs.)
| netrap wrote:
| Possibly in DVDR.
| alchemist1e9 wrote:
| Amount of data makes it less realistic. We have around 2PB
| data currently and expect to grow around 1PB next year with
| maybe 10-30% annual growth rate.
| bluedino wrote:
| Tapes are fun. You can fit a petabyte of data in a bankers box!
|
| The problem quickly becomes:
|
| - Do we have a drive that can read this tape?
|
| - Do we have server we can connect it to?
|
| - Do we have storage we can extract it to? (go ask your
| internal IT team for 10TB of drivespace...)
|
| - What program did we create this tape with? Backup Exec,
| Veritas, ArcServe, SureStore
|
| - You have the encryption keys, right?
|
| - How much of this data already exists on the previous months
| backup?
|
| - Who's going to pay for the storage to move it to Glacier/etc?
|
| -How long is it going to take to upload?
| aftbit wrote:
| If you are having trouble getting 10TB of disk space from IT,
| you have bad IT. Not saying that's uncommon or anything, but
| 10TB fits on one external hard drive for $300 from Best Buy,
| or less than $1000/mo using EBS on AWS if you need some
| better guarantees and are all-in on the cloud.
| gosub100 wrote:
| Make sure the bandwidth exists to keep up with the write speed
| of the LTO drive. For instance, the write speed for LTO-6
| (which I own as a hobbyist) is around 300MB/s, but line speed
| of gigabit Ethernet is about 100MB/s. Translate those numbers
| to LTO-9 and make sure that the NAS, network, or local storage
| can keep up. It's not a deal-breaker to underflow the drive,
| but it causes the tape to stop, rewind, and re-buffer (called
| shoe-shining) which takes more time and causes unnecessary wear
| on the drive and cartridges.
| adrian_b wrote:
| LTO-9 tapes can be easily found on Amazon in many countries,
| made by IBM, HP, Quantum or Fuji.
|
| The vendor does not matter, whichever happens to be cheaper at
| the moment is fine.
|
| For the tape drives, the internal drives can be cheaper by
| around 10%, but I prefer the tabletop drives, because they are
| less prone to accumulate dust, especially if you switch them on
| only when doing a backup or a retrieval. The tape drives have
| usually very noisy fans, because they are expected to be used
| in isolated server rooms.
|
| I believe that the cheapest tape drives from a reputable
| manufacturer are those from Quantum. I have been using a
| Quantum LTO-7 tape drive for about 7 or 8 years and I have been
| content with it. Looking now at the prices, it should be
| possible to find a tabletop LTO-9 drive for no more than $5000.
| Unfortunately, the prices for tape drives have been increasing.
| When I have bought an LTO-7 tabletop drive many years ago it
| was only slightly more than $3000.
|
| The tapes are much cheaper and much more reliable than hard
| disks, but because of the very expensive tape drive you need to
| store a few hundred TB to begin to save money over hard disks.
| You should normally make at least two copies of any tape that
| is intended for long-term archiving (to be stored in different
| places), which will shorten the time until reaching the
| threshold of breaking even with HDDs.
|
| Even if there are applications that simulate the existence of a
| file system on a tape, which can be used even by a naive user
| to just copy files on a tape, like copying files between disks,
| they are quite slow and inefficient in comparison to just using
| raw tape commands with the traditional UNIX utility "mt".
|
| It is possible to write some very simple scripts that use "mt"
| and which allow the appending of a number of files to a tape or
| the reading of a number of consecutive files from a tape,
| starting from the nth file since the beginning of a tape. So if
| you are using only raw "mt" commands, you can identify the
| archived files only by their ordinal number since the beginning
| of the tape.
|
| This is enough for me, because I prepare the files for backup
| by copying them in some directory, making an index of that
| directory, then compressing it and encrypting it. I send to the
| tape only encrypted and compressed archive files, so I disable
| the internal compression of the tape drive, which would be
| useless.
|
| I store the information about the content of the archives
| stored on tapes (which includes all relevant file metadata for
| each file contained in the compressed archives, including file
| name, path name, file length, modification time, a hash of the
| file content) in a database. Whenever I need archived data, I
| search the database, to determine that it can be found, for
| instance in tape 63, file 102. Then I can insert the
| corresponding cartridge in the drive and I give the command to
| retrieve file 102.
|
| I consider much better the utility "mt" of FreeBSD than that of
| Linux. The Linux magnetic drive utilities have seen little
| maintenance for many years.
|
| Because of that, when I make backups or retrievals they go to a
| server that runs FreeBSD, on which the SAS HBA card is
| installed. When a tabletop drive is used, the SAS HBA card must
| have external SAS connectors, to allow the use of an
| appropriate cable. I actually reboot that server into FreeBSD
| for doing backups or retrievals, which is easy because I boot
| it from Ethernet with PXE, so I can select remotely what OS to
| be booted. One could also use a FreeBSD VM on a Linux server,
| with pass-through of the SAS HBA card, but I have not tried to
| do this.
|
| My servers are connected with 10 Gb/s Ethernet links, which
| does not differ much from the SAS speed, so they do not slow
| much the backup/retrieval speed. I transfer the archive files
| with rsync over ssh. On slow computers and internal networks
| one can use rsync without ssh. I give the commands for the tape
| drive from the computer that is backed up, as one line commands
| executed remotely by ssh.
|
| The archive that is transferred is stored in a RAMdisk before
| being written on the tape, to ensure that the tape is written
| at the maximum speed. I write to the tape archive files that
| have usually a size of up to about 60 GB (I split any files
| bigger than that; e.g. there are BluRay movies of up to 100
| GB). The server has a memory of 128 GB, so I can configure on
| it a RAMDdisk of up to 80 GB without problems. This method can
| be used even with a slow 1 Gb/s or 2.5 Gb/s network, but then
| uploading a file through Ethernet would take much more time
| than writing or reading the tape.
|
| There is one weird feature of the raw "mt" commands, which is
| poorly documented, so it took me some time to discover it,
| during which I have wasted some tape space.
|
| When you append files to a partially written tape, you first
| give a command to go to the end of the written part of the
| tape. However, you must not start writing, because the head is
| not positioned correctly. You must go 2 file marks backwards,
| then 1 file mark forwards. Only then is the head positioned
| correctly and you can write the next archived file. Otherwise
| there would be 1 empty file intercalated at each point where
| you have finished appending a number of files and then you have
| rewound the tape and then you have appended again other files
| at the end.
| netrap wrote:
| Only Sony or Fuji actually make tapes. The rest are
| rebranded.
| adrian_b wrote:
| True, but the rebranded tapes are frequently cheaper than
| Fuji or Sony.
| alchemist1e9 wrote:
| A lot very interesting details in your reply - thanks. I have
| this question:
|
| If you aren't budget constrained today and had to set it all
| up again. What would you do?
|
| While I'm a Linux guy, I'll happily run BSDs when
| appropriate, like for pfSense, and if it really has better mt
| tools or driver for LTO-9 drives due to the
| culture/contributors being more old school, then I'd just
| grab a 1U server to dedicate for it run a BSD and attach the
| drive to that.
|
| You seem to have extensive practical hands on experience and
| while I was doing tapes 20 years ago this will be first time
| I'm hands on again with it since then. So I need to research
| most reliable drive vendors and state of kernel drivers and
| tools, just as you are alluding to.
|
| Pretend you have $50K if needed (doubt it). 2PB existing
| data, 1PB/year targeted rate, probably 10-20%/year
| acceleration on that rate. with a data center rack location,
| 20Gb/s interconnect via bonded 10Gb NICs to storage servers
| (45drives storinators) and then an office center
| cabinet/rack/desk (your choice) and will put a tape drive
| holding at least 8 tapes in data center, planning for worst
| case of 100TB a month and data center visits to swap in new
| tapes shouldn't be too frequent. Any details on what you
| would do would be interesting.
| adrian_b wrote:
| Like I have said, it is not necessary to dedicate a full-
| time FreeBSD server for this, you can use either a Linux
| server that is rebooted temporarily in FreeBSD or a FreeBSD
| virtual machine on the Linux server.
|
| Around $5000 to $6000 should be enough for a LTO-9 tabletop
| tape drive plus a suitable SAS HBA card and SAS cable. The
| card must have matching SAS connectors and SAS speed with
| the tape drive.
|
| More money will not bring anything extra until a much
| higher amount is reached, which would be enough to buy a
| tape autoloader/library, which would eliminate the
| necessity for a human to insert and remove the cartridges
| into the tape drive when needed. I am not sure if $50K is
| enough for a tape autoloader.
|
| Tape autoloaders/libraries are worthwhile only for very big
| organizations where the amount of data that is continuously
| written or read to or from the tapes is very large. For a
| small business or for an individual a tape autoloader is
| certainly not worthwhile, because the tape drive will be in
| use at most a small fraction of every day.
|
| 1 PB/year is less than 3 TB/day. This can be written on a
| single tape in a little more than 2 hours. Even with a
| simple non-pipelined implementation of the file uploading
| with the writing on the tape, the backup can be done in
| less than 4 hours. Even writing 2 copies can be done in
| less than 8 hours. The backup can be done mostly or
| completely overnight.
|
| For a much bigger amount of data one could buy several tape
| drives, before starting to think about an autoloader. Also
| it is possible to pipeline the network transfers with the
| tape writing, for a backup speed higher by around 50%.
|
| If money would not be a problem and if the data needs to be
| archived for a long term, so that multiple copies are
| desirable, I would buy 2 tape drives, to be able to write 2
| copies simultaneously.
|
| This would also halve the time for archiving the initial 2
| PB of existing data, which will take several months, so a
| speed-up would be desirable. Having 2 drives will also
| increase the reliability, as the system will continue to
| work if one becomes defective.
|
| With only 3 TB written per day, a LTO-9 tape, which has a
| capacity of 18 TB, will be enough for 6 days.
|
| So unless a backup must be restored, the operator would
| need to change the tape only once per week.
|
| This is a moderate amount of data, easy to handle with a
| single drive, even if two are preferable for redundancy and
| for higher speed.
|
| I do not understand your reference to a "a tape drive
| holding at least 8 tapes in data center". If you mean an
| autoloader, from what you describe it does not seem that
| the very big expense for an autoloader would be justified.
|
| The LTO tapes are best stored in suitcases that can contain
| 20 cartridges, i.e. when using LTO-9 that is 360 TB.
| Therefore 3 suitcases store more than 1 PB, i.e. a year of
| data according to your example. The suitcases should be
| stored in a secure safe or cabinet. They are usually made
| to be stackable.
|
| I have assumed that your 1 PB is of already compressed
| data. If the data is compressible than the requirements for
| the usage time of the drives and for the storage volume
| would be much smaller.
|
| I have forgotten to mention that after I compress and
| encrypt the archived files, I add redundancy with a Reed-
| Solomon code, e.g. with the par2 program. If I choose e.g.
| a redundancy of 5%, then a file retrieved from the magnetic
| tape could have defects of up to 5% of its size, while the
| original data could still be extracted from it.
| gosub100 wrote:
| Great post. You might be able to elide the RAM disk in lieu
| of the "mbuffer" command. My script uses a combination of dd
| | pv | mbuffer | mt. I omitted the options because I don't
| remember any of them. I personally use dd of an ext4
| filesystem-on-file that is exactly the size of what will fit
| on tape. This was simply because I couldn't figure out how to
| reliably advance the tape head or how to continue a write
| from one tape to another.
| antisthenes wrote:
| What's the scenario where you cannot take the old 1990's hard
| drive and back up its data in multiple cloud service providers
| cold storage (Azure/AWS/GCP) and have to keep the obsolete
| physical media on hand?
|
| I'm struggling to understand why these miles of shelves filled
| with essentially hardware junk haven't been digitized at the time
| when this media worked and didn't experience read issues.
|
| The article doesn't really provide an explanation for this other
| than incompetence and the business biting off more than it can
| reasonably chew. I'd be furious if I paid for a service that
| promised to archive my data, and 10-15 years later told me 25% of
| it was unreadable. I mean it's not like it was a surprise either.
| These workflows became digital 2-3 decades ago. There was plenty
| of time to prepare and convert this.
|
| That's kind of what I'm paying you for.
|
| As always, seems like the simple folk of /r/datahoarder and other
| archivist communities are more competent than a legacy industry
| behemoth.
| akira2501 wrote:
| > the business biting off more than it can reasonably chew
|
| It's hoarding behavior. They paid "a lot" of money for it, have
| no idea how to further exploit it, but can't shake the feeling
| that it might be massively valuable one day.
|
| The only difference is they pay someone to hold their hoard for
| them.
| eitally wrote:
| Alternatively ... they are forced to maintain it for
| compliance reasons, especially when it comes to healthcare,
| finance and other regulated industries (defense, in
| particular). This even applies to manufacturing companies
| assembly medical devices and defense products: all the data
| about the supply chain, the engineering designs & changes,
| the manufacturing and quality testing, and shipment needs to
| be kept for XX years and is subject to both audit by
| regulatory agencies and to legal discovery.
| akira2501 wrote:
| Those are all things which can be printed out and stored in
| alternative forms and possibly even recreated from other
| data. It's also the case that much of that data will never
| be permanently at rest and so several archive copies of the
| data exist.
|
| Recordings of performances are an entirely different
| category of artefact.
| surgical_fire wrote:
| I mean, even if by contract they were supposed to store
| physical media with the backups, it is still horrible
| incompetence to not have the same data backed up twice, and
| from time to time test the disks for failure to rebuild the
| backup from one of the copies.
|
| It would be extremely unlikely for both disks to fail together.
|
| What I'm describing is the bare minimum. This is their job, by
| all accounts. Amazing.
| 0cf8612b2e1e wrote:
| It depends on what specifically Iron Mountain is selling you. A
| place to store your physical data device or are they promising
| to keep your data available? The former sounds cheaper and
| easier for Iron Mountain. Given Iron Mountain started in the
| 1950s, redundantly backing up customer data was infeasibly
| expensive for most of the company's lifetime.
| Cheer2171 wrote:
| > I'd be furious if I paid for a service that promised to
| archive my data, and 10-15 years later told me 25% of it was
| unreadable.
|
| The article is very vague on this, but I thought this company
| was first doing something like a bank safety deposit box. Send
| us your media in whatever format and we will keep it secure in
| a climate controlled vault. They don't offer to archive your
| data, they offer to store your media. Now it seems they pivoted
| to archiving data. This is an ad for their existing media
| storage clients to buy their data archive service:
|
| > Iron Mountain would like to alert the music industry at large
| to the fact that, even though you may have followed recommended
| best practices at the time, those archived drives may now be no
| more easily playable than a 40-year-old reel of Ampex 456 tape.
| eitally wrote:
| They did make this pivot several years ago, with big upsell
| and a huge internal product advancement to offer housebuilt
| eDiscovery for lots of data types. I was at Google Cloud when
| they did the first big deal around this a few years ago:
| https://www.ironmountain.com/resources/blogs-and-
| articles/f/...
| tecleandor wrote:
| Also where you don't render the tracks pre and post processing
| and leave them aside to the ProTools project files. I don't
| know who expects to open a ProTools project with a bunch of
| unknown plugins after some years have passed...
| andrewf wrote:
| Back in the 2000s the Australian government provided software
| that ran on Windows to prepare and submit your personal tax
| return. I used to archive my tax return by preparing it in a
| Windows VM, then storing the whole VM image.
| chuckadams wrote:
| > I'm struggling to understand why these miles of shelves
| filled with essentially hardware junk haven't been digitized at
| the time when this media worked and didn't experience read
| issues.
|
| Because it's time consuming and expensive and the format you
| digitize it into is also in danger of decaying into oblivion.
| See also: TFA.
| bob1029 wrote:
| Iron mountain also provides services like source code escrow.
|
| With 2 parties involved in the data, you may want to impose
| additional restrictions regarding how and when it can be
| replicated. The party requesting escrow clearly has interest in
| the source being as durable as possible, but the party
| providing the source may not want it to be made available
| across an array of dropbox-style online/networked systems just
| to accommodate an unlikely black swan event.
|
| A compromise could be to require that the source reside on the
| original backup media with multiple copies and media types
| available.
| akira2501 wrote:
| > It may sound like a sales pitch, but it's not; it's a call for
| action
|
| Your entire article sounds like a sales pitch. Your solution is,
| well, it's bad, but trust us, we can maybe recover it anyways.
| Otherwise your article fails to convey anything meaningful.
| derefr wrote:
| No, the call-to-action being referenced in the article is "stop
| archiving to hard drives" (and use tape instead, every other
| industry does.)
| ganoushoreilly wrote:
| I think it is or at least tries to be more than that. Not
| only stop archiving to HD's but understand the dependency
| requirements which is a whole secondary problem.
| natch wrote:
| The last time I used a tape it broke right off the reel the
| first time I used it. Good name brand tape drive, good name
| brand tape. I felt pretty burned by that experience after the
| money spent and the result.
| MisterTea wrote:
| Makes me wish we didn't stop advancing optical media technology
| to where we have cheap and reliable archival quality 1TB discs
| for a few bucks each. I guess LTO is the best option for
| personally controlled archival.
| 0cf8612b2e1e wrote:
| We haven't, but sadly the technology is locked to big tech.
|
| Microsoft has demoed some cool technology where they store data
| in glass, Project Silica. Sadly, it seems unlikely this will
| ever be available to consumers. One neat aspect of the design
| is that writing data is significantly higher power than
| reading. So you can keep your writing devices physically
| separated from the readers and have no fear that malicious code
| could ever overwrite existing data plates.
|
| Some blurbs Project Silica is developing the
| world's first storage technology designed and built from the
| media up to address humanity's need for a long-term,
| sustainable storage technology. We store data in quartz glass:
| a low-cost, durable WORM media that is EMF-proof, and offers
| lifetimes of tens to hundreds of thousands of years. This has
| huge consequences for sustainability, as it means we can leave
| data in situ, and eliminate the costly cycle of periodically
| copying data to a new media generation. We're re-
| thinking how large-scale storage systems are built in order to
| fully exploit the properties of the glass media and create a
| sustainable and secure storage system to support archival
| storage for decades to come! We are co-designing the hardware
| and software stacks from scratch, from the media all the way up
| to the cloud user API. This includes a novel, low-power design
| for the media library that challenges what the robotics and
| mechanics of archival storage systems look like.
|
| https://www.microsoft.com/en-us/research/project/project-sil...
| actionfromafar wrote:
| Now this is proper Sci-Fi tech! Data crystals, like 1960s
| Star Trek!
| yellow_postit wrote:
| I'm partial to their DNA work: https://www.microsoft.com/en-
| us/research/project/dna-storage...
| adrian_b wrote:
| That would be great, but after 7 years since the initial
| announcement it does not seem to be any closer of a
| commercial product.
| 0cf8612b2e1e wrote:
| Why would they sell it directly? Works better if they can
| advertise their one of a kind, super stable, cloud specific
| data archival solution that nobody else can replicate. Or
| not even advertise it, but maintain lower storage costs per
| byte relative to AWS or Google.
|
| As far as I know, the technology behind Amazon Glacier has
| never been shared. Glass disks could eventually be backing
| the Microsoft equivalent.
| Twirrim wrote:
| Optical media is neat, but has a number of drawbacks when it
| comes to large scale operations.
|
| What you're talking about already sort of exists, albeit media
| hadn't reached "cheap" yet, because the manufacturing scale
| wasn't there. People weren't interested enough in it. Archival
| Disc was a standard that Sony and Panasonic produced,
| https://en.wikipedia.org/wiki/Archival_Disc. Before the
| standard was retired you could by gen3 ones with 5.5TB of
| capacity, https://pro.sony/ue_US/products/optical-disc-archive-
| cartrid...
|
| LTO tape was already at 15TB by the time their 300GB Discs came
| out, and reached 45TB capacity 3 years ago. Tape is still leaps
| and bounds ahead of anything achievable in optical media _and_
| isn 't write-once. (https://en.wikipedia.org/wiki/Linear_Tape-
| Open)
|
| Part of the problem is you can't just store and forget, you
| have to carry out fixity checks on a regular basis
| (https://blogs.loc.gov/thesignal/2014/02/check-yourself-
| how-a...). Same thing as with your backups, backups that don't
| have restores tested aren't really backups, they're just
| bitrot. You want to know that when you go to get something
| archived, it's actually there. That means you're having to load
| and validate every bit of media on a very regular basis,
| because you have to catch degradation before it's an issue.
| That's probably fine when you're talking a handful of discs,
| but it doesn't scale that well at all.
|
| The amount of space that it takes for the drives to read the
| optical disc, the machinery to handle the physical automation
| of shuffling discs around etc. combined with the costs of it,
| just make no sense compared to the pre-existing solutions in
| the space. You don't get the effective data density (GB/sq
| meter) you'd need to make it make sense, nor do the drives come
| at any kind of a price point that could possibly overcome those
| costs.
|
| To top it all off, the storage environment conditions of
| optical media isn't really any different from Tape, except
| maybe slightly less sensitive to magnetic interference.
| netrap wrote:
| Unfortunately recordable optical is on it's way out. Sony
| recently slashed the staff at the Japan plant that makes BD-R's
| (BD-R XL's). Still CMC makes CDR, DVDR, BDR though.
| kevin_thibedeau wrote:
| 25 years ago, Kodak was working on a silver halide tape media
| that was supposed to be good for 100+ years. It sadly withered
| and never came to market.
|
| https://group47.com/Introduction_to_DOTS_WEB_11-23.pdf
| nayuki wrote:
| According to their video (
| https://vimeo.com/502475794/ffbfb82b15 ), the company
| patented bit plane image storage. What the heck? That is so
| obvious and shouldn't be patentable.
|
| On a side note, they keep touting how robust their data
| archival solution is. But I have my doubts. For example, if
| an image has a big patch of 0 or 1 bits, then it might be
| impossible to accurately align the bit positions
| ("reclocking"); this is the same issue with QR codes and why
| they have a masking (scrambling) technique. Another problem
| is that their format doesn't seem to mention error correction
| codes; adding Reed-Solomon ECC is an essential technique in
| many, many popular formats already.
| simonw wrote:
| My understanding is that the only reliable way of long-term
| digital archival storage is to refresh the media you are storing
| things on every few years, copying the previous archives to the
| fresh storage.
|
| Since storage constantly gets cheaper, 100GB first stored in 2001
| can be stored on updated media for a fraction of that original
| cost in 2024.
| abracadaniel wrote:
| Pretty much. You see hobbyists getting data off of 30+ year old
| hard drives for the novelty of it, but I can't imagine relying
| on that as a preservation copy. Optical media rots, magnetic
| media rots and loses magnetic charge, bearings seize, flash
| storage loses charge, etc. Entropy wins, sometimes much faster
| than you'd expect.
| mercurialuser wrote:
| There are several articles about film preservation in digital
| format. Every X years all the data is "upgraded", from LTOn to
| LTOn+1 or +2.
|
| So it may sound like a sales pitch but I consider it more a
| warning notice
| esafak wrote:
| That means the hardware _and_ the file format.
| Clamchop wrote:
| They have lots of problems:
|
| 1. Incomplete copies with missing dependencies. 2. Old software
| and their file formats with a poor virtualization story. 3. Poor
| cataloging. 4. Obsolete physical interfaces, file systems, etc.
| 5. Long-term cold storage on media neither proven nor marketed
| for the task.
|
| Managing archives is just a cost center until it isn't, and it's
| hard to predict what will have value. The worst part of this is
| that TFA discusses mostly music industry materials. Outside
| parties and the public would have a huge interest in preserving
| all this, but of course it's impossible. All private,
| proprietary, copyrighted, and likely doomed to be lost one way or
| another.
|
| Oh well.
| cookiengineer wrote:
| Related documentary that comes to mind: Digital Amnesia (2014)
| [1]
|
| It broke my heart seeing those librarians in disbelief when
| their national library was sold off to the highest bidder. When
| they said "It seems our country does not value our own culture
| anymore".
|
| Books lasted hundreds of years. Good luck trying to read a
| floppy from the 90s, or even DVDs that are already beyond their
| lifetime and are a very recent medium.
|
| It gets worse when you read the fine print of the SSD
| specifications, wherein they state that an SSD may lose all its
| data after 2 weeks without power, and data retention rates are
| at less than 99%, meaning they will degrade after the first
| year of use. And don't get me started on SMR HDDs, I lost
| enough drives already :D
|
| Humanity has a backup problem. We surely live in Orwellian
| times because of it.
|
| [1] https://youtube.com/watch?v=NdZxI3nFVJs
| lizknope wrote:
| Tape doesn't last forever either.
|
| https://en.wikipedia.org/wiki/Linear_Tape-Open#Generations
|
| LTO-1 started in 2000 and the current LTO-9 spec is from 2021.
| But it only has backwards compatibility for 1 to 2 generations.
| You can't read an LTO-6 tape in an LTO-9 drive.
|
| https://en.wikipedia.org/wiki/Sticky-shed_syndrome
|
| > Sticky-shed syndrome is a condition created by the
| deterioration of the binders in a magnetic tape, which hold the
| ferric oxide magnetizable coating to its plastic carrier, or
| which hold the thinner back-coating on the outside of the
| tape.[1] This deterioration renders the tape unusable.
|
| Stiction Reversal Treatment for Magnetic Tape Media
|
| https://katalystdm.com/digital-transformation/tape-transcrip...
|
| > Stiction can, in many cases, be reversed to a sufficient
| degree, allowing data to be recovered from previously unreadable
| tapes. This stiction reversal method involves heating tapes over
| a period of 24 or more hours at specific temperatures (depending
| on the brand of tape involved). This process hardens the binder
| and will provide a window of opportunity during which data
| recovery can be performed. The process is by no means a permanent
| cure nor is it effective on all brands of tape. Certain brands of
| tape (eg. Memorex Green- see picture below) respond very well to
| this treatment. Others such as Mira 1000 appear to be largely
| unaffected by it.
|
| Data migration and periodic verification is the answer but it
| requires more money to hire people to actually do it.
|
| I've got files from 1992 but I didn't just leave them on a 3.5"
| floppy disk. They have migrated from floppy disk -> hard drive ->
| PD phase change optical disk -> CD-R -> DVD-R -> back to hard
| drive
|
| I verify all checksums twice a year and have 2 independent
| backups.
| adrian_b wrote:
| While tape does not last forever, the LTO tapes are specified
| for at least 30 years.
|
| The more serious problem is as you say that the older drives
| become obsolete. Even so, if you start using an up-to-date LTO
| format you can expect that suitable new tape drives will be
| available for buying at least 10 years in the future.
|
| For HDDs, the most that you can hope is a lifetime of 5 years,
| if you buy the HDDs with the longest warranties.
| lizknope wrote:
| I absolutely agree that the average tape will last longer
| than the average hard drive.
|
| I've got 30 hard drives in use right now and at least 10 are
| older than 5 years. A few are over 7 years old. I've also had
| hard drives die in less than a week.
|
| Even if the data is on tape I want to emphasize that the tape
| needs to be periodically read and verify that the data is
| still readable and correct. Assuming the data is stable for
| 30 years and you can just leave it there is a dumb idea
| unless you didn't care about the data in the first place.
| Spooky23 wrote:
| I did some work for a place that had 30 year retention
| requirements for lots of data. (And indefinite for others)
|
| Iirc, the goal was to turnover to new tape media every 8-10
| years.
| bogwog wrote:
| This article is too vague. It sounds like they're talking about
| the physical drive not working, but they're giving examples where
| you can't playback because you need to install the correct old
| software, plugins, etc... Which doesn't have anything to do with
| hard drives.
|
| So what's actually wrong with hard drives for archival? Do they
| deteriorate? Do they "rot" like DVDs/blurays/etc have been known
| to do? Or is this just an ad for their archival service?
| wmf wrote:
| Hard drives are known to suffer "sticktion" where the heads get
| stuck to the platter and either the drive won't spin up or it
| spins up and the heads damage the platters. I imagine hard
| drives could also have bad capacitors but I haven't heard of
| that happening.
| mystified5016 wrote:
| Magnetic HDDs do suffer bit rot, yes. But perhaps more
| importantly the mechanisms suffer physical failure over time.
| You can't just pop the platters into a new drive, even if you
| had an identical model.
|
| That's really the main disadvantage of hard drives: the media
| is permanently coupled to the drive. If your tape drive fails,
| you can just pop the tape into a working drive and still get
| your data back.
| kkfx wrote:
| People should learn a thing: data are not tied to the physical
| media hosting them, like words on paper, and the sole way to
| preserve data is migrating them from a physical support to
| another regularly, also converting their formats sometimes,
| because things changes and an old format could end up unreadable
| in the future.
|
| We can't preserve bits like books.
| Thrymr wrote:
| > We can't preserve bits like books.
|
| The only reason we have any copies of "books" (i.e. long
| written works) from the ancient world is that they were
| painstakingly copied over centuries from one medium to another,
| by hand for most of that time.
| hulitu wrote:
| > Of the thousands and thousands of archived hard disk drives
| from the 1990s that clients ask the company to work on, around
| one-fifth are unreadable
|
| Some 25 years ago, the hardest part in booting some Apollo
| workstations, was to make hard drives spin.
| bell-cot wrote:
| Policy at $Job - all _important_ data is backed up to a rotation
| of high-quality hard drives. Which are stored off-site, powered
| down. Every N weeks, each one of them is powered up (in an off-
| line system) and checked - both with the SMART long test, and
| `zfs scan` (which verifies ZFS 's additional anti-bit-rot
| checksums for the data).
|
| Yes, it's a bit of a PITA. OTOH, modern HD's are huge, so a
| relative few are needed. And we've lost 0 bits of our off-site
| data in our >25 years of using that system.
| nayuki wrote:
| I had to do a double take on this, as I associate Iron Mountain
| as the brand that shreds papers and hard drives as their most
| common service.
| RobRivera wrote:
| The second M doesn't help.
|
| Almost like the title was purposely crafted to mislead you to
| draw eyeballs.
___________________________________________________________________
(page generated 2024-09-10 23:01 UTC)