[HN Gopher] Show HN: Automedia - a tool for managing bitrot and ...
___________________________________________________________________
Show HN: Automedia - a tool for managing bitrot and formats in
media libraries
Author : mmastrac
Score : 52 points
Date : 2022-05-08 13:31 UTC (9 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| arendtio wrote:
| Just to give you an early warning: Someone else seems to have a
| project with a similar name. At least the Arch Linux User
| Repositories (AUR) have this package and it seems to be a
| different project:
|
| https://aur.archlinux.org/packages/automedia-git
| mmastrac wrote:
| Hey all, a tiny project I wrote for myself over the last couple
| of months.
|
| I've been ripping my old physical audio and video media to a
| server running locally at home and have been worried about
| bitrot. I built this tool to check a large media library for
| correctness (by test-decoding it with FFMPEG), and to prevent
| future bitrot by automatically adding per-directory PAR2 files.
|
| The tool has another feature to transcode your entire library to
| a given format - I'm using this to keep a USB key in my car
| filled with AAC 64k files for listening when Spotify is out of
| range or I'm too lazy to connect via Bluetooth.
|
| It's designed to be somewhat easy to extend with media operations
| that consume a directory at a time. For example, you could add or
| check music metadata using a third-party service like
| MusicBrainz.
| Wowfunhappy wrote:
| Did you consider storing your media in a ZFS pool?
| pronoiac wrote:
| Checksum and PAR2 files are also useful alongside backup
| copies.
| chrisweekly wrote:
| Great idea! Thanks for building and sharing it; looking fwd to
| looking closer at it after Mother's Day!
| sirius87 wrote:
| This is pretty good!
|
| Detecting bitrot is fine, can also be done by storing a
| MD5/SHA256 of the file content. `md5sum <file>` barfs if its
| unable to read the file from disk. Challenge is rebuilding the
| broken file.
|
| I was worried about bitrot on my old external drives too. I
| eventually went with btrfs[1] filesystem running RAID1 across 2
| disks.
|
| [1] https://wiki.archlinux.org/title/btrfs
| mmastrac wrote:
| Adding/verifying hashes on media might not be a bad feature
| to add for those who don't want to burn the extra space
| required for PAR2. I run w/two extra parity disks on unRAID,
| but I'm too paranoid to leave my data at rest without an
| extra file-level of parity.
| aspenmayer wrote:
| What's a good method for making the original lossless rips,
| making parity files, and keeping them organized for use with
| Automedia? Is there something like Calibre for video/audio
| archives?
| mmastrac wrote:
| MusicBrainz Picard will do metadata tagging and some basic
| directory organization. It's what I use to create my music
| directories.
|
| With a great metadata manager like that, any software that
| rips to FLAC or Apple Lossless should be sufficient. Exact
| Audio Copy is one of the best ones on the Windows side.
| tomc1985 wrote:
| Curious as to how frequently bitrot shows up? Has anyone modeled
| this in terms of bits flipped per gigabyte per year or something
| like this?
|
| I have about half a terabyte of media, including some original
| music I made a long time ago and lost the project files for.
| Backups are great but it will just back of the corrupted versions
| saamerm wrote:
| Great work! Thanks for sharing and making it Open Source!
| daedalus2027 wrote:
| hi, I don't understand why they use ffmpeg for bitrot checking
| and not a simple hash checksum or a merkle tree.
| arendtio wrote:
| From my very limited understanding of the software I
| understand, that it doesn't compare an earlier version with a
| later version as you would with hash sums, but rather utilizes
| ffmpeg to check if something odd/irregular shows up. In
| addition, it can use PAR2 files which can be used to restore
| the original version which would not easily be possible with
| hash sums.
|
| But as I said, I just read the description, nothing more.
| mmastrac wrote:
| This is correct. I have a large media library and I wasn't
| sure how rotted it was. Test decoding won't catch every
| error, but it identified albums where there has clearly been
| a bitflip in a song (at which point I can try to find a
| lossless copy of that song to replace it).
|
| Once I'm happy with the state of the library, I use automedia
| to add on par2 files to avoid this going forward.
|
| Verification is for libraries that have no existing hashes or
| parity, essentially.
| UncleEntity wrote:
| I think par2 just uses a list of hashes to determine if it
| needs to do some error correction.
|
| Though I do see the value of checking the files when first
| 'checked in' to this recovery scheme so you start with a
| known good copy.
|
| The idea is solid and since it has transcoding build in it
| makes perfect sense to just use ffmpeg.
|
| --edit--
|
| The author beat me to the answer...
| fnordpiglet wrote:
| In addition to the other comments I don't think a merkle tree
| would help in this situation. Bitrot is a random event and
| wouldn't trigger anything that would take advantage of the tree
| data structure.
___________________________________________________________________
(page generated 2022-05-08 23:01 UTC)