Post A4ZxSC72XSsFYyl2oK by pcrock@fosstodon.org
 (DIR) More posts by pcrock@fosstodon.org
 (DIR) Post #A4Zw7dzzYyD61269Gi by frostygoth@fosstodon.org
       2021-02-23T21:52:01Z
       
       0 likes, 1 repeats
       
       I’ve got a riddle for you. What’s the best way to move 1PB if data from one state to another over a 1gbps MPLS link? I was thinking ZFS send with compression and deduplication, but really I just need anything that reduce the bits on wire as much as possible. Any ideas?
       
 (DIR) Post #A4Zw7eRzsqbzPtCVv6 by codesections@fosstodon.org
       2021-02-23T21:57:38Z
       
       0 likes, 1 repeats
       
       @frostygoth > I’ve got a riddle for you. What’s the best way to move 1PB if data from one state to anothersneekernet, https://en.wikipedia.org/wiki/Sneakernet /a SSD sent via FedEx will honestly probably be the fastest. If you really need to send it over the wire, then yeah, zfs send seems as good as anything else I can think of.  But I'll boost is case anyone else has brilliant insights.
       
 (DIR) Post #A4Zx38boQc85SPCg1g by Ateriath@fosstodon.org
       2021-02-23T22:08:24Z
       
       0 likes, 0 repeats
       
       @codesections @frostygoth"What bandwidth do you get?""Eh, depends on traffic."
       
 (DIR) Post #A4ZxSC72XSsFYyl2oK by pcrock@fosstodon.org
       2021-02-23T22:12:55Z
       
       0 likes, 0 repeats
       
       @codesections @frostygoth Or several boxes of SSDs, but yeah. By my calculations, it will take 3 months to transfer 1 PB over a 1 gbps connection, _assuming_ you can maintain a perfect connection the whole time.I have a lot of experience with shipping Pelican cases full of hard drives just because FedEx gives you way better bandwidth than most Internet connections.
       
 (DIR) Post #A4ZxhShFA7vjLCyJZw by frostygoth@fosstodon.org
       2021-02-23T22:15:41Z
       
       0 likes, 0 repeats
       
       @codesections Thanks! I figured shipping and then syncing was the best way to do it, but my buddy says it’s got to be done over the wire and uncompressed it’s gonna take like 93 days to happen.
       
 (DIR) Post #A4a14ed7GBAwsIn9pg by smallsees@social.dropbear.xyz
       2021-02-23T22:53:33Z
       
       0 likes, 0 repeats
       
       @codesections @frostygoth I was hoping someone suggested sneakernet.It's by far the best way for huge amount of data.
       
 (DIR) Post #A4aBxhzvRGk0myLqXQ by suetanvil@mastodon.technology
       2021-02-24T00:55:26Z
       
       0 likes, 0 repeats
       
       @codesections @frostygoth Fun fact: one of the ways Amazon provides for putting your data into AWS is to send around a semi truck full of hard drives and related servers.(I don't know whether they'll provide it for getting your data out again, though.)