[HN Gopher] New physical AWS Data Transfer Terminals let you upl...
___________________________________________________________________
New physical AWS Data Transfer Terminals let you upload to the
cloud faster
Author : vinni2
Score : 43 points
Date : 2024-12-02 07:41 UTC (5 days ago)
(HTM) web link (aws.amazon.com)
(TXT) w3m dump (aws.amazon.com)
| DamonHD wrote:
| Never underestimate the bandwidth of a station wagon full of
| tapes hurtling down the highway.
|
| - Andrew S. Tanenbaum
| aitchnyu wrote:
| I just learned the AWS truck has been retired for months
| https://www.datacenterdynamics.com/en/news/aws-retires-snowm...
| thanksgiving wrote:
| I wonder how this process worked in practice. Do you simply
| send your only one set of hard disks and hope for the best?
| Do you keep your original data on one set of disks, copy it
| to another set of disks and send it off? Do you make multiple
| copies of the same data and send them all together on the
| same truck? Multiple copies on multiple trucks? How would you
| do reconciliation on the other end once the disks arrive at
| the destination?
|
| Like everything feels so simple and straightforward from afar
| but once I try to actually reason about something even the
| simplest of tasks feels complicated.
| jamessb wrote:
| > I wonder how this process worked in practice. Do you
| simply send your only one set of hard disks and hope for
| the best?
|
| No, you don't put your own disks in the
| Snowmobile/Snowball/Snowclone. It contains disks, so when
| it arrives you connect it to your network and copy data
| onto it, and then it is driven to an Amazon datacentre.
|
| See, e.g.
| https://docs.aws.amazon.com/snowball/latest/developer-
| guide/...
|
| https://aws.amazon.com/blogs/aws/aws-importexport-
| snowball-t...
| thanksgiving wrote:
| Thank you. That makes much more sense. So this way once
| the data is on the AWS device it is Amazon.com's
| responsibility so the customer doesn't need to worry
| about the truck or whatever.
| hulitu wrote:
| > so when it arrives you connect it to your network and
| copy data onto it, and then it is driven to an Amazon
| datacentre.
|
| And if there is an accident on the road, bad luck. /s
| oatmeal_croc wrote:
| Judging by the locations (New York and LA) I wonder if this is to
| cater to folks from production houses who want to upload large
| video files for processing or backup.
| bigs wrote:
| Or law firms with large discovery data sets
| aitchnyu wrote:
| How big are they? I thought its the 100gb order of magnitude
| as (dead tree) libraries.
| berbec wrote:
| It adds up quick. I know of a law firm (under a 100
| employees) with over 20TB.
| bbatha wrote:
| That's still peanuts, you can get consumer grade HDDs
| with that capacity in a single drive. A business grade
| line would have no trouble uploading all of that data in
| less than a week, even with a bunch of extenuating
| circumstances.
| BrentOzar wrote:
| > no trouble uploading all of that data in less than a
| week
|
| When you're doing e-discovery, deadlines are often
| measured in days - not just for the upload time, but for
| the analysis and finding the needle in the haystack.
| oatmeal_croc wrote:
| I'd imagine with LLMs today, discovery work is probably
| done on the cloud by bots.
| bigs wrote:
| A few years ago there was definitely document processing
| automation and query based filtering but still alot of
| human work.
|
| I assume you're right and AI now does some of the work
| but I doubt all of it. Also how reliable would the AI
| be... you'd hate to not have critical evidence at trial
| because you trusted the AI fully and it missed something.
|
| Discovery data includes audio, video, social site data,
| as well as the usual documents and emails.
| akira2501 wrote:
| It'd be a great way to get sued for negligence. You can't
| even assume the counterparty has correctly put everything
| into discovery for you. What you don't know is what gets
| you into trouble.
|
| An example from the Karen Reed case, the police, somehow,
| uploaded a video that had been put through a "mirror
| filter" and thus showed a vehicle in the opposite
| orientation from reality. Is your LLM going to notice
| that?
| bigs wrote:
| Also gotta think of what else is using the corporate
| internet pipe you can't drown it in one aws upload for
| days.
| eightys3v3n wrote:
| Some smaller businesses may have a huge data store, but
| not the money to pay for a business grade internet
| connection to upload it in a reasonable amount of time.
| I've worked for clients who have a 10 megabit full duplex
| fiber connection for over $1,000 a month (probably
| because of low ISP competition and because they were in a
| newly built, low density area). If they were working on
| migrating to the cloud, they would certainly consider
| taking a few hard drives one time to AWS rather than
| maxing their 10 MB full duplex connection for weeks or
| months.
| bigs wrote:
| Hmm. A couple years ago I think one large firm I knew their
| aws instance had about 400tb I think. Constantly growing
| with new cases.
|
| They had instances around the world this was just one.
| arjvik wrote:
| > Don't be surprised if there are no AWS signs in the building or
| room. This is for security reasons to keep your work location as
| secret as possible.
|
| Huh?
| trallnag wrote:
| From time to time around midnight Bezos can be seen peeking
| through the "A" of random AWS signs. This prevents that
| vladde wrote:
| I don't think I've ever handled so much data I'd need to increase
| data throughput.
|
| How much are we talking? Like petabytes? (Do you just stroll in
| with on a huge disk array?)
| bombcar wrote:
| I don't think it's really so much about throughput as it is
| about avoiding any ingress chargers at all.
| IanCal wrote:
| Does aws charge ingress?
|
| I've got a good connection for the UK but still about a 100
| meg upload. I have external drives that are 4TB, so if
| everything goes perfectly that could take about two full days
| to upload.
| kondro wrote:
| AWS doesn't charge for ingress.
| canucker2016 wrote:
| FTA: "There will be no per GB charge for the data transfer
| if you upload data into AWS Regions in the same continent
| of your location."
|
| N.B. I'm not contradicting the parent, but reframing the
| concept - it seems that the Data Transfer Terminal is
| ALREADY in AWS so ingress isn't a thing since your data is
| ALREADY in AWS as soon as you connect the optic fiber cable
| to whatever storage you've brought onsite. But since you're
| only renting the connection, your data can't stay forever
| in AWS unless you copy to S3 or somewhere else in AWS.
| unsnap_biceps wrote:
| They mention it as a way to take your snowball and upload it
| and walk out and continue using it without shipping it back and
| forth. Those look to go into the 210 TB range of raw storage.
|
| In the past, at some companies I was at, I could see using
| something like this once a quarter to upload full quarterly
| backups, depending on the price per hour.
| Joe8Bit wrote:
| In a previous role we used their (first party) physical
| transfer appliances to upload ~600PB of video into S3. It was a
| complex logistical exercise to take it from the physical SANs,
| but the AWS specialists we worked with were great and it went
| without a hitch.
| KaiserPro wrote:
| I think it depends on your network connection.
|
| We used a snowball a couple of times to either move data from
| S3 to or it.
|
| In some cases its because we didn't have enough local storage
| to shuffle the data, or because we only had a 100 meg net link
| and a couple of TBs to move.
| unsnap_biceps wrote:
| I presume that given they mention that it only supports public
| endpoints, that it's just a directly peered connection on the
| public internet and there's no special security stuff at play
| here?
| vineyardmike wrote:
| They probably mentioned public endpoints to highlight that they
| allow access to AWS services but they're not renting out access
| to a massive bandwidth pipe for non-customers or to upload to
| non-AWS. The "public endpoints" part probably is used to warn
| customers that existing technical limitations apply wrt
| abilities- eg you can't dump data directly to the S3 data
| plane, you still have to go through the public API.
|
| Considering the fiber connection is "part of the AWS network",
| it may not have access to the outside internet, but also
| contextually probably doesn't have any privileged access to AWS
| servers.
| unsnap_biceps wrote:
| My statement that it's on the public internet is more that
| it's directly connected to someone like level3 and not a
| direct fiber drop to an AWS datacenter.
|
| I would be surprised if the building in NYC has direct fiber
| to IAD or if the building in LA has direct fiber to SFO.
| Snoozus wrote:
| It looks like there is no local buffer, so one needs to stick
| around till the upload is done? Did anyone see a mention of
| upload bandwidth?
| canucker2016 wrote:
| from the FAQ, https://aws.amazon.com/data-transfer-
| terminal/faqs/ - What is the connection type?
| - Each Data Transfer Terminal facility will have at least two
| (2) 100G optical fiber cables that are connected to the AWS
| network. - What are the key requirements for
| preparing my device to use the Data Transfer Terminal facility?
| - To prepare for using the Data Transfer Terminal facility and
| connecting to the network, you need to ensure your uploading
| device is prepared to connect to the network. You should have
| the following for an optimal data upload experience:
| * A transceiver type 100G LR4 QSFP * An active IP
| auto configuration (DHCP) * Up-to-date
| software/transceiver drivers
|
| Also, from https://aws.amazon.com/data-transfer-
| terminal/pricing/: Per port charge for data
| transfer US to US @ $300 US to EU @ $500
| US to Other @ Contact us
| solardev wrote:
| I wonder if this will cause devices like ATM skimmers to pop up
| at these secret locations, skimming the traffic via MITM attacks
| on the network or tampered laptops.
| averageRoyalty wrote:
| They appear to be rooms inside data centers, so it's pretty
| doubtful.
| DonHopkins wrote:
| Are the facilities available to rent late at night for LAN
| parties?
|
| And do they serve good refreshments?
| Havoc wrote:
| I would have thought download would be more interesting. Dodge
| the egress charges on cloud migrations
| canucker2016 wrote:
| Egress charges for migrations hasn't been a problem since 2024
| March.
|
| from https://aws.amazon.com/blogs/aws/free-data-transfer-out-
| to-i... We believe this choice must include
| the one to migrate your data to another cloud provider or on-
| premises. That's why, starting today, we're waiving data
| transfer out to the internet (DTO) charges when you want to
| move outside of AWS. If you need more than 100
| gigabytes of data transfer out per month while transitioning,
| you can contact AWS Support to ask for free DTO rates for the
| additional data. It's necessary to go through support because
| you make hundreds of millions of data transfers each day, and
| we generally do not know if the data transferred out to the
| internet is a normal part of your business or a one-time
| transfer as part of a switch to another cloud provider or on
| premises. We will review requests at the AWS
| account level. Once approved, we will provide credits for the
| data being migrated. We don't require you to close your account
| or change your relationship with AWS in any way. You're welcome
| to come back at any time. We will, of course, apply additional
| scrutiny if the same AWS account applies multiple times for
| free DTO.
| EwanToo wrote:
| If you request this, AWS requires:
|
| "After your move away from AWS services, within the 60-day
| period, you must delete all remaining data and workloads from
| your AWS account, or you can close your AWS account."
|
| So egress charges are still a significant problem.
|
| https://aws.amazon.com/ec2/faqs/#Data_transfer_fees_when_mov.
| ..
| greatgib wrote:
| Just for a reference of the price, it will be 300$ per hour in US
| and 500$ in EU.
|
| That looks quite expensive in my opinion, even if this target big
| professionals.
___________________________________________________________________
(page generated 2024-12-07 23:01 UTC)