[HN Gopher] 5M item limit for Google Drive: File unable to gener...
___________________________________________________________________
5M item limit for Google Drive: File unable to generate or upload
due to 403
Author : throwaway2056
Score : 127 points
Date : 2023-03-27 16:59 UTC (6 hours ago)
(HTM) web link (issuetracker.google.com)
(TXT) w3m dump (issuetracker.google.com)
| mike503 wrote:
| Part of this seems like some of the wonky design that seems to be
| under the hood of Drive. Just looking around at how rclone has to
| do things, the difference between shared and individual drives,
| the fact I couldn't even originally find a way in the console to
| see my shared drive usage (even though I was the owner of it and
| it was on my account) it just seems like there is a very very
| wonky odd design to how the system works. Not like an object
| store under the hood with a layer of "document" logic applied,
| but rather what started as initially some API-esque thoughts on
| document storage that grew into a monstrosity. I wouldn't be
| shocked if there's just a ton of tech debt there.
| danpalmer wrote:
| From reading the thread it sounds like this is 5M _per user_.
|
| Google Drive appears to be targeted at "human" usage, i.e. people
| uploading or creating files. I would guess that this is also
| worked into the cost - that the assumptions of the amount of work
| a human can do are a part of the price formulation. The reporter
| of this bug seems to be using this as a storage backend for
| software though, which I don't believe is the intended use-case.
|
| Looking at S3 pricing, just the storage is ~5x the cost of Google
| Drive, and then you need to add transfer and API calls on top of
| that.
|
| I don't personally think that there are reasonable use-cases for
| human users with 5 million files. There may be some specialist
| software that produces data sets that a human might want to back
| up to Google Drive, but that software is unlikely to run happily
| on drive streamed files so even those would be unlikely to be
| stored directly on Drive.
|
| (Disclaimer, I work at Google, not on Drive, this is my personal
| reading and interpretation of the public info, I don't have any
| inside info here)
| Gigachad wrote:
| I did some quick math on this. The top plan Google sells is
| 30TB, to reach 5M files in 30TB, you'd have to fill the entire
| thing with 0.75MB files. So yeah, if you are using Google drive
| as some kind of JSON blob storage, it isn't designed for your
| use case.
| elcritch wrote:
| Makes me wonder if the limit was actually as much for finding
| run away internal code. I'm sure there's cases deploying
| something like Google Drive where code went rampant making
| duplicates or something.
| danpalmer wrote:
| For 30TB I think it would be 6MB, but regardless, that
| appears to be per-user. If you're an individual needing 5m
| files, maybe you need a specialist storage solution, and if
| you're a business then that's one hell of a bus factor.
| Gigachad wrote:
| Ah yeah you are right, I had selected terabit instead of
| terabyte in the converter.
| TazeTSchnitzel wrote:
| What if you check out a few large git repos in a directory
| synced with Google Drive?
| danpalmer wrote:
| I thought about this, but I accidentally put the Webkit repo
| (~1m files) into Dropbox once and it didn't go well. Git
| needs a consistent view of the filesystem across many files
| to work, and that's not really possible with that many files
| stored in a sync system that works per-file like Drive does.
| I put Git in the category of specialist software that
| shouldn't be operated in Drive. By all means tar and backup,
| that should be effective, but there's no need to store all
| the individual files.
| augusto-moura wrote:
| I recently moved on to using Google Drive as a backup drive for
| my computer, the tool I'm using for it, Restic[1], does file
| deduplication at file block level. Restic allow multiple hosts
| to upload to the same backend/repository. So far I already have
| 25k files, even though I started backing up a only few weeks
| ago.
|
| I imagine that if you setup multiple servers/personal computers
| backups that could scale significantly.
|
| [1]: https://restic.net/
| mort96 wrote:
| Your analysis of why this limit exists is probably correct.
|
| However, when you're paying for a product, the limits should be
| disclaimed. If I'm investing in Google Drive, I should be able
| to easily see the limits which apply to the product I've
| bought; that means total storage caps, file count caps,
| bandwidth caps, and whatever else.
| danpalmer wrote:
| That's a fair point. It looks like most limitations are
| documented, but I couldn't find a mention of the 5m file
| limit.
| paxys wrote:
| While a 5 million file limit sounds reasonable at first glance,
| it isn't just the free tier that is affected. Google Drive also
| sells storage plans for up to 30 TB for an individual user, and
| there is no mention of such a limitation anywhere on any plan. If
| I bought and actually wanted to use that 30 TB (or heck even 5
| TB), it isn't unreasonable to want to store a few million files.
| Regardless of the technical reasons behind it, not mentioning
| this restriction up front is before the sale is inexcusable.
| Gigachad wrote:
| I did the math and to reach 5M files in 30 TB, they all have to
| be averaging 6MB. If you actually have this many files, you
| should be using S3 or mongodb.
| steponlego wrote:
| Sounds like they want to protect against resource exhaustion.
| Touch a file, it's empty with a size of zero bytes. But depending
| on the block size of the storage device, it's likely to be
| between 4k and up to even 1M.
| ec109685 wrote:
| They should incorporate a minimum size for every file when
| calculating how much quota you have.
|
| Because they have no retrieval fees for gsuite, I bet it is
| expensive dealing with drives with tons of small files.
|
| Agree they are rolling this out terribly. They should have
| granted everyone 2x the number of files they currently have in
| their drive quota (or 4M, whichever was smaller).
| encoderer wrote:
| Tangent: I was wrong to ever consider it sus that Google includes
| gsuite with their cloud revenue. Clearly it's being used that
| way.
| amf12 wrote:
| So does Microsoft. Intelligent Cloud = Azure + Office 365
| [deleted]
| duxup wrote:
| What is Google Drive... for?
|
| I see folks describe Google Drive errors / limits like this and
| people describe "a business critical operational system" and I
| wonder. Is Google Drive even supposed to do this thing?
|
| Not pointing the finger back at them, if Google doesn't make it
| clear they probably should, but at least as an individual I
| always see Google Drive as a single user cloud light system that
| allows for some sharing and organizational functions... but still
| not an "industrial" type cloud service.
| chimeracoder wrote:
| > at least as an individual I always see Google Drive as a
| single user cloud light system that allows for some sharing and
| organizational functions... but still not an "industrial" type
| cloud service.
|
| That may be how you view the product, but Google Drive
| absolutely is used as an enterprise cloud service by many large
| companies. And as a result, there are lots of applications that
| integrate with its API, etc., to serve those enterprise use
| cases.
| duxup wrote:
| I don't doubt people do it. Humans are industrious and users
| will push things to their limits.
|
| I just wonder .... is that what Google built Drive for? / how
| much of that they were thinking of.
| unmole wrote:
| Google Drive is part Google Workspace which is a paid
| enterprise offering.
| duxup wrote:
| I know. But enterprise user and enterprise, back end for
| an enterprise application is different.
| lazide wrote:
| At Google's scale for consumer users, on their backend it
| has to be an enterprise scale product anyway or the whole
| thing would have imploded shortly after launch.
|
| Sometimes there are random UX/UI/API limitations, but those
| usually get fixed relatively quickly when someone hits
| them, because they're typically just some hardcoded limit
| someone put in a front end layer because it was 'more than
| enough' and they needed something. A bit of elbow grease to
| clean up that path if it seems useful, and it's usually
| good again.
| chimeracoder wrote:
| > I just wonder .... is that what Google built Drive for? /
| how much of that they were thinking of.
|
| They sell enterprise plans, and have for years. It's
| definitely an intended use of their product.
|
| https://workspace.google.com/products/drive/
|
| It's funny to read your comment, because to me, Google
| Drive feels like an enterprise product that incidentally
| happens to provide a consumer-facing version as well, not
| the other way around.
| hirako2000 wrote:
| The assumption stems from the facts that a/ it started as
| personal use, like gmail. Only way later workspace and
| that the gsuite name even came out. Certainly no
| enterprise offering for years after personal use was
| broadly well adopted. b/ contrary to most SaaS
| businesses, Google is rather subtle and gentle when it
| comes to promoting or upselling, many people don't even
| know a paid/pro version of this thing exists.
|
| But you are right, google offers enterprise grade
| solutions so is expected to support over a few million
| files per account or even per user. Maybe something
| changed due to hasty cost cutting measures given the
| sudden interest by many on the issue tracker.
| chimeracoder wrote:
| > The assumption stems from the facts that a/ it started
| as personal use, like gmail. Only way later workspace and
| that the gsuite name even came out. Certainly no
| enterprise offering for years after personal use was
| broadly well adopted.
|
| Yes - I mean, sort of. It depends on what you define as
| the starting point for Google Drive. Like many Google
| products, the brand has been reorganized and reused a
| number of times.
|
| If you look at Google Drive as an extension of Google
| Docs, which existed prior to 2012, then yes, it started
| off as personal use. But if you look at it as a thing
| that launched in 2012 (the introduction of the Google
| Drive brand), then it arguably belongs as part of the
| reorganization that involved ending G Suite and turning
| it into Google Apps for Work[0], which is now known as
| Google Workspace.
|
| In fact, now that the consumer version of Google Drive
| has been reorganized into Google One, you could argue
| that the enterprise version of Google Drive is actually
| _older_ than the consumer version!
|
| Of course, all of these distinctions are somewhat
| arbitrary, because it's the same product that you've used
| all along, and the only thing that changes is the pricing
| and the marketing. But if we're talking about consumer
| vs. enterprise versions of a commodity product that
| essentially operates at enterprise-scale for consumers,
| that's also the only real difference.
|
| It's actually quite funny that the same product (which
| has been improved slightly, but is largely identical to
| what it was over a decade ago) has had so many different
| brands attached to it, especially when you consider that
| some of those brands are essentially container formats
| that repackage the same underlying brand (e.g., Google
| Workspace contains Google Drive).
|
| [0] There was actually a rebrand in between, from
| 2012-2015, but because Google's product names are
| Alphabet soup, I can't be sure of what they called it in
| the interim, though I'm positive it was a distinct
| product from before 2012.
| CaliforniaKarl wrote:
| You don't have anything in your HN profile, so I can't be
| certain, but I'm guessing you haven't been in the
| Research/Education space? It's scary how much Google Drive is
| used for storage of stuff.
| hirako2000 wrote:
| +1 And, it's not like Google didn't make huge marketing
| pushes to enter that space and businesses at large. Gsuite,
| Classroom, ever increasing storage limits. There is even an
| education package offering "unlimited" storage.
| andbberger wrote:
| google drive has unlimited storage, they opened that
| pandora's box themselves. yes it's janky as hell, but I'm
| looking at tens of thousands per year to backup to cloud
| services for my lab's data
| Robadob wrote:
| I work at a university that is fully invested in Google Apps.
|
| We did have unlimited storage, however last week they
| announced that's soon to change:
|
| > A quota will soon be introduced on the amount of data that
| staff and students at the University can store in their
| Google account. This is part of a larger piece of work being
| carried out by the Google Workspace team in IT Services, to
| review the overall amount of storage being used by the
| University.
|
| Makes me wonder if the timing is somehow related.
| SamuelAdams wrote:
| Looks like your admins changed the policy. Google ended
| unlimited storage for Education in 2021.
|
| https://blog.google/outreach-initiatives/education/google-
| wo...
| secabeen wrote:
| Those of us who had it before 2021 were allowed a grace
| period of a few years, but the end date is approaching
| soon.
| kevin_thibedeau wrote:
| Its purpose is to prevent people disabling third party cookies.
| lostmsu wrote:
| Can you clarify? Is this about single sign-on from multiple
| sites to be able to store data to Google Drive?
| kevin_thibedeau wrote:
| You can't download anything without 3PC. It's the only
| Google service set up that way, which isn't a coincidence.
| Drive is essentially a loss leader to protect their cash
| cow.
| hirako2000 wrote:
| What's to do with cookies? I understand the comment to
| refer to ads being by far their main revenue making
| business, and shaping users browswer preference to keep
| cookies helps Google make more money via ads.
|
| I didn't even know 3PC was used by google drive. Not sure
| what could be non cpincidental otuer than a technical
| reason for it given the very collaborative nature of this
| tool.
| kevin_thibedeau wrote:
| The downloads are served from a different domain. It
| needs 3PC to retrieve the Google auth token. No other
| service with downloads works that way.
| shadowgovt wrote:
| So that's the reason they're required: to preserve the
| credentials story between the static resource provider
| and the non-static layer. Nothing nefarious; just an
| implementation detail.
| kevin_thibedeau wrote:
| That's how all dark patterns are constructed. Present a
| plausible cover story. Keep the underlings in the dark
| about the true reason for the system architecture. Don't
| write anything down.
| shadowgovt wrote:
| That's also how all _patterns_ are constructed: present a
| plausible cover story (because it is true). Keep the
| underlings in the dark about the true reason for the
| system architecture (because the delta between the true
| reason and apparent reason is null). Don 't write
| anything down (because there's nothing else to write
| down).
|
| Suspicion of dark pattern is not proof of dark pattern.
|
| (ETA: at Google in particular, the dark-pattern-
| conspiracy you're describing between two unrelated teams,
| in this case Drive and static storage, is hard to pull
| off because coordinating those teams _requires_
| documenting design, and it turns out a lot of folks
| working at Google are Hacker News-type personalities who
| will ask inconvenient questions if they come up. But what
| they have that we lack is observation of the larger
| security picture and the ways failure to, for example,
| 3PC-guard your static files can result in failure modes).
| rootsudo wrote:
| It's "free" cloud hosting. You'd be surprised how often people
| mix their personal google account with _work_ stuff. It annoys
| me greatly, but that 's how it is.
|
| There is general ignorance of what google does/is/isn't and how
| online storage works. I agree with the sentiment, but, I've
| seen many a thing from recipes, shopping lists to FPA/Proposals
| all hosted on google drive. Even people sell content that is
| hosted on google drive.
| dragonwriter wrote:
| > It's "free" cloud hosting.
|
| Drive is paid (consumer via One, business via Workspace)
| cloud hosting with a free consumer tier.
| steponlego wrote:
| It's so Google can look at people's files. If you're silly
| enough to put a file on there, you give Google the right to
| copy and use it whatever which way they want. So it's a good
| deal if you're Google.
|
| EDIT: I just verified that this is all true.
| crazygringo wrote:
| All of what you said is factually untrue.
| djha-skin wrote:
| Google Drive has the "MS Excel" problem. It's made cloud
| storage really accessible to people, so they turn to it for
| everything, even stuff that's more appropriately put somewhere
| else. In Excel's case, it's database data. In Google Drive,
| it's files. My company uses it for video assets, because the
| media folks are really familiar with it, but we're moving it to
| S3 now because it's not got the chops for our seriously
| "larger" needs.
| Sohcahtoa82 wrote:
| For personal use, Google Drive creates a way for me to have
| automatic backups of files I wouldn't want to lose in case of
| drive failure. It's also a way to easily share files between my
| desktop and my phone.
|
| For professional use, I've usually seen it used as a document
| repository that can be shared by teams.
|
| In both use cases, 5 million files seems like it'd be a hard
| limit to hit.
| QuercusMax wrote:
| Disclaimer: I'm a Googler but I don't work anywhere near this
| stuff, and this is just my own personal opinion.
|
| In my mind, Drive isn't a _file_ sharing / storage system, but
| rather a _document_ sharing / storage system. If you want to
| store large numbers of files, you should use Cloud Storage,
| which can handle that. Drive is the cloud equivalent of LAN
| shared folders - you know those janky NFS and SMB shares that
| were always a giant headache because of permissions and other
| nonsense? That's what Drive is trying to be.
|
| Seems like they could do it without the jankiness, though.
| thinkingemote wrote:
| Is there a limit on the number of files in cloud storage?
| duxup wrote:
| TY
|
| That does seem how it operates. A lot of the UI is directed
| at document sharing.
|
| That's what makes me double take when I hear about folks
| using it for some very creative, but not what I think of uses
| for Google Drive.
| toast0 wrote:
| > always a giant headache because of permissions and other
| nonsense? That's what Drive is trying to be.
|
| It definitely lives up to that. I'm _always_ getting sent
| links to files I can 't open.
| stuartaxelowen wrote:
| https://www.google.com/drive/ 's title does call it a "File
| Sharing Platform".
| pyb wrote:
| I don't think the average user would understand the nuance,
| though. (I don't think I do)
| singlow wrote:
| But the average user is not writing automated systems to
| populate google drive with millions of files. Once you are
| doing something that unusual you should understand that
| nuance. Sure it is fun to hack a tool designed for one
| purpose to serve another, but if you are writing code for
| use in production you have to understand these types of
| design differences.
| M3tcalf3 wrote:
| ~50% of the users are above average. I doubt that Google
| was aiming to make only ~50% of the user base happy.
| lambdaloop wrote:
| Google drive is great for storing research data.
|
| I work it in a fly neuroscience lab and we use it to store all
| our electrophysiology and video data. Each person in the lab is
| storing on average 5TB of data, and the lab as a whole stores
| 100TB.
|
| The graphical user interface combined with unlimited storage
| for Google Workspaces is essentially an unbeatable deal.
| Researchers can upload their data easily through the interface.
| Any custom solution based on S3 or equivalent would take some
| time to teach and more time to maintain. Also, we're paying
| about $200 / month total to store 100TB of data in the cloud,
| which is hard to beat with other services.
|
| I tried setting up a single account for the whole lab once, but
| we ran into the above 5M file limit, so we just have individual
| accounts per researcher and it's mostly fine for now.
| ukd1 wrote:
| Do y'all back this up? If so, how?
| lambdaloop wrote:
| Everyone has their own machine with a hard drive containing
| their data. Google drive is the secondary cloud backup.
| secabeen wrote:
| > Also, we're paying about $200 / month total to store 100TB
| of data in the cloud, which is hard to beat with other
| services.
|
| You should expect this to go away soon. I support science
| researchers and our unlimited storage option is going away in
| the coming months. Options for purchasing space are limited,
| and not cheap.
| lambdaloop wrote:
| This is with a regular Google Workspace plan, independent
| of the university-wide research plan. The research plan
| unlimited storage is indeed going away for us as well (
| https://itconnect.uw.edu/tools-services-support/software-
| com... ), which is why we migrated to our own private
| Google Workspace.
|
| I haven't heard about the google workspace enterprise
| unlimited storage going away anytime soon, although perhaps
| you know something I don't?
| hobo_mark wrote:
| Looks like it'll be gone by next summer:
|
| https://www.nyu.edu/life/information-technology/about-nyu-
| it...
|
| https://case.edu/utech/about/utech-news/unlimited-google-
| sto...
|
| https://scripps.ucsd.edu/it/google-storage-changes
|
| https://it.wisc.edu/about/it-project-portfolio/reshaping-
| the...
|
| etc...
| remus wrote:
| How expensive are we talking? Looking at the workspace
| pricing page the enterprise plan gets you a pooled 5TB per
| user with the option to add more by talking to them, does
| that mean you have to pay more for the extra storage?
| a2tech wrote:
| Interesting that this was enabled with apparently no outreach.
| Maybe Google needed some of those employees that it recently laid
| off
| londons_explore wrote:
| And it would have been pretty simple to run a query for users
| with over 4 million objects, and fire off an email to them all
| saying "We are about to implement a limit, and you will be
| impacted. Here are potential workarounds/mitigations".
|
| I suspect though that the simple process of dropping a few
| hundred customers who will be directly impacted an email
| requires approval from 27 middle managers, and it's easiest to
| just ignore them.
| shadowgovt wrote:
| More often, this kind of error out of Google comes from the
| left hand not knowing what the right hand is doing.
|
| Joe or Jane Noogler has just been brought into the team, and
| their starting project was to improve indexing. They
| succeeded in doing so by attaching the Drive data to an
| indexing service created after Drive existed. It's working
| great and has a 10-20X speedup. But oops... That new indexing
| system has a 5-million-element hard limit built in, and
| nobody caught it until they went to production. J. Noogler is
| too new to have realized this _could_ have been an issue so
| they never started the escalation / customer messaging
| processs.
| hirako2000 wrote:
| Precisely what I felt skimming through the comments on the
| issue.
|
| A lot of high scale businesses have taken a huge risk. A
| company like Google had over 10y long tenure engineers who were
| swept by the layoffs. The argument that the year prior was a
| hiring spree only makes it worse: now you have hands to throw
| at the problem but unlikely the right pairs, which, given the
| climate of job loss fear, aren't likely to admit they are
| incompetent.
|
| Off topic, but this isn't an isolated case of mega size top
| tier companies seemingly dropping the ball more far often
| and/or with greater blast radius impacts. See github leaking
| secrets publicly on their own git repo.
|
| Call me paranoid, but to me one of these things is going on:
|
| 1/ tipping point of too many under qualified for the job
| engineers running things, due to C-suites looking at keyboard
| monkeys the way they see factory workers, naively applying
| "cost cutting" measures, plucking through spreadsheets who "are
| the biggest seemingly disposable weights" and fire those as
| part of waves of x% layoffs
|
| 2/ Overworked remaining know-how crews - added the weight of
| dealing with inspiring politicians turned enginners for the
| juicy 200k + rsu package who, demand better articulated
| instructions, knowledge transfers and more comprehensive formal
| documentation materials, so that they too can shine too - on
| top. Competent folks having less and less time but more more
| tears and blue bags below their eyes finally realising that
| ain't worth it as it keeps getting worse anyway.
|
| 3/ sabotage
|
| Or all those, since there could be some smowball effects there.
|
| Call me paranoid, that it isn't what's going on, we clearly
| aren't seeing the slow fall of major infrastructures the few
| who were there to build them now having mostly packed their bag
| and long gone, replaced by swaths of bootcamp trophy coders
| hoarding cloud certificates like North korean generals like to
| collect shiny pins.
|
| Maybe it isn't, the future is still in the making anyway, but I
| felt glimpses of that several years ago, then about each
| passing year, now every few months. like an seeing accelerating
| meteroite seemingly going away but which keeps looking bigger
| and bigger.
|
| More so related to the topic: just drop drive as a centralized
| storage, there are e2e solutions out there and open source
| scripts to entirely migrate out of these wall gardens. You
| won't miss google sheets: you can still use it!
| EricE wrote:
| So are there different limits for Google for
| Business/Workspace/whatever they are branding it this week? Or is
| this for all Google drive accounts?
| throwaway81523 wrote:
| It seems like there is a much lower limit for "download all".
| There is a 100 or so item directory that I want to download, and
| "download all" tries to wrap them in a zip file and fails. I
| think it wants to pre-scan all the files for viruses before
| zipping, but is unwilling to do that many. So I have to download
| the files one at a time.
| [deleted]
| bobbylarrybobby wrote:
| I've definitely downloaded folders with 100 items before. I
| think it might have to do with how big the individual files are
| because g drive won't create a zip that's larger than 2GB.
| synctheship wrote:
| Well this is scary, I wonder if all users/plans are impacted? I
| have north of 8.5million files in GDrive and I luckily have not
| hit an error yet.
| blitzar wrote:
| They also have what looks like an arbitary limit on "Shared
| Drives"
|
| The number of stored files: 400,000
| danpalmer wrote:
| This is true, but you can just create more shared drives. It
| looks like storage quota is shared across all drives on the
| account.
| [deleted]
| fbn79 wrote:
| Maybe Google cloud using NTFS disks (Maximum number of files on
| disk: 4,294,967,295)? :))
| overthrow wrote:
| That would be 5 billion, not 5 million
| bobsmooth wrote:
| Just checked, my C drive has a bit more than 1 million files.
| What are you doing that you have 5 million separate files in your
| Google Drive?
| jackson1442 wrote:
| If you're using Google Earth Engine or a similar tool, it's
| generally highly recommended to use Google Drive as the storage
| backend for it - especially since Google sells Enterprise plans
| with "as much storage as you need."
|
| Geospatial artifacts can be very large, and also can be spread
| across thousands and thousands of files (20+ zoom levels on
| Earth, with enough 256x256 tiles to cover the planet at each
| zoom level, for example).
| justin_oaks wrote:
| Nobody should ever need more than 640K of RAM, right?
|
| Just because your workflow doesn't involve lots of tiny files
| doesn't mean other people's don't.
|
| Besides, most of those commenting in the issue are talking
| about their organization having hit the limit, not individuals.
| bobsmooth wrote:
| As kulahan said, why are you using a personal backup solution
| for a business?
| kulahan wrote:
| Why is an organization using a primarily-personal file
| storage/transfer system for real work
| shadowgovt wrote:
| The same reason anyone uses anything: because it works
| until it doesn't.
| hirako2000 wrote:
| Many use some rsync-like tools to use gdrive as a flat backup
| system. Perfectly fine use case. You would be over a million as
| a single user. Some account have thousands of users.
|
| See some comments on the tracker, some orgs have a multi sites
| set up across an entire nation. I wouldn't be surprised if 10k+
| volunteers at some NGOs are on that sort of plan and use it as
| their personal storage solution on top of all the things they
| sync for document counts heavy work.
| Gigachad wrote:
| You can easily reach this number if you are doing something
| like storing json/xml user data from some service. But I just
| can't imagine you could ever hit this limit as a normal user.
| And if you do, just zip some old stuff in to one file and carry
| on.
| londons_explore wrote:
| 5M items sounds suspiciously like one machine has to keep in RAM
| the complete list of a users files for certain operations. User
| accounts with large numbers of objects were probably causing
| those machines to OOM. 5M items, at about 1000 bytes per item
| (name, metadata, a few uuids, etc) is 5Gbytes. 5Gbytes is about
| the amount of RAM most tasks will be given.
|
| I could imagine that some services within Google haven't been
| carefully designed - for example, perhaps the quota service reads
| a complete file listing into RAM, adds up the sizes, and then
| writes the available quota.
|
| This is probably the easy fix, rather than redesigning every
| service to be able to stream objects correctly.
| londons_explore wrote:
| It could also be a sharding/checkpointing issue. Imagine you
| want to scan every file in every user account, for example for
| blank documents (maybe because you want to downrank blank
| 'untitled documents' in search results, because they are easily
| accidentally made).
|
| You write your scanner to divide up the list of user accounts
| into chunks, process all chunks in different worker machines,
| and combine the results. Simple mapreduce.
|
| However, if there is one huge user account, then you either
| have to wait for the entire process to take far longer, or you
| need to have multiple workers working on the single account
| (adding a lot of complexity to every operation you wish to run
| across all accounts).
| cabirum wrote:
| More like an arbitrary number from the old times "cause 5M
| items ought to be enough for everybody."
| charcircuit wrote:
| >from the old times
|
| That doesn't make sense because this is a new restriction.
| mike503 wrote:
| Probably a reference to Bill Gates saying "640K ought to be
| enough for anybody"
| 8organicbits wrote:
| I think the parent is referring to the 42 year old meme
| quote "640K ought to be enough for anybody".
|
| https://www.computerworld.com/article/2534312/the--640k--
| quo...
| genewitch wrote:
| Couldn't one programmatically test this by hitting 5GB sooner
| with excessively long filenames? Say 500 byte names?
| [deleted]
| pdw wrote:
| As one of the comments points out, Google sells 30TB accounts.
| With a 5M file limit, you need an average file size of 6MB to
| fill your drive.
| shadowgovt wrote:
| This checks out; in general, the 30TB consumers are media
| storage, editing, and archival.
| up2isomorphism wrote:
| The key is that every company is very happy to lose this kind of
| customers.
| dkjaudyeqooe wrote:
| I'm surprised it knows how to count that low.
| ccheney wrote:
| Someone must've copied their node_modules folder into Google
| Drive by mistake...
___________________________________________________________________
(page generated 2023-03-27 23:01 UTC)