[HN Gopher] Starting October 19, storage limit will be enforced ...
___________________________________________________________________
Starting October 19, storage limit will be enforced on all Gitlab
Free accounts
Author : reimertz
Score : 148 points
Date : 2022-08-08 15:06 UTC (7 hours ago)
(HTM) web link (docs.gitlab.com)
(TXT) w3m dump (docs.gitlab.com)
| bachmeier wrote:
| This doesn't affect me, but a better way to handle this would be
| to sell extra storage at, say, double GitLab's cost. Digital
| Ocean sells 250 GB object storage at $5/month and $0.02/GB beyond
| that.
| john_cogs wrote:
| GitLab team member here. GitLab offers additional storage for
| $60/year for an additional 10GB of storage and 20GB of
| transfer/month.
| bachmeier wrote:
| I see now that it was mentioned at the end of the post.
| (Though I did specify "double GitLab's cost".)
| oy6amn8RDPw8w9n wrote:
| Yes, $0.50/GB/month seems like very reasonable pricing /s
|
| That's over twice as much as the disks to store that much
| data cost. Every month.
| glenneroo wrote:
| Yeah that really hurts. A single Unity project can easily
| reach that in a short amount of time. When I get that
| warning email in a few weeks, I'll be shopping around.
| jordemort wrote:
| What's the point of tapering it down in stages like this? Between
| the October 19th quotas and the October 20th quotas, if you wait
| until the last minute, you have 24 hours to move 37.5TB of data.
| Then 4 more days to move another 7TB; does that actually help
| anyone? The proposition of getting that much data out of it at
| that speed seems a bit unrealistic. Why not just say "the quota
| will be 5GB on November 9th" and be done with it?
| john_cogs wrote:
| The phased enforcement of the limits are a part of the
| technical rollout plan for this change that was added to our
| docs. Related comment:
| https://news.ycombinator.com/item?id=32387597.
|
| The communication sent to impacted users via email and future
| in-app notifications includes only the applicable enforcement
| dates and limits.
|
| Issue to follow: https://gitlab.com/gitlab-
| org/gitlab/-/issues/368150
| M3L0NM4N wrote:
| Definitely sounds like there was some bureaucracy in there.
| giancarlostoro wrote:
| I'm assuming this is only for the ones they host and not the
| self-hosted solution. That is insane that anyone uploads
| terrabytes of data into gitlab, is there an actual valid non-
| illegal / weird backup choice use case? Is there some big ass GIS
| open source project out there that could use the attention of
| GitLab before they nuke some vital data somehow?
| john_cogs wrote:
| You are correct in your assumption - the storage limit impacts
| SaaS only. I added some context on the rollout in a previous
| comment. https://news.ycombinator.com/item?id=32387597
| bedosh wrote:
| The quota includes not just the git repo, but also other
| services that gitlab offers. In particular the container and
| package registries are easy to setup in a way that accumulates
| a lot of data (e.g. by building and tagging a docker image in
| every commit and not cleaning up old images). In most cases
| this will be a misconfiguration and all the quota enforcement
| is doing is to force people to configure their projects a bit
| more considerate.
|
| As far as I know, the quota itself isn't new, it just wasn't
| enforced in the past.
| Beltalowda wrote:
| GitHub limits the package registry to 500M for free accounts,
| so 5G doesn't seem so bad. Git repos themselves usually don't
| take up that much space; all my 135 projects on GitHub are
| ~250M combined (quick count, could be off a bit, but roughly
| on that order). Even larger repos at $dayjob with daily
| commits typically take up a few hundred M at the most.
|
| Basically you need to either be in the top 0.1% of highly
| active accounts or do something specific that requires a lot
| of disk space, but for most people 5G doesn't seem so
| unreasonable.
| ev1 wrote:
| I'm actually somewhat curious how much of an impact this will
| actually have - I think I've only seen a 5 GB+ repo once or
| twice and they were not really source code, but mildly
| "abusing" github CDN/releases for downloads.
|
| At least gitlab is not deleting any data, just rejecting pushes
| if you're over the limit.
| Macha wrote:
| Does the storage measurement work now? The container registry and
| build artifacts used to not be measured
| john_cogs wrote:
| Yes, the container registry and build artifacts are now
| measured for Free users.
| rwmj wrote:
| https://gitlab.com/nbdkit/nbdkit/-/usage_quotas
|
| The headline number is 1.1G but the container registry is
| 16G.
|
| Edit: You should definitely add a simpler way to delete old
| pipelines. Having to mnnage it yourself through the API is a
| pain. (https://stackoverflow.com/questions/71513286/clean-up-
| histor...)
| james-skemp wrote:
| Mine is still showing negative usage for some repos. The most
| negative is showing -1.08 GB, both from my profile and from
| the project page.
|
| Repo 130.82 MB Artifacts 12.74 MB Wiki 51.20 KiB Everything
| else 0 bytes.
|
| Been like this for at least a couple weeks.
|
| This one is public, but I have others that are private with
| negative values.
| james-skemp wrote:
| Looks like the issue is tracked in
| https://gitlab.com/gitlab-org/gitlab/-/issues/368326
|
| Example of a public project with negative project storage:
| https://gitlab.com/strivinglife/book-raspberry-pi
| raverbashing wrote:
| How much did data hoarders use Gitlab for their backups?
| noasaservice wrote:
| So, we either go to Github, where our licenses are abused for
| their shitty ML.
|
| Or we pay $20/month to Gitlab. And I can't figure out how the
| quotas will intersect with "professional", if at all.
|
| For us Open Source devs, neither is a good option. Although I
| have heard good things about sr.ht / sourcehut. And for the
| service, it appears to be fair https://sourcehut.org/pricing/
| iMarv wrote:
| I have recently migrated my projects to fossil (https://fossil-
| scm.org), hosting them myself on a cheap vps.
|
| I also really got hooked on their code/project management
| philosophy, rethinking a lot on how I want to run collaborative
| projects in general
| john_cogs wrote:
| Hi there, in another reply I mentioned GitLab for Open Source
| provides Ultimate features and higher limits to Open Source
| projects for free:
| https://news.ycombinator.com/item?id=32387621
| encryptluks2 wrote:
| Yes, and even the Ultimate plan limits will be significantly
| impacted. If they are not already self hosting now is a good
| time to look into it or explore other lightweight options
| like Gitea.
| uo21tp5hoyg wrote:
| Personally I use codeberg[1] which doesn't have all the fancy
| bells and whistles but for personal projects it runs like
| butter.
|
| [1] https://codeberg.org/
| Lutger wrote:
| Why do you need more than 5gb for a git repo? IMHO, any repo
| above 100mb should have an exceptional reason for being so big,
| and if it's just code a normal repo is more like 1 to 10 mb in
| size, max.
|
| You don't even need to pay gitlab, free tier can do for most
| stuff and there's a generous sponsorship for Open Source.
|
| I don't get all the hate Gitlab is receiving these days.
| noasaservice wrote:
| I do hardware hacking, presentations, code, pictures, and
| full media to reproduce what I've shared.
|
| FreeCAD files get big. And to accommodate easy printing, STLs
| are also needed.
|
| KiCAD for board layout and schematics can also get larger.
| And remember, these also have 3d board components too.
|
| Presentations are naturally larger.
|
| Full high-rez pictures eat storage like you wouldn't believe.
|
| Code is small, thankfully.
|
| One such device I have created is hovering around 4.5GB for a
| full reproduction for the current snapshot. And if you do the
| command to pull the whole history (and not current), its
| around 10GB. And, I'm not sure if GL is counting the whole
| history, or the current? And are they pruning old after a
| certain date?
| dnsmichi wrote:
| > And, I'm not sure if GL is counting the whole history, or
| the current? And are they pruning old after a certain date?
|
| GitLab stores the Git history with all data revisions as
| commits. To reduce the repository storage [0], older Git
| commits can be pruned but this would rewrite the Git
| history as an invasive change [1]. To store larger files,
| it is recommended to use Git LFS [2]. Alternatively, you
| can use the generic package registry to store data. [3]
|
| [0]
| https://docs.gitlab.com/ee/user/usage_quotas.html#manage-
| you...
|
| [1] https://docs.gitlab.com/ee/user/project/repository/redu
| cing_...
|
| [2] https://docs.gitlab.com/ee/topics/git/lfs/
|
| [3]
| https://docs.gitlab.com/ee/user/packages/generic_packages/
| robertlagrant wrote:
| It's not just Git. Gitlab hosts a container registry for
| every repo, so if you build and store a new container image
| regularly then you could easily top 5GB.
| jsmith45 wrote:
| Larger heavily contributed to repositories can be quite
| large, even if they are just plain code. Sure, not quite 5 GB
| large, but still.
|
| Linux: $ git clone https://git.kernel.org/pub
| /scm/linux/kernel/git/torvalds/linux.git $ du -h
| linux/.git ... 2.8G linux/.git
|
| Git: $ git clone
| git://git.kernel.org/pub/scm/git/git.git $ du -h
| git/.git ... 120M git/.git
|
| DefinitelyTyped: $ git clone
| https://github.com/DefinitelyTyped/DefinitelyTyped $ du
| -h DefinitelyTyped/.git 850M DefinitelyTyped/.git
|
| home-assistant/core: $ git clone
| https://github.com/home-assistant/core $ du -h
| DefinitelyTyped/.git 380M core/.git
|
| The above is also only taking into account repo size, while
| the GitLab limit applies to everything including release
| artifacts, CI build artifacts, hosted containers, etc. Many
| of which GitLab currently provides poor or sometimes even
| zero support for implementing purging.
| pxeger1 wrote:
| Git works perfectly fine without a web interface like that.
| It's not popular nowadays, but you can just keep the primary
| repository on your own server and merge patches by email.
| marcinzm wrote:
| >So, we either go to Github, where our licenses are abused for
| their shitty M
|
| As long as it's on the internet MS can scrape the data so not
| being on Github is only a temporary defense.
| jacooper wrote:
| But GitHub's TOS can only protect them from GitHub repos.
|
| If it has impact on CoPilot or not, is not clear.
| kmeisthax wrote:
| Well, at least in the US training an AI will probably fall
| under Fair Use. In the EU there is an explicit copyright
| exception for data mining. So I don't think there's a
| _legal_ obligation for Microsoft to only train within the
| bounds of public GitHub repos.
| gaius_baltar wrote:
| > Well, at least in the US training an AI will probably
| fall under Fair Use.
|
| Provoking question: so, will it be Ok to train an AI on
| leaked Microsoft code the publish the models (not the
| code)?
|
| Of course, MSFT won't accept this, but w know that big
| corps are hypocrite by default.
| jacooper wrote:
| AI fair use is not really clear anywhere.
|
| I think SFconservancy's articles about CoPilot are very
| helpful.
| marcinzm wrote:
| Except not all contributors to projects signed the TOS
| since you can upload an existing project with existing non-
| Github contributors. One person uploading a project does
| not override or change the license of that project. So MS
| is already not relying on the TOS for legal protection
| which means they feel legally they can train on any OSS
| projects.
| T3RMINATED wrote:
| [deleted]
| lol768 wrote:
| I guess this is the next step to reduce costs after the brakes
| were put on the "let's delete old OSS repositories" leaked plan.
|
| For comparison, I think GitHub just have a cap of 100MB on any
| single individual file, plus:
|
| > We recommend repositories remain small, ideally less than 1 GB,
| and less than 5 GB is strongly recommended. Smaller repositories
| are faster to clone and easier to work with and maintain. If your
| repository excessively impacts our infrastructure, you might
| receive an email from GitHub Support asking you to take
| corrective action. We try to be flexible, especially with large
| projects that have many collaborators, and will work with you to
| find a resolution whenever possible.
|
| Which is a bit wishy-washy, but sounds like there's room for
| discretion / exceptions to be made there rather than a hard cap
| at 5GB.
| Bayart wrote:
| The difference being that the value Microsoft gets from GitHub
| isn't its revenue, it's its influence. Whereas Gitlab is just
| another corporate software suite.
| ollien wrote:
| I don't have the diffs in front of me, but I'm fairly sure
| this predates the MS acquisition by quite a while.
| xtracto wrote:
| Prior to MS acquisition, Github free offering was really
| bad: No private repos, no CI pipeline and I think there was
| not real "team".
| ollien wrote:
| Well, Github (free or not) had no CI until Github
| Actions, which was after the acquisition. Integration
| with things like Travis were already there and free.
| tonmoy wrote:
| Do you have a source for this? I vaguely remember GitHub
| starting to offer private repos when GitLab started
| getting popular.
| ollien wrote:
| Github raised the 5 repo limit to infinite in 2016
| https://github.blog/2016-05-11-introducing-unlimited-
| private...
|
| And made private repos free after the acquisition
| https://github.blog/2019-01-07-new-year-new-github/
| password4321 wrote:
| > _the value Microsoft gets from GitHub_ [is] _influence_
|
| Don't forget easy access to all the code to train Copilot,
| the AI code launderer. With my tinfoil hat on: even the
| private repo edition!
| DaiPlusPlus wrote:
| > I think GitHub just have a cap of 100MB on any single
| individual file
|
| There's a 100MB limit on the size of a push, which also limits
| the size of a any single git object (i.e. file) to 100MB too.
| However GitHub supports LFS for large files, and their
| documentation says to use LFS for files over 100MB:
|
| https://docs.github.com/en/repositories/working-with-files/m...
|
| > GitHub blocks pushes that exceed 100 MB.
|
| > To track files beyond this limit, you must use Git Large File
| Storage (Git LFS).
|
| https://docs.github.com/en/billing/managing-billing-for-git-...
|
| According to my own GitHub account (
| https://github.com/account/billing/data/upgrade?packs=1 ), I'm
| paying $37/yr for 600GB LFS storage on top of my existing
| GitHub Pro subscription.
| dark-star wrote:
| GitHub has a repo size limit of 100GB (https://web.archive.org/
| web/20200521202931/https://help.gith...) and you'll get a
| warning if you exceed 75GB
| JLCarveth wrote:
| > Github _had_ a repo size limit of 100GB
|
| The source you link is 2 years old.
| profile53 wrote:
| Can you provide a source to the contrary? I'm fairly
| certain it's still in place.
| imalerba wrote:
| Reading between lines it also says it going to enforce a 10GB
| limit on Paid tiers.
|
| > Namespaces on a GitLab SaaS paid tier (Premium and Ultimate)
| have a storage limit on their project repositories. A project's
| repository has a storage quota of 10 GB.
|
| Even it's not mentioned as a change nor in the timeline, but that
| limit does not exist currently.
| JZerf wrote:
| Just wanted to point out that it looks like people will be able
| to go over that limit but will have to pay extra for it
| according to
| https://docs.gitlab.com/ee/subscriptions/gitlab_com/index.ht...
| . Seems somewhat reasonable.
| imalerba wrote:
| This change is one of the only MRs about these new limits
| discussed in the open.
|
| https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91418
| lakomen wrote:
| Ergo don't use anything where Micro$oft has their hands in.
|
| Glad I moved off Github years ago.
|
| Sid Sijbrandij should be ashamed. I know the $7.5b are making it
| easier for him. It's literally everyone's dream on this site
| here, but not mine.
| bdcravens wrote:
| I'm confused at the connection between Microsoft/Github and the
| decision that Gitlab made here.
| likortera wrote:
| I think GP just rushed the free hate
| likortera wrote:
| You are confusing Gitlab with GitHub?
| MaxDPS wrote:
| But this is related to GitLab...
| whoomp12342 wrote:
| 5 gigs? what is this, repo hosting for ants?
| bluedino wrote:
| That does seem ridiculously small. It's probably fine for 95%
| of projects though.
| duskwuff wrote:
| Git change compression is quite effective. Unless you start
| checking in large binaries, most _companies_ will never get
| close to 5 GB. (Even 1 GB is a pretty substantial amount of
| code.)
| hinkley wrote:
| That will depend on how well they handle forks. I could
| easily see myself exceeding such a limit just issuing bug
| fixes.
|
| Especially since one of the bugs I'm likely to fix is, "why
| is this library so damned big?"
| john_cogs wrote:
| GitLab team member here. Forks of projects get
| deduplicated, so only the changes you make will contribute
| to your storage consumption as long as the fork
| relationship is maintained.
| john_cogs wrote:
| Update: our team recently identified an issue impacting
| how we calculate storage which results in forks being
| counted towards Usage Quota. This will be addressed
| before we begin enforcement.
| tomjakubowski wrote:
| Shrinking the library won't make the repo any smaller. You
| have to erase history to do that.
| hinkley wrote:
| Yeah I thought that might be confusing.
|
| Artifact size depends on what garbage from the project
| infrastructure gets included into the artifact. So simple
| things like not having a fully populated ignore file will
| cause things to get included into the project or the
| artifact.
|
| People who don't care about file size don't care about
| file size. If the artifact is ridiculous, sometimes the
| repo is ridiculous too. Therefore if you take a bunch of
| projects with outsized artifacts, you are going to have
| above average repository sizes as well.
| prepend wrote:
| I need solutions that fit 100% of my projects. I don't want
| to use service X for 95% and service Y for 5%.
| ozim wrote:
| You are also willing to pay for it?
|
| Because it sounds like you want to use 100% of the service
| and not pay for it.
| elcomet wrote:
| Then you can pay for more data, seems fair no?
| xeromal wrote:
| Do personal accounts really have 5gigs of code??? Unless you
| have a lot of models/artifacts, images, plain text code should
| be well under 5gb for 100s of projects
| oy6amn8RDPw8w9n wrote:
| Most repo's are in the singles or tens of MB. If those are
| ants, then sure.
| Jensson wrote:
| That is what github has now, doesn't seem to be out of the
| ordinary.
| jacooper wrote:
| That per repo, GitLab's is for the whole account.
| cercatrova wrote:
| I'm just going to move to GitHub at this point.
| djohnston wrote:
| Why are people checking build artefacts into the repo?
| cesarb wrote:
| Playing devil's advocate: sometimes your build is non-
| reproducible, annoying to build, or both (for instance, needing
| a proprietary tool which can only run on a particular
| developer's laptop because the license is tied to that
| particular hardware, and which crashes half of the time for no
| particular reason). Keeping the build artifacts in the
| repository means you can reproducibly obtain that exact
| artifact, even years into the future.
| encryptluks2 wrote:
| 5GB isn't much different than the storage limits of other
| services, but their storage pricing is atrocious. I've seen the
| writing on the wall for a while and watched as GitLab went from
| being the cool open source alternative to GitHub to becoming a
| bloated oversized mess. I know several popular open source
| projects were offered premium tier upgrades for free. I am
| curious to see if these changes, especially transfer limits, will
| impact them enough to move away.
| uo21tp5hoyg wrote:
| Am I reading it right that the original Free Tier had a quota of
| 45,000GB? That seems absurdly high and not very sustainable
| (hence the change I assume).
| john_cogs wrote:
| No, 45,000GB wasn't a quota. At the moment, the 5GB limit is a
| soft limit, and the rollout plan included tiered enforcement.
| This is an internal implementation detail of our technical
| rollout plan.
| NovemberWhiskey wrote:
| I'm reading that as a "stop-the-bleeding" number; presumably
| there is someone out there with a 44TB repository and they want
| to impose initial quotas that don't actually impact anyone
| immediately.
|
| I guess someone has been backing up their movie collections to
| Gitlab or something.
| bluedino wrote:
| One case I've seen on our local Gitlab server is someone in
| data science/HPC (accidentally?) adding the output of a
| solver to their repo. Easily hundreds or thousands of
| gigabytes of data.
| hinkley wrote:
| I had to learn svn surgery because someone imported a 1GB
| archive to test the ol' signed 32 bit file size bug, got
| yelled at, and deleted it. Well, except it's still in the
| repository, mate, it's just hidden.
|
| I replaced it with a fixture that was 2GB of space
| characters, which compressed down to about 3KB. I know
| there's a canonical file bomb zip file that's under 1K but
| there's clever and then there's clever.
| vbezhenar wrote:
| I might be wrong about that but I'm pretty sure that I saw that
| limit (100 MB) when I was reading about GitLab plans in the
| past. They just didn't enforce it for some reason.
|
| IMO GitLab does the wrong thing. They should have enforced
| those limits from the beginning. And if they didn't, they
| should've eaten those expenses or at least grandfathered old
| repos.
| gabeio wrote:
| > And if they didn't, they should've eaten those expenses or
| at least grandfathered old repos.
|
| Why should they eat the expenses of abusive users? As well
| they clearly have and are finally taking action about
| preventing it. Your entire post seems extremely entitled to
| their money.
| oy6amn8RDPw8w9n wrote:
| bombcar wrote:
| That number sounds like "unlimited, but the system requires a
| number" kind of thing.
| YorickPeterse wrote:
| No, originally there was no quota enforced to begin with if my
| memory serves me right. The limits discussed here are likely
| meant to gradually tighten up the limits, rather than
| immediately locking out projects that exceed these limits.
| uo21tp5hoyg wrote:
| I guess an infinitely large storage quota is only slightly
| less sustainable than a 45,000GB one.
| szundi wrote:
| I won't publish my gitlabfs plugin for 1.7k different
| programming languages then.
| oefrha wrote:
| A lot of discussions about how git repos are supposed to be small
| are totally missing the point. This storage quota applies to
| everything, including release artifacts, containers, etc. Forget
| containers or CI artifacts on every commit, let's look at a very
| common scenario: using goreleaser to build binaries and
| deb/rpm/etc. packages for multiple architectures every release.
| This way a moderately sized Go project can easily consume
| 50-100MB or more per release. That gives you at most 50-100
| releases across all your projects.
|
| Using hosted GitLab for open source projects is looking less and
| less appealing.
|
| I also posted about issue trackers on gitlab.com not allowing
| search without signing in a while back:
| https://news.ycombinator.com/item?id=32252501
|
| Edit: An open source program that upgrades the quota is mentioned
| elsewhere in the thread: https://about.gitlab.com/solutions/open-
| source/ I don't use hosted GitLab for my open source work, so no
| idea how many people get approved.
| sieabahlpark wrote:
| john_cogs wrote:
| Hi, GitLab team member here. The GitLab for Open Source program
| provides open source projects with Ultimate benefits and higher
| limits. More info in previous comment:
| https://news.ycombinator.com/item?id=32387621
| KMnO4 wrote:
| Why do you have to "apply" for these benefits? Shouldn't
| providing the source publicly in the open (ie not a private
| repository) by definition be enough for "open source"?
| bluedino wrote:
| What kind of projects are using 500GB much less TB's of data?
| kodah wrote:
| Desktop Environment components, I suspect.
| ryanlitalien wrote:
| In gaming, you can have a single branch that is 8TB with 1+
| million files. But those projects aren't using git.
| alexb_ wrote:
| I didn't even know there was no storage limit - that seems like
| an immediate way to get your platform used to store non-code data
| in very large quantities.
| pantulis wrote:
| I would guess controls to avoid people storing pirated movies
| on Gitlab repos and similar stuff have been in place for a long
| time.
| kerblang wrote:
| Perhaps now is a good time to recommend the ever-popular BFG to
| anyone unaware: https://rtyley.github.io/bfg-repo-cleaner/
|
| Also my team's biggest repo is a 2.5 GB checkout but gitlab
| (self-managed) reports it as 185MB "files" and 353 MB "storage"
| (no CI/CD artifacts).
| sofixa wrote:
| Similarly there's also git-filter-repo:
| https://github.com/newren/git-filter-repo
|
| It's in Python so runs pretty much everywhere *nix out of the
| box.
| cookiengineer wrote:
| Anybody found out which projects on gitlab exceed the 45 TB
| limit?
|
| I'm curious what kind of project would even need such a
| repository size. From a distant view this sounds like heavily
| mismanaged build artifacts in the project's git history; or
| abused storage for free CDN of video data or similar.
| game-of-throws wrote:
| If you build an image for testing on every commit and don't have
| a retention policy set up, you could be using a massive amount of
| space without realizing it. I can see why they did this.
| snoopy_telex wrote:
| Are retention policies easy? I recall fighting on GitHub to
| setup a sane retention policy awhile back, but haven't
| revisited in years.
| dnsmichi wrote:
| GitLab offers ways to manage retention policies for your
| container registry. Please see the docs for guidance: https:/
| /docs.gitlab.com/ee/user/packages/container_registry/...
| sofixa wrote:
| I wouldn't say easy because regexes are needed for complex
| ones (e.g. retain -prod images for 6 months, -staging for 24
| days) but they are powerful and not that complicated.
| pm215 wrote:
| Also, for some things there is no retention policy configurable
| at all -- eg pipelines and their associated stdout logs appear
| to not be auto-cleaned-up at any expiry date (no default,
| nothing configurable). If you want to get rid of those you need
| to script it via the API, it seems: eg
| https://gitlab.com/eskultety/gitlab_cleaner
|
| (I just ran that on the QEMU project and reduced the usage from
| 295GB to 165GB by deleting pipelines older than 1 Jan... so
| that's a lot of low-hanging logfile fruit gitlab could be auto-
| deleting.)
| khuey wrote:
| Gitlab didn't even _have_ retention policies until sometime in
| 2020. I have a six year old project that 's consuming something
| like 4TB of space in their container registry.
| akudha wrote:
| Woah. 4 TB? What kind of project is it?
| georgyo wrote:
| Not op, but a simple node application can very easily make
| container images that are well over a gigabyte. Build and
| store that container images on every push and boom, you can
| explode in usage.
| elevation wrote:
| Git excels at tracking human keyboard output. A productive
| developer might write 100KB of code annually so a git repo can
| represent many developer years of collaborative effort in just a
| few MB. That is, unless you require git to track large media
| files, third party BLOBs, or build output.
|
| However, sometimes tracking these things are necessary, and since
| there isn't an obvious companion technology to git for caching
| large media assets ("blob hub?") or tracking historical build
| output ("release hub?"), devs abuse git itself.
|
| I wish there were a widely accepted stack that would make it easy
| to keep the source in the source repo, and track the multi-gb
| blobs by reference.
| foepys wrote:
| You mean something like git Large File Storage? It comes with
| git for Windows by default and every Linux distro I know has it
| in its repos. MacOS also has it in Homebrew.
|
| https://git-lfs.github.com/
|
| It's basically one click/command away.
| adobrawy wrote:
| > companion technology to git for caching large media assets
| ("blob hub?") or There is Amazon S3 or any other cloud storage.
|
| > tracking historical build output ("release hub?")
|
| There are OCI artifact registries.
|
| For me, there is rather issue that when all you have is a
| hammer, everything looks like a nail.
| WorldMaker wrote:
| Git LFS is a pretty widely accepted stack for managing binary
| blobs by reference in git: https://git-lfs.github.com/
|
| The plugin is installed out of the box in many git
| distributions now. Many hosts support it today, including
| Gitlab, which is relevant to this article's discussion:
| https://docs.gitlab.com/ee/topics/git/lfs/
| john_cogs wrote:
| GitLab team member here. The impacted users are notified via
| email and in-app notifications will begin 2022-08-22, so far
| we've contacted 30,000 users. Only GitLab SaaS users are impacted
| - the limits are not applicable to self-managed users.
| martypitt wrote:
| We're affected by this change, and have been trying to get
| support in forums, but can't get hold of anyone.
|
| Build Artifacts are listed as part of what contributes to the
| quota (which is fair enough), but there's no way (that I could
| find in the docs) to manage build artifacts that are stored.
|
| I suspect we have a large historic storage, which we don't use
| / need, but there's no way to browse this, no way to verify it,
| and no way to delete what we don't need.
|
| I'm dreading getting a big ole bill in a few months for storage
| we had no way to opt out of.
| lbotos wrote:
| GitLab Support Engineering Manager here. These limits will
| first be soft limits as they are now. Impacted users will be
| notified and there will be about 2 months to take action
| before enforcement starts. After that point, limits will be
| enforced.
|
| If you'd like to investigate now you can use the GitLab API
| to fetch job artifacts and their storage size:
| https://docs.gitlab.com/ee/api/jobs.html#list-project-jobs
| and calculate the summary.
|
| I'd suggest collecting all pipeline jobs to be deleted using
| this API endpoint
| https://docs.gitlab.com/ee/api/job_artifacts.html#delete-
| job...
|
| We are looking to improve the visibility for job artifacts in
| the UI as we get nearer to the storage limit being enforced.
| xiaq wrote:
| I have a repo that was almost exactly 5GB and it turned out
| that the storage was also used exclusively by pipeline logs
| and artifacts.
|
| Found this recipe on StackOverflow to just nuke everything
| and worked for me:
| https://stackoverflow.com/questions/71513286/clean-up-
| histor.... You can tweak it a bit if you want to keep the
| more recent pipeline runs.
|
| For some reason I had to run this multiple times to
| completely remove everything; each run removes about half of
| all the pipelines and only when there were about 20 remaining
| did the script remove everything.
| stjohnswarts wrote:
| What's the difference between self-managed and SaaS users? To
| me gitlab is a SaaS ?
|
| I mean if I'm self hosting on digitalocean doesn't that just
| mean that I'm using your FOSS and doing all the work myself and
| completely separate from gitlab anyway other than the common
| codebase?
|
| Sorry, I'm not a guru of gitlab and don't know all the common
| parlance.
| JonathonW wrote:
| GitLab SaaS is the instance at gitlab.com.
|
| Anything else is self-managed.
| Kelteseth wrote:
| My Qt/C++ cross-platform FOSS Wallpaper Engine project[1]
| currently uses 47gb of storage. This is because I compile for
| every platform and store the artifacts for 4 weeks. Not sure what
| I will do in the future, because having older builds around to
| try out without recompiling is always nice.
|
| [1] https://gitlab.com/kelteseth/ScreenPlay
| john_cogs wrote:
| GitLab for Open Source provides OSS projects with Ultimate tier
| benefits, and includes 250GB of storage and 500GB
| transfer/month. Please apply to join the program here:
| https://about.gitlab.com/solutions/open-source/
| TheRealPomax wrote:
| And note that even if you're just "a random someone with a
| bunch of open source repos", you almost certainly already
| qualify for this program:
|
| ---
|
| In order to be accepted into the GitLab for Open Source
| Program, applicants must:
|
| - Use OSI-approved licenses for their projects: Every project
| in the applying namespace must be published under an OSI-
| approved open source license. [1]
|
| - Not seek profit: An organization can accept donations to
| sustain its work, but it can't seek to make a profit by
| selling services, by charging for enhancements or add-ons, or
| by other means.
|
| - Be publicly visible: Both the applicant's GitLab.com group
| or self-managed instance and source code must be publicly
| visible and publicly available.
|
| ---
|
| [1] https://opensource.org/licenses/alphabetical
| oy6amn8RDPw8w9n wrote:
| tfw GitLab admits that their own product isn't open-source
| ;)
|
| ("can't seek to make a profit by selling services, by
| charging for enhancements or add-ons")
| synu wrote:
| They've been saying that for a long time.
| https://about.gitlab.com/blog/2016/07/20/gitlab-is-open-
| core...
| jffry wrote:
| That isn't Gitlab saying those are the requirements for
| being "open-source", it's just their requirements for who
| they are willing to give a generously large amount of
| free services to.
|
| I think it's quite fair to say you're not going to give
| free services out to open source projects that are
| seeking to fundraise beyond covering their costs.
| TheRealPomax wrote:
| That's... an incredible footprint. Surely even if there was no
| size limit, deduplicating the build output would have made a
| lot of sense?
| TheRealPomax wrote:
| Having said that: if you run a public open source project
| that has requirements beyond what a regular free user gets,
| then why not apply to the open source program[1] (which you
| almost certainly already qualify for) so you're not space
| constrained?
|
| Quoting the requirements:
|
| ---
|
| Who qualifies for the GitLab for Open Source Program? In
| order to be accepted into the GitLab for Open Source Program,
| applicants must:
|
| - Use OSI-approved licenses for their projects: Every project
| in the applying namespace must be published under an OSI-
| approved open source license. [2]
|
| - Not seek profit: An organization can accept donations to
| sustain its work, but it can't seek to make a profit by
| selling services, by charging for enhancements or add-ons, or
| by other means.
|
| - Be publicly visible: Both the applicant's GitLab.com group
| or self-managed instance and source code must be publicly
| visible and publicly available.
|
| ---
|
| [1] https://about.gitlab.com/handbook/marketing/community-
| relati...
|
| [2] https://opensource.org/licenses/alphabetical
| shakow wrote:
| I guess that having old artefacts is nice, but expecting 47GB
| of storage for free per repo seems ludicrous to me.
|
| Maybe it's a generational question, but I would be pretty happy
| with 1-2GB (i.e. git history, and one artefact per platform).
| spyremeown wrote:
| >but expecting 47GB of storage for free per repo seems
| ludicrous to me.
|
| I agree with this. This move makes perfect sense. I think
| GitLab does a lot already for the people that don't pay for
| the service.
| rohfle wrote:
| A couple of notes - If I tag a docker image with
| multiple tags, and then push it to Gitlab, each tag counts
| towards the storage limits even though SHAs are identical. eg
| 100MB container tagged with "latest" and "v0.5" uses 200MB of
| storage. - The storage limit is not per repository, but per
| namespace. So 5GB free combined for all repositories under your
| user. If you create a group, then you get 5GB free combined for
| that group. Does this include forks? Does this include
| compression server side? - The 10GB egress limit per month
| includes egress to self-hosted Gitlab Runners in free tier.
| Consider this with the 400 minutes per month limit on shared
| runners.
|
| These limits feel less like curbing abuse and more like squeezing
| to see who will jump to premium while reducing operating costs.
| Is this a consequence to Gitlab hosting on GCP with associated
| egress and storage costs? Is this a move to improve financials /
| justify a market cap with fiscal storm clouds on the horizon? Is
| this being incentivized by $67m in awarded stock between the CFO
| and 2 directors?
|
| Stock history over last year for GTLB (since IPO in 2021?):
| https://yhoo.it/3QaExCs
|
| From the golden era of 2015:
| https://about.gitlab.com/blog/2015/04/08/gitlab-dot-com-stor...
|
| > To celebrate today's good news we've permanently raised our
| storage limit per repository on GitLab.com from 5GB to 10GB. As
| before, public and private repositories on GitLab.com are
| unlimited, don't have a transfer limit and they include unlimited
| collaborators.
| nabakin wrote:
| With this change, the 5 user limit[1], and original intent to
| delete dormant repositories[2][3], it seems as though GitLab is
| no longer able to support the free side of its business. GitLab
| has been touted as more OSS-friendly than GitHub, but a large
| part of the OSS ecosystem depends on free repositories. With
| these changes and this trajectory, I can't see myself putting
| another OSS project on GitLab.
|
| It's a shame it's come to this, but I'm confident GitLab didn't
| make this choice lightly. It must be done in order for them to
| stay afloat.
|
| Thank you GitLab team for your efforts. I hope you guys are
| successful in your future endeavors.
|
| [1] https://about.gitlab.com/blog/2022/03/24/efficient-free-
| tier...
|
| [2]
| https://www.theregister.com/2022/08/04/gitlab_data_retention...
|
| [3]
| https://www.theregister.com/2022/08/05/gitlab_reverses_delet...
| DoesntMatter22 wrote:
| I think a lot of the problem is how they communicated it. It
| isn't like "Well hey we have a problem here", it's just "Hey
| here's this super complicated to this problem you didn't know
| existed and you can take it or leave it"
| jacooper wrote:
| Honestly for me the bigger issue is the upcoming bandwidth limit.
|
| There is no clear date to when its going to be enforced.
|
| But its definition is so wide its crazy, its basically any egress
| data except the web interface and shared runners.
|
| AFAIK it will also include git clones!, so if your project
| suddenly gets popular the users clones will cost you many too.
|
| Also if you use your own runner, cloning the repo to the runner
| will also be included in your bandwidth limit.
|
| And since it should apply to GitLab pages, it becomes useless for
| anything the you want to get few visits.
|
| Since GitLab is behind cloudflare, you might as well just use
| Cloudflare Pages at this point.
| john_cogs wrote:
| GitLab team member here. Thanks for the comment.
|
| We understand people don't want their quotas being filled by
| something outside of their control. For Open Source projects,
| we have GitLab for Open Source which contains higher limits
| from the Ultimate tier (250GB Storage, 500GB transfer/month).
| In addition, we intend to look into other ways to address your
| concern such as counting only your own traffic or allowing you
| to limit external traffic.
| DoesntMatter22 wrote:
| I love the fact that they haven't thought out a lot of how
| this needs to work. You can't even decide if it's worth
| sticking with GitLab because they don't even know how they
| are going to handle half of this stuff.
| nslzk wrote:
| karamanolev wrote:
| > Also if you use your own runner, cloning the repo to the
| runner will also be included in your bandwidth limit.
|
| You make it sounds like it's horrible. I'd offer a different
| take in that sure, it can be a huge negative if done recklessly
| on GitLab's part. But if done correctly, it can actually be a
| good thing. I think CI pipelines doing a full clone from
| scratch on every build (or npm installs or other bootstraps) is
| extremely wasteful in terms of resources, so I'd be glad if
| that reduces it significantly.
| jacooper wrote:
| GitLab CI default clone configuration already AFAIK only
| includes the last 20 commits, so its already slimmed down.
| karamanolev wrote:
| If you start from scratch, regardless of history, you're
| fetching at least the size of the repo. In many cases,
| that's hundreds of MBs for every build and I'm a huge fan
| of building for every commit. I find that a colossal waste.
|
| I have a very old fashioned and "not recommended" setup
| with Jenkins that only pulls new commits, because it works
| out of a persistent working directory. Works wonders for a
| Django/ES6 project and sips bandwidth. I wish more of the
| modern containerized, start-from-scratch and so on setups
| would work in a similar way wrt the bandwidth they use.
| unintendedcons wrote:
| Gitlab just can't shooting themselves in the foot.
|
| Driving away individuals is apparently their strategy now.
|
| Sad. I used to teach new developers starting with Gitlab pages.
| xaxaxb wrote:
| Haha, great.
| balentio wrote:
| Gitlab keeps giving me more reason to go to Gittea.
| nix23 wrote:
| I stay on:
|
| https://sourcehut.org/
|
| No blingbling no social-media drama, just clean straightforward
| code hosting.
| eliseumds wrote:
| Fair enough.
| bm5k wrote:
| Looks like everyone is rushing to check their storage.
|
| > something went wrong while loading usage details
|
| On my free group's storage page.
| john_cogs wrote:
| Yes, the container registry and build artifacts are now
| correctly displayed in the free plans.
___________________________________________________________________
(page generated 2022-08-08 23:02 UTC)