[HN Gopher] S3 as a Git remote and LFS server
___________________________________________________________________
S3 as a Git remote and LFS server
Author : kbumsik
Score : 100 points
Date : 2024-10-19 10:37 UTC (12 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| mdaniel wrote:
| All this mocking when moto exists is just :-(
| https://github.com/awslabs/git-remote-s3/blob/v0.1.19/test/r...
|
| Actually, moto is just one bandaid for that problem - there are
| SO MANY s3 storage implementations, including the pre-license-
| switch Apache 2 version of minio (one need not use a bleeding
| edge for something as relatively stable as the S3 Api)
| SahAssar wrote:
| Do you mean boto (the python SDK for AWS)?
|
| EDIT: They probably do not, I'm guessing they mean
| https://docs.getmoto.org/en/latest/index.html ?
| flakes wrote:
| moto server for testing S3 is pretty great. It's about the
| same experience as using a minio container to run integration
| tests against.
|
| I use this, and testing.postgresql for unit testing my api
| servers with barely any mocks used at all.
| neeleshs wrote:
| There is also testcontainers. Supports multiple languages.
| Uses containers though.
|
| https://testcontainers-python.readthedocs.io/en/latest/
| mdaniel wrote:
| Happy 10,000th Day to you :-D Yes, moto and its friend
| localstack are just fantastic for being able to play with AWS
| without spending money, or to reproduce kabooms that only
| happen once a month with the real API
|
| I believe moto has an "embedded" version such that one need
| not even have in listen on a network port, but I find it
| much, much less mental gymnastics to just supersede the
| "endpoint" address in the actual AWS SDKs to point to
| 127.0.0.1:4566 and off to the races. The AWS SDKs are even so
| friendly as to not mandate TLS or have allowlists of endpoint
| addresses, unlike their misguided Azure colleagues
| SahAssar wrote:
| > Happy 10,000th Day to you :-D
|
| Sorry, not sure what you mean?
| mdaniel wrote:
| https://xkcd.com/1053/
| notpushkin wrote:
| > there are SO MANY s3 storage implementations
|
| I suppose given this is under the AWS Labs org, they don't
| really care about non-AWS S3 implementations.
| mdaniel wrote:
| Well, I look forward to their `docker run awslabs/the-
| real-s3:latest` implementation then. Until such time,
| monkeypatching api calls to always give the exact answer the
| consumer is looking for is damn cheating
| chrsig wrote:
| it wouldn't be unprecedented. dynamodb-local exists.
| notpushkin wrote:
| Agreed, haha. Well, I think it _should_ work with Minio &
| co. just as well, but be prepared to have your issues
| closed as unsupported. (Pesonally, I might give it a go
| with Backblaze B2 just to play around, yeah)
| philsnow wrote:
| I'm surprised they just punt on concurrent updates [0] instead of
| locking with something like dynamodb, like terraform does.
|
| [0] https://github.com/awslabs/git-remote-s3?tab=readme-ov-
| file#...
| mdaniel wrote:
| I thank goodness I have access to a non-stupid Terraform state
| provider[1] so I've never tried that S3+dynamodb setup but, if
| I understand the situation correctly, introducing Yet Another
| AWS Service ™ into this mix would mandate that _callers_
| also be given a `dynamo:WriteSomething` IAM perm, which is
| actually different from S3 in that in S3 one can -- at their
| discretion -- set the policies on the _bucket_ such that it
| would work without any explicit caller IAM
|
| 1:
| https://docs.gitlab.com/ee/user/infrastructure/iac/terraform...
| ncruces wrote:
| Google Cloud Storage is good enough to implement locks all by
| itself:
| https://reddit.com/r/golang/comments/t52d4f/gmutex_a_global_...
|
| Doesn't S3 provide primitives to do the same? At least since
| moving to strong read-after-write consistency?
|
| PS: I wrote the above package. Happy to answer questions about
| it.
| noctune wrote:
| S3 recently got conditional writes and you can use do locking
| entirely in S3 - I don't think they are using this though. Must
| be too recent an addition.
| fortran77 wrote:
| Amazon has deprecated Amazon Code Commit, so this may be an
| interesting alternative.
| Scribbd wrote:
| This is something I was trying to implement myself. I am
| surprised it can be done with just an s3 bucket. I was messing
| with API Gateways, Lambda functions and DynamoDB tables to
| support the s3 bucket. It didn't occur to me to implement it
| client side. I might have stuck a bit too much to the lfs test
| server implementation. https://github.com/git-lfs/lfs-test-server
| tonymet wrote:
| how does it handle incremental changes? If it's writing your
| entire repo on a loop, I could see why AWS would promote it.
___________________________________________________________________
(page generated 2024-10-19 23:00 UTC)