[HN Gopher] Package managers keep using Git as a database, it ne...
___________________________________________________________________
Package managers keep using Git as a database, it never works out
Author : birdculture
Score : 533 points
Date : 2025-12-26 12:46 UTC (10 hours ago)
(HTM) web link (nesbitt.io)
(TXT) w3m dump (nesbitt.io)
| eviks wrote:
| Indeed, the seductive nature of bad tools lying close to your
| hand - no need to lift your butt to get them!
| twoodfin wrote:
| What made git special & powerful from the start was its data
| model: Like the network databases of old, but embedded in a
| Merkle tree for independent evolution and verifiability.
|
| Scaling that data model beyond projects the size of the Linux
| kernel was not critical for the original implementation. I do
| wonder if there are fundamental limits to scaling the model for
| use cases beyond "source code management for modest-sized, long-
| lived projects".
| amluto wrote:
| Most of the problems mentioned in the article are not problems
| with using a content-addressed tree like git or even with using
| precisely git's schema. The problems are with git's protocol
| and GitHub's implementation thereof.
|
| Consider vcpkg. It's entirely reasonable to download a tree
| named by its hash to represent a locked package. Git knows how
| to store exactly this, but git does not know how to _transfer_
| it efficiently.
| mananaysiempre wrote:
| > Git knows how to store [a hash-addressed tree], but git
| does not know how to _transfer_ it efficiently.
|
| Naively, I'd expect shallow clones to be this, so I was quite
| surprised by a mention of GitHub asking people not to use
| them. Perhaps Git tries too hard to make a good packfile?..
|
| Meanwhile, what Nixpkgs does (and why "release tarballs" were
| mentioned as a potential culprit in the discussion linked
| from TFA) is request a gzipped tarball of a particular
| commit's files from a GitHub-specific endpoint over HTTP
| rather than use the Git protocol. So that's already more or
| less what you want, except even the tarball is 46 MB at this
| point :( Either way, I don't think the current problems with
| Nixpkgs actually support TFA's thesis.
| quaintdev wrote:
| I host my own code repository using Forgejo. It's not public. In
| fact, it's behind mutual tls like all the service I host. Reason?
| I don't want to deal with bots and other security risks that come
| with opening port to the world.
|
| Turns out Go module will not accept package hosted on my Forgejo
| instance because it asks for certificate. There are ways to make
| go get use ssh but even with that approach the repository needs
| to be accessible over https. In the end, I cloned the repository
| and used it in my project using replace directive. It's really
| annoying.
| xyzzy_plugh wrote:
| > There are ways to make go get use ssh but even with that
| approach the repository needs to be accessible over https.
|
| No, that's false. You don't need anything to be accessible over
| HTTP.
|
| But even if it did, and you had to use mTLS, there's a whole
| bunch of ways to solve this. How do you solve this for any
| other software that doesn't present client certs? You use a
| local proxy.
| agwa wrote:
| If you add .git to the end of your module path and set
| $GOPRIVATE to the hostname of your Forgejo instance, then Go
| will not make any HTTPS requests itself and instead delegate to
| the git command, which can be configured to authenticate with
| client certificates. See https://go.dev/ref/mod#vcs-find
| irusensei wrote:
| Have a look at Tailscale DNS and certs. Its gives you a valid
| cert through lets encrypt without exposing your services to the
| internet.
| baobun wrote:
| If you add the instance TLS cert (CA) to your trust store then
| go will happily download over https. It can be finicky
| depending on how you run go but I can confirm it works.
| Zambyte wrote:
| The issues with using Git for Nix seem to entirely be issues with
| using GitHub for Nix, no?
| femiagbabiaka wrote:
| Yeah, it's inclusion in here is baffling because none of the
| listed issues have anything to do with the particular issue
| nixpkgs is having.
| Rucadi wrote:
| I also got the same feeling from that, in fact, I would go as
| far as to say that nixpkgs and nix-commands integration with
| git works quite well and is not an issue.
|
| So the phrase the article says "Package managers keep falling
| for this. And it keeps not working out" I feel that's untrue.
|
| The most issue I have with this really is "flakes" integration
| where the whole recipe folder is copied into the store (which
| doesn't happen with non-flakes commands), but that's a tooling
| problem not an intrinsic problem of using git
| ekjhgkejhgk wrote:
| Do the easy thing while it works, and when it stops working, fix
| the problem.
|
| Julia does the same thing, and from the Rust numbers on the
| article, Julia has about 1/7th the number of packages that Rust
| does[1] (95k/13k = 7.3).
|
| It works fine, Julia has some heuristics to not re-download it
| too often.
|
| But more importantly, there's a simple path to improve. The top
| Registry.toml [1] has a path to each package, and once
| donwloading everything proves unsustainable you can just download
| that one file and use it to download the rest as needed. I don't
| think this is a difficult problem.
|
| [1]
| https://github.com/JuliaRegistries/General/blob/master/Regis...
| zahlman wrote:
| > 00000000-1111-2222-3333-444444444444 = { name =
| "REPLTreeViews", path = "R/REPLTreeViews" }
|
| ... Should it be concerning that someone was apparently able to
| engineer an ID like that?
| adestefan wrote:
| It's as random as any other UUID.
| Severian wrote:
| Incorrect, only some UUIDs are random, specifically v4 and
| v7 (v7 uses time as well).
|
| https://en.wikipedia.org/wiki/Universally_unique_identifier
|
| > 00000000-1111-2222-3333-444444444444
|
| This would technically be version 2, which would be built
| from the date-time and MAC address, and DCE security
| version.
|
| But overall, if you allow any yahoo to pick a UUID, its not
| really a UUID, its just some random string that looks like
| one.
| ekjhgkejhgk wrote:
| > if you allow any yahoo to pick a UUID, its not really a
| UUID
|
| universally unique identifier (UUID)
|
| > 00000000-1111-2222-3333-444444444444
|
| It's unique.
|
| Anyway we're talking about a package that doesn't matter.
| It's abandoned. Furthermore it's also broken, because it
| uses REPL without importing it. You can't even precompile
| it.
|
| https://github.com/pfitzseb/REPLTreeViews.jl/blob/969f04c
| e64...
| anonymars wrote:
| Which is to say, not guaranteed at all. GUIDs are designed
| to be unique, not random/unpredictable
|
| https://devblogs.microsoft.com/oldnewthing/20120523-00/?p=7
| 5...
| skycrafter0 wrote:
| If you read the repo README, it just says "generate a uuid".
| You can use whatever you want as long as it fits the format,
| it seems.
| ekjhgkejhgk wrote:
| Could you please articulate specifically why that should be
| concerning?
|
| Right now I don't see the problem because the only criterion
| for IDs is that they are unique.
| zahlman wrote:
| I didn't know whether they were supposed to be within the
| developer's control (in which case the only real concern is
| whether someone else has already used the id), or generated
| by the system (in which case a developer demonstrated
| manipulation of that system).
|
| Apparently it is the former, and most developers
| independently generate random IDs because it's easy and is
| extremely unlikely to result in collisions. But it seems
| the dev at the top of the list had a sense of vanity
| instead.
| KenoFischer wrote:
| You're supposed to generate a random one, but the only
| consequence of not doing so is that you won't be able to
| register your package if someone else already took the
| UUID (which is a pain if you have registered versions in
| a private registry). That said, "vanity" UUIDs are a bad
| look, so we'd probably reject them if someone tried that
| today, but there isn't any actual issue with them.
| galenlynch wrote:
| I believe Julia only uses the Git registry as an authoritative
| ledger where new packages are registered [1]. My understanding
| is that as you mention, most clients don't access it, and
| instead use the "Pkg Protocol" [2] which does not use Git.
|
| [1] https://github.com/JuliaRegistries/General
|
| [2] https://pkgdocs.julialang.org/dev/protocol/
| 0xbadcafebee wrote:
| This is basically unethical. Imagine anything important in the
| world that worked this way. "Do nuclear engineering the easy
| way while it works, and when it stops working, fix the
| problem."
|
| Software engineers always make the excuse that what they're
| making now is unimportant, so who cares? But then everything
| gets built on top of that unimportant thing, and one day the
| world crashes down. Worse, "fixing the problem" becomes near
| impossible, because now everything depends on it.
|
| But really the reason not to do it, is there's no need to.
| There are plenty of other solutions than using Git that work as
| well or better without all the pitfalls. The lazy engineer
| picks bad solutions not because it's necessarily easier than
| the alternatives, but because it's the path of least resistance
| for themselves.
|
| Not only is this not better, it's often actively worse. But
| this is excused by the same culture that gave us "move fast and
| break things". All you have to do is use any modern software to
| see how that worked out. Slow bug-riddled garbage that we're
| all now addicted to.
| hombre_fatal wrote:
| On the other hand, GitHub wants to be the place you choose to
| build your registry for a new project, and they are clearly
| on board with the idea given that they help massive projects
| like Nix packages instead of kicking them off.
|
| As opposed to something like using a flock of free
| blogger.com blogs to host media for an offsite project.
| baobun wrote:
| ...For now. The writing is on the wall.
| ModernMech wrote:
| Hold up... "lazy engineers" are the problem here? What about
| a society that insists on shoving the work product of
| unfunded, volunteer engineers into critical infrastructure
| because they don't want to pay what it costs to do things the
| right way? Imagine building a nuclear power plant with an
| army of volunteer nuclear engineers.
|
| It cannot be the case that software engineers are labelled
| lazy for not building the at-scale solution to start with,
| but at the same time everyone wants to use their work, and
| there are next to no resources for said engineer to actually
| build the at scale solution.
|
| > the path of least resistance for themselves.
|
| Yeah because they're investing their own personal time and
| money, so of course they're going to take the path that is of
| least resistance for them. If _society_ feels that 's
| "unethical", maybe pony up the cash because you all still
| want to rely on their work product they are giving out for
| free.
| rovr138 wrote:
| > If society feels that's "unethical", maybe pony up the
| cash because you all still want to rely on their work
| product they are giving out for free.
|
| I like OSS and everything.
|
| Having said that, ethically, should society be paying for
| these? Maybe that is what should happen. In some places, we
| have programs to help artists. Should we have the same for
| software?
| ekjhgkejhgk wrote:
| Fixing problems as they appear is unethical? Ok then.
|
| You realize, there are people who think differently? Some
| people would argue that if you keep working on problems you
| don't have but might have, you end up never finishing
| anything.
|
| It's a matter of striking a balance, and I think you're way
| on one end of the spectrum. The vast majority of people using
| Julia aren't building nuclear plants.
| BenjiWiebe wrote:
| Fixing problems when they appear is ethical.
|
| Refusing to fix a problem that hasn't appeared yet, but has
| been/can be foreseen - that's different. I personally
| wouldn't call it unethical, but I'd consider it a negative.
| zephen wrote:
| The problem is that popularity is governed by power laws.
|
| Literally anybody could forsee that, _if_ something
| scales to millions of users, there will be issues. Some
| of the people who forsee that could even fix it. But they
| might spend their time optimizing for something that will
| never hit 1000 users.
|
| Also, the problems discussed here are not that things
| don't work, it's that they get slow and consume too many
| resources.
|
| So there is certainly an optimal time to fix such
| problems, which is, yes, OK, _before_ things get _too_
| slow and consume _too_ many resources, but is most
| assuredly _after_ you have a couple of thousand users.
| xboxnolifes wrote:
| Most of the world _does_ work this way. Problems are solved
| within certain conditions and for use over a certain time
| frame. Once those change, the problem gets revisited.
|
| Most software gets to take it to more of an extreme then many
| engineering fields since there isn't physical danger. Its
| telling that the counter examples always use the potentially
| dangerous problems like medicine or nuclear engineering. The
| software in those fields _are_ more stringent.
| IshKebab wrote:
| > when it stops working, fix the problem
|
| This is too naive. Fixing the problem costs a different amount
| depending on _when you do it_. The later you leave it the more
| expensive it becomes. Very often to the point where it is
| prohibitively expensive and you just put up with it being a bit
| broken.
|
| This article even has an example of that - see the vcpkg entry.
| mi_lk wrote:
| > Do the easy thing while it works, and when it stops working,
| fix the problem
|
| Another way to phrase this mindset is "fuck around and find
| out" in gen-Z speak. It's usually practical to an extent but
| I'm personally not a fan
| sagarm wrote:
| I've mostly heard FAFO used to describe something obviously
| stupid.
|
| Building on the same thing people use for code doesn't seem
| stupid to me, at least initially. You might have to migrate
| later if you're successful enough, but that's not a sign of
| bad engineering. It's just building for where you are, not
| where you expect to be in some distant future
| zephen wrote:
| Not at all.
|
| When you fuck around optimizing prematurely, you find out
| that you're too late and nobody cares.
|
| Oh, well, optimization is always fun, so there's that.
| bencornia wrote:
| > Grab's engineering team went from 18 minutes for go get to 12
| seconds after deploying a module proxy. That's not a typo.
| Eighteen minutes down to twelve seconds.
|
| > The problem was that go get needed to fetch each dependency's
| source code just to read its go.mod file and resolve transitive
| dependencies. Cloning entire repositories to get a single file.
|
| I have also had inconsistent performance with go get. Never
| enough to look closely at it. I wonder if I was running into the
| same issue?
| zahlman wrote:
| > needed to fetch each dependency's source code just to read
| its go.mod file and resolve transitive dependencies.
|
| Python used to have this problem as well (technically still
| does, but a large majority of things are available as a wheel
| and PyPI generally publishes a separate .metadata file for
| those wheels), but at least it was only a question of
| downloading and unpacking an archive file, not cloning an
| entire repo. Sheesh.
|
| Why would Go need to do that, though? Isn't the go.mod file in
| a specific place relative to the package root in the repo?
| klooney wrote:
| Go's lock files arrived at around the same time as the proxy,
| before then you didn't have transitive dependencies pre
| baked.
| fireflash38 wrote:
| How long ago were you having issues? That was changed in go
| 1.13.
| c-linkage wrote:
| This seems like a tragedy of the commons -- GitHub is free after
| all, and it has all of these great properties, so why not? -- but
| this kind of decision making occurs whenever externalities are
| present.
|
| My favorite hill to die on (externality) is user time. Most
| software houses spend so much time focusing on how expensive
| engineering time is that they neglect user time. Software houses
| optimize for feature delivery and not user interaction time. Yet
| if I spent one hour making my app one second faster for my
| million users, I can save 277 user hour per year. But since user
| hours are an externality, such optimization never gets done.
|
| Externalities lead to users downloading extra gigabytes of data
| (wasted time) and waiting for software, all of which is waste
| that the developer isn't responsible for and doesn't care about.
| ekjhgkejhgk wrote:
| I wouldn't call it tragedy of the commons, because it's not a
| commons. It's owned by microsoft. They're calculating that it's
| worth it for them, so I say take as much as you can.
|
| Commons would be if it's owned by nobody and everyone benefits
| from its existence.
| TeMPOraL wrote:
| Still, because reality doesn't respect boundaries of human-
| made categories, and because people never define their
| categories exhaustively, we can safely assume that something
| almost-but-not-quite like a commons, is subject to an almost-
| but-not-quite tragedy of the commons.
| reactordev wrote:
| An A- is still an A kind of thinking. I like this approach
| as not everything perfectly fits the mold.
| ttiurani wrote:
| The whole notion of the "tragedy of the commons" needs to
| be put to rest. It's an armchair thought experiment that
| was disproven at the latest in the 90s by Elinor Ostrom
| with actual empirical evidence of commons.
|
| The "tragedy", if you absolutely need to find one, is only
| for unrestricted, free-for-all commons, which is obviously
| a bad idea.
| b00ty4breakfast wrote:
| yeah, it's a post-hoc rationalization for the enclosure
| and privatization of said commons.
| TeMPOraL wrote:
| And here I thought the standard, obvious solution to
| tragedy of the commons is _centralized governance_.
| dpark wrote:
| People invoke the tragedy of the commons in bad faith to
| argue for privatization because "the alternative is
| communism". i.e. Either an individual or the government
| has to own the resource.
|
| This is of course a false dichotomy because governance
| can be done at any level.
| AnthonyMouse wrote:
| It also seems to omit the possibility that the thing
| could be privately operated but not for profit.
|
| Let's Encrypt is a solid example of something you could
| reasonably model as "tragedy of the commons" (who is
| going to maintain all this certificate verification and
| issuance infrastructure?) but then it turns out the value
| of having it is a million times more than the cost of
| operating it, so it's quite sustainable given a modicum
| of donations.
|
| Free software licenses are another example in this
| category. Software frequently has a much higher value
| than development cost and incremental improvements
| decentralize well, so a license that lets you use it for
| free but requires you to contribute back improvements
| tends to work well because then people see something that
| would work for them except for this one thing, and it's
| cheaper to add that themselves or pay someone to than to
| pay someone who has to develop the whole thing from
| scratch.
| Saline9515 wrote:
| Ostrom showed that it wasn't necessarily a tragedy, if
| tight groups involved decided to cooperate. This common
| in what we call "trust-based societies", which aren't
| universal.
|
| Nonetheless, the concept is still alive, and anthropic
| global warming is here to remind you about this.
| wongarsu wrote:
| A high-trust community like a village can prevent a
| tragedy of the commons scenario. Participants feel
| obligations to the community, and misusing the commons
| actually does have real downsides for the individual
| because there are social feedback mechanisms. The classic
| examples like people grazing sheep or cutting wood are
| bad examples that don't really work.
|
| But that doesn't mean the tragedy of the commons can't
| happen in other scenarios. If we define commons a bit
| more generously it does happen very frequently on the
| internet. It's also not difficult to find cases of it
| happening in larger cities, or in environments where
| cutthroat behavior has been normalized
| TeMPOraL wrote:
| > _A high-trust community like a village can prevent a
| tragedy of the commons scenario. Participants feel
| obligations to the community, and misusing the commons
| actually does have real downsides for the individual
| because there are social feedback mechanisms._
|
| That works while the size of the community is ~100-200
| people, when everyone knows everyone else personally. It
| breaks down rapidly after that. We compensate for that
| with _hierarchies of governance_ , which give rise to
| written laws and bureaucracy.
|
| New tribes break off old tribes, form alliances, which
| form larger alliances, and eventually you end up with
| countries and counties and vovoidships and cities and
| districts and villages, in hierarchies that gain a level
| per ~100x population increase.
|
| This is sociopolitical history of the world in a
| nutshell.
| lukan wrote:
| "and eventually you end up with countries and counties
| and vovoidships and cities and districts and villages, in
| hierarchies that gain a level per ~100x population
| increase."
|
| You say it like this is a law set in stone, because this
| is what happened im history, but I would argue it
| happened under different conditions.
|
| Mainly, the main advantage of an empire over small
| villages/tribes is not at all that they have more power
| than the villages combined, but that they can concentrate
| their power where it is needed. One village did not stand
| a chance against the empire - and the villages were not
| coordinated enough.
|
| But today we would have the internet for better
| communication and coordination, enabling the small
| entieties to coordinate a defense.
|
| Well, in theory of course. Because we do not really have
| autonomous small states, but are dominated by the big
| players. And the small states have mowtly the choice
| which block to align with, or get crushed. But the trend
| might go towards small again.
|
| (See also cheap drones destroying expensive tanks,
| battleships etc.)
| ajuc wrote:
| Internet is working exactly the opposite way to what your
| describing - it's making everything more centralized.
| Once we had several big media companies in each country
| and in each big city. Now we have Google and Facebook and
| tik tok and twitter and then the "whatevers".
|
| NETWORK effect is a real thing
| lukan wrote:
| Yes, but there is a difference between having the choice
| of joining FB or not having a choice at all when the
| empire comes to claim you (like in Ukraine).
| vlovich123 wrote:
| I've heard stories from communist villages where everyone
| knew everyone. Communal parks and property was not
| respected and frequently vandalized or otherwise
| neglected because it didn't have an owner and it was
| treated as something for someone else to solve.
|
| It's easier to explain in those terms than assumptions
| about how things work in a tribe.
| xorcist wrote:
| > That works while the size of the community is ~100-200
| people,
|
| Yet we regularly observe that working with millions of
| people; we take care of our young, we organize, when we
| see that some action hurt our environment we tend to
| limit its use.
|
| It's not obvious why some societies break down early and
| some go on working.
| AnthonyMouse wrote:
| I get the feeling it's the combination of Schelling
| points and surplus. If everyone else is being pro-social,
| i.e. there is a culture of it, and the people aren't so
| hard up that they can reasonably afford to do the same,
| then that's what happens, either by itself (Hofstadter's
| theory of superrationality) or via anything so much as
| light social pressure.
|
| But if a significant fraction of the population is barely
| scraping by then they're not willing to be "good" if it
| means not making ends meet, and when other people see
| widespread defection, they start to feel like they're the
| only one holding up their end of the deal and then the
| whole thing collapses.
|
| This is why the tendency for people to propose rent-
| seeking middlemen as a "solution" to the tragedy of the
| commons is such a diabolical scourge. It extracts the
| surplus that would allow things to work more efficiently
| in their absence.
| TeMPOraL wrote:
| > _Yet we regularly observe that working with millions of
| people; we take care of our young, we organize, when we
| see that some action hurt our environment we tend to
| limit its use._
|
| That's more like human universals. These behaviors
| generally manifest to smaller or larger degree, depending
| on how secure people feel. But those are extremely local
| behaviors. And in fact, one of them is exactly the thing
| I'm talking about:
|
| > _we organize_
|
| We organize. We organize for many reasons, "general
| living" is the main one but we're mostly born into it
| today (few got the chance to be among the founding people
| of a new village, city or country). But the same patterns
| show up in every other organizations people create, from
| companies to charities, from political interests groups
| to rural housewives' circles -- groups that grow past
| ~100 people split up. Sometimes into independent groups,
| sometimes into levels of hierarchies. Observe how
| companies have regional HQs and departments and areas and
| teams; religious groups have circuits and congregations,
| etc. Independent organizations end up creating joint
| ventures and partnerships, or merge together (and
| immediately split into a more complex internal
| structure).
|
| The key factor here is, IMO, for everyone in a given
| group to be in regular contact with everyone else. Humans
| are well evolved for living in such small groups - we
| come with built-in hardware and software to navigate
| complex interpersonal situations. Alignment around shared
| goals and implicit rules is _natural_ at this scale.
| There 's no space for cheaters and free-loaders to
| thrive, because everyone knows everyone else - including
| the cheater and their victims. However, once the group
| crosses this "we're all a big family, in it together"
| size, coordinating everyone becomes hard, and free-
| loaders proliferate. That's where explicit laws come into
| play.
|
| This pattern repeats daily, in organizations people
| create even today.
| lo_zamoyski wrote:
| Even here, the state is the steward of the common good.
| It is a mistaken notion that the state only exists
| because people are bad. Even if people were perfectly
| conscientious and concerned about the common good, you
| still need a steward. It simply wouldn't be a steward who
| would need to use aggressive means to protect the common
| good from malice or abuse.
| ttiurani wrote:
| > But that doesn't mean the tragedy of the commons can't
| happen in other scenarios.
|
| Commons can fail, but the whole point of Hardin calling
| commons a "tragedy" is to suggest it _necessarily_ fails.
|
| Compare it to, say, driving. It can fail too, but you
| wouldn't call it "the tragedy of driving".
|
| We'd be much better off if people didn't throw around
| this zombie term decades after it's been shown to be
| unfounded.
| jandrewrogers wrote:
| > A high-trust community like a village can prevent a
| tragedy of the commons scenario.
|
| No it does not. This sentiment, which many people have,
| is based on a fictional and idealistic notion of what
| small communities are like having never lived in such
| communities.
|
| Empirically, even in high-trust small villages and
| hamlets where everyone knows everyone, the same
| incentives exist and the same outcomes happen. Every
| single time. I lived in several and I can't think of a
| counter-example. People are highly adaptive to these
| situations and their basic nature doesn't change because
| of them.
|
| Humans are humans everywhere and at every scale.
| gmfawcett wrote:
| Ostrom's results didn't disprove ToC. She showed that
| common resources _can_ be communally maintained, not that
| tragic outcomes could never happen.
| dpark wrote:
| She not "disprove" the existence of the tragedy of the
| commons. What she established was that controlling the
| commons can be done communally rather than through
| privatization or through government ownership.
|
| Communal management of a resource is still government,
| though. It just isn't central government.
|
| The thesis of the tragedy of the commons is that an
| uncontrolled resource will be abused. The answer is
| governance at some level, whether individual, collective,
| or government ownership.
|
| > The "tragedy", if you absolutely need to find one, is
| only for unrestricted, free-for-all commons, which is
| obviously a bad idea.
|
| Right. And that's what people are usually talking about
| when they say "tragedy of the commons".
| lo_zamoyski wrote:
| There is an analogy in the sense that for the users a
| resource is, for certain _practical_ intents and purposes,
| functionally common. Social media is like this as well.
|
| But I would make the following clarifications:
|
| 1. A private entity is still the steward of the resource
| and therefore the resource figures into the aims, goals,
| and constraints of the private entity.
|
| 2. The common good is itself under the stewardship of the
| state, as its function is guardian of the common good.
|
| 3. The common good is the default (by natural law) and
| prior to the private good. The latter is instituted in
| positive law for the sake of the former by, e.g., reducing
| conflict over goods.
| TeMPOraL wrote:
| > _There is an analogy in the sense that for the users a
| resource is, for certain practical intents and purposes,
| functionally common. Social media is like this as well._
|
| I think it's both simpler and deeper than that.
|
| _Governments and corporations don 't exist in nature_.
| Those are just human constructs, mutually-recursive
| shared beliefs that emulate agents following some rules,
| as long as you don't think too hard about this.
|
| "Tragedy of the commons" is a _general_ coordination
| problem. The name itself might 've been coined with some
| specific scenarios in mind, but for the phenomenon
| itself, it doesn't matter what kind of entities exploit
| the "commons"; the "private" vs. "public" distinction
| itself is neither a sharp divide, nor does it exist in
| nature. All that matters is that there's some resource
| used by several independent parties, and each of them
| finds it more beneficial to defect than to cooperate.
|
| In a way, it's basically a 3+-player prisonner's dilemma.
| The solution is the same, too: introducing a party that
| forces all other parties to cooperate. That can be a
| private or public or any other kind of org taking
| ownership of the commons and enforcing quotas, or in case
| of prisonners, a mob boss ready to shoot anyone who
| defects.
| bee_rider wrote:
| That seems to assume some sort of... maybe unfounded
| linearity or something? I mean, I'm not sure I agree that
| GitHub is nearly a commons in any sense, but let's put that
| aside as a distraction...
|
| The idea of the tragedy of the commons relies on this
| feedback loop of having these unsustainably growing herds
| (growing because they can exploit the zero-cost-to-them
| resources of the commons). Feedback loops are notoriously
| sensitive to small parameter changes. MS could presumably
| impose some damping if they wanted.
| TeMPOraL wrote:
| > _That seems to assume some sort of... maybe unfounded
| linearity or something_
|
| Not linearity but _continuity_ , which I think is a well-
| founded assumption, given that it's our categorization
| that simplifies the world by drawing sharp boundaries
| where no such bounds exist in nature.
|
| > _The idea of the tragedy of the commons relies on this
| feedback loop of having these unsustainably growing herds
| (growing because they can exploit the zero-cost-to-them
| resources of the commons)_
|
| AIUI, zero-cost is not a necessary condition, a _positive
| return_ is enough. Fishermen still need to buy fuel and
| nets and pay off loans for the boats, but as long as
| their expected profit is greater than that, they 'll
| still overfish and deplete the pond, unless stronger
| external feedback is introduced.
|
| Given that the solution to tragedy of the commons is
| having the commons owned by someone who can boss the
| users around, GitHub being owned by MS makes it more of a
| commons in practice, not less.
| thayne wrote:
| And indeed MS/GitHub does impose some "damping" in the
| form of things like API request throttling, CPU limits on
| CI, asking Homebrew not to use shallow cloning, etc. And
| those limits are one of the reasons given why using git
| as a database isn't good.
| jasonkester wrote:
| It has the same effect though. A few bad actors using this
| "free" thing can end up driving the cost up enough that
| Microsoft will have to start charging for it.
|
| The jerks get their free things for a while, then it goes
| away for everyone.
| Y_Y wrote:
| I think the jerks are the ones who bought and enshittified
| GitHub after it had earned significant trust and become an
| important part of FOSS infrastructure.
| irishcoffee wrote:
| Scoping it to a local maxima, the only thing worse than
| git is github. In an alternate universe hg won the clone
| wars and we are all better off for it.
| MarsIronPI wrote:
| Excuse me if this is obvious, but how is Mercurial better
| than Git from a repo format perspective?
| dahart wrote:
| Why do you blame MS for predictably doing what MS does,
| and not the people who sold that trust & FOSS infra to MS
| for a profit? Your blame seems misplaced.
|
| And out of curiosity, aside from costing more for some
| people, what's worse exactly? I'm not a heavy GitHub
| user, but I haven't really noticed anything in the core
| functionality that would justify calling it enshittified.
| mastax wrote:
| Plenty of blame to go around.
|
| Probably the worst thing MS did was kill GitHub's nascent
| CI project and replace it with Azure DevOps. Though to be
| fair the fundamental flaws with that approach didn't
| really become apparent for a few years. And GitHub's
| feature development pace was far too slow compared to its
| competitors at the time. Of course GitHub used to be a
| lot more reliable...
|
| Now they're cramming in half baked AI stuff everywhere
| but that's hardly a MS specific sin.
|
| MS GitHub has been worse about DMCA and sanctioned
| country related takedowns than I remember pre acquisition
| GitHub being.
|
| Did I miss anything?
| Y_Y wrote:
| I don't blame them uniquely. I think it's a travesty the
| original GitHub sold out, but it's just as predictable.
| Giant corps will evilly make the line go up, individual
| regular people will have a finite amount of money for
| which they'll give up anything and everything.
|
| As for how the site has become worse, plenty of others
| have already done a better job than I could there. Other
| people haven't noticed or don't care and that's ok too I
| guess.
| PunchyHamster wrote:
| Well, till you choose to host something yourself and it
| becomes popular
| rvba wrote:
| I doubt anyone is calculating
|
| Remember how GTA5 took 10 minutes to start and nobody cared?
| Lots of software is like this.
|
| Some Blizzard games download 137 MB file every time you run
| them and take few minutes to start (and no, this is not due
| to my computer).
| ericyd wrote:
| Tragedy of the Microsoft just doesn't sound as nice though
| groundzeros2015 wrote:
| A public park suffers from tragedy of the commons even though
| it's managed by the city.
| dahart wrote:
| > so I say take as much as you can. Commons would be if it's
| owned by nobody
|
| This isn't what "commons" means in the term 'tragedy of the
| commons', and the obvious end result of your suggestion to
| take as much as you can is to cause the loss of access.
|
| Anything that is free to use is a commons, regardless of
| ownership, and when some people use too much, everyone loses
| access.
|
| Finite digital resources like bandwidth and database sizes
| within companies are even listed as examples in the Wikipedia
| article on Tragedy of the Commons.
| https://en.wikipedia.org/wiki/Tragedy_of_the_commons
| nkmnz wrote:
| No, the word and its meaning both point to the fact that
| there's no exclusive ownership of a commons. This is
| importantl, since ownership is associated with bearing the
| cost of usage (i.e., deprecation) which would lead an owner
| to avoid the tragedy of the commons. Ownership is regularly
| the solution to the tragedy (socialism didn't work).
|
| The behavior that you warn against is that of a free rider
| that make use of a positive externality of GitHub's
| offering.
| dahart wrote:
| That is _one_ meaning of "commons", but not all of them,
| and you might be mistaking which one the phrase 'tragedy
| of the commons' is using.
|
| "Commons can also be defined as a social practice of
| governing a resource not by state or market but by a
| community of users that self-governs the resource through
| institutions that it creates."
|
| https://en.wikipedia.org/wiki/Commons
|
| The actual mechanism by which ownership resolves tragedy
| of the commons scenarios is by making the resource non-
| free, by either charging, regulating, or limiting access.
| The effect still occurs when something is owned but free,
| and its name is still 'tragedy of the commons', even when
| the resource in question is owned by private interests.
| bawolff wrote:
| How does that differ from what the person you are arguing
| against is saying?
| dahart wrote:
| Ownership, I guess. The 2 parent comments are claiming
| that "tragedy of the commons" doesn't apply to privately
| owned things. I'm suggesting that it does.
|
| Edit: oh, I do see what you mean, and yes I misunderstood
| the quote I pulled from WP - it's talking about non-
| ownership. I could pick a better example, but I think
| that's distracting from the fact that 'tragedy of the
| commons' is a term that today doesn't depend on the
| definition of the word 'commons'. It's my mistake to have
| gotten into any debate about what "commons" means, I'm
| only saying today's usage and meaning of the phrase
| doesn't depend on that definition, it's a broader
| economic concept.
| drob518 wrote:
| Right. Microsoft could easily impose a transfer fee if over a
| certain amount that would allow "normal" OSS development of
| even popular software to happen without charge while imposing
| a cost to projects that try to use GitHub like a database.
| loloquwowndueo wrote:
| Just a reminder that GitHub is not git.
|
| The article mentions that most of these projects did use GitHub
| as a central repo out of convenience so there's that but they
| could also have used self-hosted repos.
| justincormack wrote:
| They probably would have experienced issues way sooner, as
| the self hosted tools don't scale nearly as well.
| machinationu wrote:
| Explain to me how you self-host a git repo which is accessed
| millions of time a day from CI jobs pulling packages.
| ozim wrote:
| FTFY:
|
| Explain to me how you self-host a git repo without spending
| any money and having no budget which is accessed millions
| of time a day from CI jobs pulling packages.
| freedomben wrote:
| I'm not sure whether this question was asked in good faith,
| but is actually a damn good one.
|
| I've looked into self hosting and git repo that has
| horizontal scalability, and it is indeed very difficult. I
| don't have the time to detail it in a comment here, but for
| anyone who is curious it's very informative to look at how
| GitLab handled this with gitaly. I've also seen some clever
| attempts to use object storage, though I haven't seen any
| of those solutions put heavily to the test.
|
| I'd love to hear from others about ideas and approaches
| they've heard about or tried
|
| https://gitlab.com/gitlab-org/gitaly
| adrianN wrote:
| You git init ---bare on a host with sufficient resources.
| But I would recommend thinking about your CI flow too.
| machinationu wrote:
| no, hundred of thousands of thousands of individual
| projects CI jobs. OP was talking about package managers
| for the whole world, not for one company
| adrianN wrote:
| If people depend on remote downloads from different
| companies for their CI pipelines they're doing it wrong.
| Every sensible company sets up a mirror or at least a
| cache on infra that they control. Rate limiting downloads
| is the natural course of action for the provider of a
| package registry. Once you have so many unique users that
| even civilized use of your infrastructure becomes too
| much you can probably hire a few people to build
| something more scalable.
| machinationu wrote:
| numpy had 16M downloads yesterday, at 10 MB that's 160 TB
| of traffic. It's one package. And there are no rate
| limits on pypi.
|
| https://clickpy.clickhouse.com/dashboard/numpy
| fweimer wrote:
| These days, people solve similar problems by wrapping their
| data in an OCI container image and distribute it through
| one of the container registries that do not have a
| practically meaningful pull rate limit. Not really a joke,
| unfortunately.
| mystifyingpoi wrote:
| Even Amazon encourages this, probably not intentionally,
| more like as a bandaid for bad EKS config that people can
| do by mistake, but still - you can pull 5 terabytes from
| ECR for free under their free tier each month.
| XorNot wrote:
| I'd say it'd just Kubernetes in general should've shipped
| with a storage engine and an installation mechanism.
|
| It's a very hacky feeling addon that RKE2 has a
| distributed internal registry _if_ you enable it and use
| it in a very specific way.
|
| For the rate at which people love just shipping a Helm
| chart, it's actually absurdly hard to ship a self
| contained installation without just trying to hit
| internet resources.
| zahlman wrote:
| > Most software houses spend so much time focusing on how
| expensive engineering time is that they neglect user time.
| Software houses optimize for feature delivery and not user
| interaction time. Yet if I spent one hour making my app one
| second faster for my million users, I can save 277 user hour
| per year. But since user hours are an externality, such
| optimization never gets done.
|
| This is what people mean about speed being a feature. But "user
| time" depends on more than the program's performance. UI design
| is also very important.
| solatic wrote:
| If you think too hard about this, you come back around to Alan
| Kay's quote about how people who are really serious about
| software should build their own hardware. Web applications, and
| in general loading pretty much anything over the network, is a
| horrible, no-good, really bad user experience, and it always
| will be. The only way to really respect the user is with native
| applications that are local-first, and if you take that
| _really_ far, you build (at the very least) peripherals to make
| it even better.
|
| The number of companies that have this much respect for the
| user is _vanishingly_ small.
| ghosty141 wrote:
| Yes because users don't appreciate this enough to pay for the
| time this takes.
| hombre_fatal wrote:
| Software I don't have to install at all "respects me" the
| most.
|
| Native software being an optimum is mostly an engineer
| fantasy that comes from imagining what you can build.
|
| In reality that means having to install software like Meta's
| WhatsApp, Zoom, and other crap I'd rather run in a browser
| tab.
|
| I want very little software running natively on my machine.
| freedomben wrote:
| Yes, amen. The more invasive and abusive software gets, the
| less I want it running on my machine natively. Native
| installed applications for me now are limited only to apps
| I trust, and even those need to have a reason to be native
| apps rather than web apps to get a place in my app drawer
| solatic wrote:
| Your browser is acting like a condom, in that respect (pun
| not intended).
|
| Yes, there are many cases when condoms are indicative of
| respect between parties. But a great many people would
| disagree that the best, most respectful relationships
| involve condoms.
|
| > Meta
|
| Does not sell or operate respectful software. I will agree
| with you that it's best to run it in a browser (or similar
| sandbox).
| tormeh wrote:
| Desktop operating systems really dropped the ball on
| protecting us from the software we run. Even mobile OSs
| are so-so. So the browser is the only protection we
| reasonably have.
|
| I think this is sad.
| shash wrote:
| You mean you'd rather run unverified scripts using a good
| order of magnitude more resources with a slower experience
| and have an entire sandboxing contraption to keep said
| unverified scripts from doing anything to your machine...
|
| I know the browser is convenient, but frankly, its been a
| horror show of resource usage and vulnerabilities and
| pathetic performance
| whstl wrote:
| The #1 reason the web experience universally sucks today
| is because companies add an absurd amount of third-party
| code on their pages for tracking, advertisement, spying
| on you or whatever non-essential purpose. That, plus an
| excessive/unnecessary amount of visual decoration.
|
| The idea that somehow those companies would respect your
| privacy were they running a native app is extremely
| naive.
|
| We can already see this problem on video games, where
| copy protection became resource-heavy enough to cause
| performance issues.
| cosmic_cheese wrote:
| Web apps are great until you want to revert to an older
| version from before they became actively user-hostile or
| continue to use them past EoL or company demise.
|
| In contrast as long as you have a native binary, one way or
| another you can make the thing run and nobody can stop you.
| phkahler wrote:
| >> The number of companies that have this much respect for
| the user is vanishingly small.
|
| I think companies shifted to online apps because #1 it solved
| the copy protection problem. FOSS apps are not in any hurry
| to become centralized because they dont care about that
| issue.
|
| Local apps and data are a huge benefit of FOSS and I think
| every app website should at least mention that.
|
| "Local app. No ads. You own your data."
| xorcist wrote:
| Another important reason to move to online applications is
| that you can change the terms of the deal at any time. This
| may sound more nefarious than it needs to be, it just means
| you do not have to commit fully to your licensing terms
| before the first deal is made, which is tempting for just
| about anyone.
| inapis wrote:
| >Yet if I spent one hour making my app one second faster for my
| million users, I can save 277 user hour per year. But since
| user hours are an externality, such optimization never gets
| done.
|
| I have never been convinced by this argument. The aggregate
| number sounds fantastic but I don't believe that any meaningful
| work can be done by each user saving 1 second. That 1 second
| (and more) can simply be taken by me trying to stretch my body
| out.
|
| OTOH, if the argument is to make software smaller, I can get
| behind that since it will simply lead to more efficient usage
| of existing resources and thus reduce the environmental impact.
|
| But we live in a capitalist world and there needs to be
| external pressure for change to occur. The current RAM
| shortage, if it lasts, might be one of them. Otherwise, we're
| only day dreaming for a utopia.
| adrianN wrote:
| Time saved to increased productivity or happiness or whatever
| is not linear but a step function. Saving one second doesn't
| help much, but there is a threshold (depending on the
| individual) where faster workflows lead to a better
| experience. It does make a difference whether a task takes a
| minute or half a second, at least for me.
| Aerroon wrote:
| One second is long enough that it can put a user off from
| using your app though. Take notifications on phones for
| example. I know several people who would benefit from a
| habitual use of phone notifications, but they never stick to
| using them because the process of opening (or switching over
| to) the notification app and navigating its UI to leave a
| notification takes too long. Instead they write a physical
| sticky note, because it has a faster "startup time".
| tehbeard wrote:
| All depends on the type of interaction.
|
| A high usage one, absolutely improve the time of it.
|
| Loading the profile page? Isn't done often so not really
| worth it unless it's a known and vocal issue.
|
| https://xkcd.com/1205/ gives a good estimate.
| jorvi wrote:
| But there isn't just one company deciding externalizing cost
| on the rest of us is a great way to boost profit since it
| costs them very little. Especially for a monopoly like
| YouTube that can decide that eating up your battery is fine
| if it saves them a few cents in bandwidth costs.
|
| Not all of those externalizing companies abuse your time but
| whatever they abuse can be expressed in a $ amount and $ can
| be converted to a median's person time via median wage. Hell,
| free time is _more_ valuable than whatever you produce during
| work.
|
| Say all that boils down to companies collectively stealing 20
| minutes of your time each day. 140 minutes each week. 7280
| (!) minutes each year, which is 5.05 days, which makes it
| almost a year over the course of 70 years.
|
| So yeah, don't do what you do and sweettalk the fact that
| companies externalize costs (private the profits, socialize
| the losses). They're sucking your blood.
| ozim wrote:
| About apps done by software houses, even though we should
| strive for doing good job and I agree with sentiment...
|
| First argument would be - take at least two 0's from your
| estimation, most of applications will have maybe thousands of
| users, successful ones will maybe run with 10's of thousands.
| You might get lucky to work on application that has 100's of
| thousands, millions of users and you work in FAANG not a
| typical "software house".
|
| Second argument is - most users use 10-20 apps in typical
| workday, your application is most likely irrelevant.
|
| Third argument is - most users would save much more time
| learning how to use applications (or to use computer) properly
| they use on daily basis, than someone optimizing some function
| from 2s to 1s. But of course that's hard because they have
| 10-20 apps daily plus god know how many other not on daily
| basis. Though still I see people doing super silly stuff in
| tools like Excel or even not knowing copy paste - so not even
| like any command line magic.
| Y-bar wrote:
| You'll enjoy "Saving Lives" by Andy Hertzfied:
| https://www.folklore.org/Saving_Lives.html
|
| > "The Macintosh boots too slowly. You've got to make it
| faster!"
| kkjjjjw wrote:
| https://news.ycombinator.com/item?id=44843223#44879509
| pastor_williams wrote:
| This was something that I heavily focused on for my feature
| area a year ago - new user sign up flow. But the decreased
| latency was really in pursuit of increased activation and
| conversion. At least the incentives aligned briefly.
| robmccoll wrote:
| I don't think most software houses spend enough time even
| focusing on engineering time. CI pipelines that take tens of
| minutes to over an hour, compile times that exceed ten seconds
| when nothing has changed, startup times that are much more than
| a few seconds. Focus and fast iteration are super important to
| writing software and it seems like a lot of orgs just kinda
| shrug when these long waits creep into the development process.
| JohnHaugeland wrote:
| > This seems like a tragedy of the commons -- GitHub is free
| after all, and it has all of these great properties, so why
| not?
|
| because it's bad at this job, and sqlite is also free
|
| this isn't about "externalities"
| vlovich123 wrote:
| I think it's naive to think engineers or managers don't realize
| this or don't think in these ways.
|
| https://www.folklore.org/Saving_Lives.html
| brightball wrote:
| User time is typically a mix of performance tuning and UX
| design isn't it?
| Aurornis wrote:
| > Most software houses spend so much time focusing on how
| expensive engineering time is that they neglect user time.
| Software houses optimize for feature delivery and not user
| interaction time.
|
| I don't know what you mean by software houses, but every
| consumer facing software product I've worked on has tracked
| things like startup time and latency for common operations as a
| key metric
|
| This has been common wisdom for decades. I don't know how many
| times I've heard the repeated quote about how Amazon loses $X
| million for every Y milliseconds of page loading time, as an
| example.
| dijit wrote:
| I worked in e-commerce SaaS in 2011~ and this was true _then_
| but I find it less true these days.
|
| Are you sure that you're not the driving force behind those
| metrics; or that you're not self-selecting for like-minded
| individuals?
|
| I find it really difficult to convince myself that even large
| players (Discord) are measuring startup time. Every time I
| start the thing I'm greeted by a 25s wait and a `RAND()%9`
| number of updates that each take about 5-10s.
| jama211 wrote:
| Discord's user base is 99% people who leave it running 100%
| of the time, it's not a typical situation
| dijit wrote:
| I think that they make the startup so horrible that
| people are more likely to leave it running.
| hexer292 wrote:
| As a discord user, it's the kind of platform that I would
| want to have running to receive notifications, sort of
| like the SMS of gaming.
|
| A large part of my friend group use discord as the
| primary method of communication, even in an in person
| context (was at a festival a few months ago with a
| friend, and we would send texts over discord if we got
| split up) so maybe its not a common use case.
| drob518 wrote:
| Yep, indeed. Which is the main reason I don't run Discord.
| spockz wrote:
| I have the same experience on windows. On the other hand,
| starting up discord on my cachyos install is virtually
| instant. So maybe there is also a difference between the
| platform the developers use and that their users use.
| rovr138 wrote:
| There was a thread here earlier this month,
|
| > Helldivers 2 devs slash install size from 154GB to 23GB
|
| https://news.ycombinator.com/item?id=46134178
|
| Section of the top comment says,
|
| > It seems bizarre to me that they'd have accepted such a
| high cost (150GB+ installation size!) without entirely
| verifying that it was necessary!
|
| and the reply to it has,
|
| > They're not the ones bearing the cost. Customers are.
| ux266478 wrote:
| That's not how it works. The demand for engineering hours
| is an order of magnitude higher than the supply for any
| given game, you have to pick and choose your battles
| because there's always much, much more to do. It's not
| bizarre that nobody verified texture storage was being done
| in an optimal way at launch, without sacrificing load times
| at the altar or visual fidelity, particularly given the
| state the rest of the game was in. Who the hell has time to
| do that when there are crashes abound and the network stack
| has to be rewritten at a moments notice?
|
| Gamedev is very different from other domains, being in the
| 90th percentile for complexity and codebase size, and the
| 99th percentile for structural instability. It's a foregone
| conclusion that you will rewrite huge chunks of your
| massive codebase many, many times within a single year to
| accomidate changing design choices, or if you're lucky, to
| improve an abstraction. Not every team gets so lucky on
| every project. Launch deadlines are hit when there's a huge
| backlog of additional stuff to do, sitting atop a mountain
| of cut features.
| swiftcoder wrote:
| > It's not bizarre that nobody verified texture storage
| was being done in an optimal way at launch
|
| The inverse, however, is bizarre. That they spent
| potentially quite a bit of engineering effort
| implementing the (extremely non-optimal) system that
| duplicates all the assets half a dozen time to
| potentially save precious seconds on spinning rust - all
| without validating it was worth implementing in the first
| place.
| rovr138 wrote:
| Yes.
|
| They talk about it being an optimization. They also talk
| about the bottleneck being level generation, which
| happens at the same time as loading from disk.
| MBCook wrote:
| Was Helldivers II built from the ground up? Or grown from
| the v1 codebase?
|
| The first was on PS3 and PS4 where they had to deal with
| spinning disks and that system would absolutely be
| necessary.
|
| Also if the game ever targeted the PS4 during
| development, even though it wasn't released there, again
| that system would be NEEDED.
| viraptor wrote:
| There was also the GTA wasting minutes to load/parse JSON
| files at startup. https://nee.lv/2021/02/28/How-I-cut-GTA-
| Online-loading-times...
|
| And Skylines rendering teeth on models miles away https://w
| ww.reddit.com/r/CitiesSkylines/comments/17gfq13/the...
|
| Sometimes the performance is really ignored.
| kibwen wrote:
| _> They're not the ones bearing the cost. Customers are._
|
| I think this is uncharitably erasing the context here.
|
| AFAICT, the reason that Helldivers 2 was larger on disk is
| because they were following the standard industry practice
| of deliberately duplicating data in such a way as to
| improve locality and thereby reduce load times. In other
| words, this seems to have been a deliberate attempt to
| improve player experience, not something done out of sheer
| developer laziness. The fact that this attempt at
| optimization is obsolete these days just didn't filter down
| to whatever particular decision-maker was at the reins on
| the day this decision was made.
| pjmlp wrote:
| An exception that confirms the rule.
| mindslight wrote:
| > _every consumer facing software product I've worked on has
| tracked things like startup time and latency for common
| operations as a key metric_
|
| Are they evaluating the shape of that line with the same goal
| as the stonk score? Time spent by users is an "engagement"
| metric, right?
| eviks wrote:
| The issue here is not tracking, but developing. Like, how do
| you explain the fact that whole classes of software have
| gotten worse on those "key metrics"? (and that includes web-
| selling webpages)
| ponector wrote:
| Contrary, every consumer facing product I've worked had no
| performance metrics tracked. And for enterprise software it
| was even worse as the end user is not the one who makes a
| decision to buy and use software.
|
| >>what you mean by software houses
|
| How about Microsoft? Start menu is a slow electron app.
| philipallstar wrote:
| > How about Microsoft? Start menu is a slow electron app.
|
| If your users are trapped due to a lack of competition then
| this can definitely happen.
| julianz wrote:
| The Start menu is not an Electron app. Don't believe
| everything you read on the internet.
| Spooky23 wrote:
| That makes the usability and performance of the windows
| start menu even more embarrassing.
|
| The decline of Windows as a user facing product is
| amazing, especially as they are really good at developing
| things they care about. The "back of house" guts of
| Windows has improved alot, for example. They should just
| have a cartoon Bill Gates pop up like clippy and flip you
| the bird at this point.
| kortilla wrote:
| People believing it says something about the start menu
| TehShrike wrote:
| hey, haven't seen that one in the wild for a little bit
| :-D https://www.smbc-comics.com/comic/aaaah
| odo1242 wrote:
| React Native, not Electron. Though it is slower than it
| was
| moregrist wrote:
| > I don't know how many times I've heard the repeated quote
| about how Amazon loses $X million for every Y milliseconds of
| page loading time, as an example.
|
| This is true for sites that are trying to make sales. You can
| quantify how much a delay affects closing a sale.
|
| For other apps, it's less clear. During its high-growth
| years, MS Office had an abysmally long startup time.
|
| Maybe this was due to MS having a locked-in base of
| enterprise users. But given that OpenOffice and LibreOffice
| effectively duplicated long startup times, I don't think it's
| just that.
|
| You also see the Adobe suite (and also tools like GIMP) with
| some excruciatingly long startup times.
|
| I think it's very likely that startup times of office apps
| have very little impact on whether users will buy the
| software.
| croes wrote:
| Then why do many software house favor cloud software over on
| premise?
|
| They often have a recognizable delay to user data input
| compared to local software
| threatofrain wrote:
| > Most software houses spend so much time focusing on how
| expensive engineering time is that they neglect user time.
| Software houses optimize for feature delivery and not user
| interaction time.
|
| Oh no no no. Consumer-facing companies will burn 30% of your
| internal team complexity budget on shipping the first "frame"
| of your app/website. Many people treat Next as synonymous with
| React, and Next's big deal was helping you do just this.
| massysett wrote:
| > Externalities lead to users downloading extra gigabytes of
| data (wasted time) and waiting for software, all of which is
| waste that the developer isn't responsible for and doesn't care
| about.
|
| This is perfectly sensible behavior when the developers are
| working for free, or when the developers are working on a
| project that earns their employer no revenue. This is the case
| for several of the projects at issue here: Nix, Homebrew,
| Cargo. It makes perfect sense to waste the user's time, as the
| user pays with nothing else, or to waste Github's bandwidth,
| since it's willing to give bandwidth away for free.
|
| Where users pay for software with money, they may be more picky
| and not purchase software that indiscriminately wastes their
| time.
| BobbyTables2 wrote:
| Microsoft would have long gone out of business if users cared
| about their time being wasted.
|
| Windows 11 should not be more sluggish than Windows 7.
| imiric wrote:
| > GitHub is free after all, and it has all of these great
| properties, so why not?
|
| The answer is in TFA:
|
| > The underlying issue is that git inherits filesystem
| limitations, and filesystems make terrible databases.
| gritzko wrote:
| Let's make a thought experiment. Suppose that I have a data
| format and a store that resolves the issues in the post. It is
| like git meets JSON meets key-value.
| https://github.com/gritzko/go-rdx
|
| What is the probability of it being used? About 0%, right?
| Because git is proven and GitHub is free. Engineering aspects
| are less important.
| stkdump wrote:
| Sorry, I am turned off by the CRDT in there. It immediately
| smells of overengineering to me. Not that I believe git is a
| better database. But why not just SQL?
| gritzko wrote:
| Merges require revisioning. JSON or SQL do not have that in
| the model. This variant of CRDT is actually quite
| minimalistic.
| stkdump wrote:
| I would argue LWW is the opposite of a merge. It is
| better to immediately know at the time of writing that
| there is a conflict. CRDTs either solve or (in this case)
| don't solve a problem that doesn't really exist,
| especially for package managers.
| gritzko wrote:
| Git solves that problem and it definitely exists.
| Speaking of package managers, it really depends. Like,
| can we use one SQLite file for that? So easy, why no one
| is doing that?
| 3371 wrote:
| The user hour analogy sounds weird tho, 1s feels 1s regardless
| how many users you have. It's like the classic Asian teachers'
| logic of "if you come in 1 min late you are wasting N minutes
| for all of us in this class." It just does not stack like that.
| BenjiWiebe wrote:
| If the class takes N minutes and one person arrives 1 minute
| late, and the rest of the class is waiting for them, it does
| stack. Every one of those students lost a minute. Far worse
| than one student losing one minute.
| bawolff wrote:
| > Software houses optimize for feature delivery and not user
| interaction time. Yet if I spent one hour making my app one
| second faster for my million users, I can save 277 user hour
| per year. But since user hours are an externality, such
| optimization never gets done.
|
| Google and amazon are famous for optimizing this. Its not an
| externality to them though, even 10s of ms can equal an extra
| sale.
|
| That said, i don't think its fair to add time up like that.
| Saving 1 second for 600 people is not the same as saving 10
| minutes for 1 person. Time in small increments does not have
| the same value as time in large increments.
| esafak wrote:
| 1. If you can price the cost of the externality, you can
| justify optimizing it.
|
| 2. Monopolies and situations with the principal/agent dilemma
| are less sensitive to such concerns.
| bawolff wrote:
| > 1. If you can price the cost of the externality, you can
| justify optimizing it.
|
| An externality is usually a cost you don't pay (or pay only
| a negligible amount of). I don't see how pricing it helps
| justify optimizing it.
| miyuru wrote:
| Funnily enough, I clicked the homebrew GitHub link in the post,
| only to get a rate limited error page from GitHub.
| mikkupikku wrote:
| People who put off learning SQL for later end up using anything
| other than a database as their database.
| redog wrote:
| SQL killed the set theory star
| groundzeros2015 wrote:
| Is sql over ssh a thing?
| steeleduncan wrote:
| The other conclusion to draw is "Git is a fantastic choice of
| database for starting your package manager, almost all popular
| package managers began that way."
| saidinesh5 wrote:
| I think the conclusion is more that package definitions can
| still be maintained on git/GitHub but the package manager
| clients should probably rely on a cache/db/a more efficient
| intermediate layer.
|
| Mostly to avoid downloading the whole repo/resolve deltas from
| the history for the few packages most applications tend to
| depend on. Especially in today's CI/CD World.
| reactordev wrote:
| This is exactly the right approach. I did this for my package
| manager.
|
| It relies on a git repo branch for stable. There are yaml
| definitions of the packages including urls to their repo,
| dependencies, etc. Preflight scripts. Post install checks.
| And the big one, the signatures for verification. No
| binaries, rpms, debs, ar, or zip files.
|
| What's _actually_ installed lives in a small SQLite database
| and searching for software does a vector search on each
| packages yaml description.
|
| Semver included.
|
| This was inspired by brew/portage/dpkg for my hobby os.
| pseufaux wrote:
| This is how WinGet works. It has a small SQLite db it
| downloads from a hosted url. The DB contains some minimal
| metadata and a url path to access the full metadata. This way
| WinGet only has to make API calls for packages it's actually
| interacting with. As a package manager, it has plenty of
| problems still, but it's a simple, elegant solution for the
| git as a DB issue.
| bluGill wrote:
| Git isn't a fantastic choice unless you know nothing about
| databases. A search would show plenty of research on databases
| and what works when/why.
| kibwen wrote:
| For the purposes of the article, git isn't just being used as
| a database, it's being used as a protocol to replicate the
| database to the client to allow for offline operation and
| then keep those distributed copies in sync. And even for that
| purpose you can do better than git if you know what you're
| doing, but knowledge of databases alone isn't going to help
| you (let alone make your engineering more economical than
| relying on free git hosting).
| freedomben wrote:
| Exactly. It's not just about the best solution to the
| problem, it's also heavily about the economics around it.
| If I wanted to create a new package manager today, I could
| get started by utilizing Git and existing git hosting
| solutions with very little effort, and effort translates to
| time, and time is a scarce resource. If you don't know
| whether your package manager will take off or not, it may
| not be the best use of your scarce resources to invest in a
| robust and optimized solution out of the gate. I wish that
| weren't the case, I would love to have an infinite amount
| of time, but wishing is not going to make it happen
| adastra22 wrote:
| Git is an absolute shit database for a package manager even in
| the beginning. It's just that GitHub subsidizes hosting and
| that is hard to pass up.
| fn-mote wrote:
| Sure, but can you back up the expletive with some reason why
| you think that?
|
| As it is, this comment is just letting out your emotion, not
| engaging in dialogue.
| IshKebab wrote:
| What's a better option? One that keeps track of history and
| has a nice review interface?
| edolstra wrote:
| Indeed. Nixpkgs wouldn't have been as successful if it hadn't
| been using Git (or GitHub).
|
| Sure, eventually you run into scaling issues, but that's a
| first world problem.
| ori_b wrote:
| Alternatively: Downloading the entire state of all packages when
| you care about just one, it never works out.
|
| O(1) beats O(n) as n gets large.
| gruez wrote:
| Seems to still work out for apt?
| ajb wrote:
| Not in the same sense. An analogy might be: apt is like
| fetching a git repo in which all the packages are submodules,
| so lazily fetched. Some of the package managers in the
| article seem to be using a monorepo for all packages -
| including the content. Others seem to have different issues -
| go wasn't including enough information in the top level, so
| all the submodules had to be fetched anyway. vcpkg was doing
| something with tree hashes which meant they weren't really
| addressible.
| born-jre wrote:
| lol I see this as I plan on using Git for my thing store.
| https://github.com/blue-monads/potatoverse
| gjvc wrote:
| sqlite seems to be ideal for a package manager
| sigwinch wrote:
| I feel like the rqlite people would have a lot to say about how
| to coordinate your installations, especially for the high-
| bandwidth non-desktop installs.
|
| https://news.ycombinator.com/item?id=45257349
| mirekrusin wrote:
| ...or scm [0]
|
| [0] https://fossil-scm.org
| hk1337 wrote:
| I like Go but it's dependency management is weird and seems to be
| centered around GitHub a lot.
| Hendrikto wrote:
| There is nothing tying Go to GitHub.
| rewgs wrote:
| Not at all. It can grab git repos (as well as work with other
| VCSs). There's just a lot of stuff on GitHub, hence your
| impression.
| andreashaerter wrote:
| It's mostly tradition rather than a hard requirement. Go has
| long supported vanity import paths:
| https://pkg.go.dev/cmd/go#hdr-Remote_import_paths
|
| For example, we use Hugo to provide independent Go package URLs
| even though the code is hosted on GitHub. That makes migrating
| away from GitHub trivial if we ever choose to do so (Repo:
| https://github.com/foundata/hugo-theme-govanity; Example:
| https://golang.foundata.com/hugo-theme-dev/). Usage works as
| expected: go get golang.foundata.com/hugo-
| theme-dev
|
| Edit: Formatting
| hogrug wrote:
| The facts are interesting but the conclusion a bit strange. These
| package managers have succeeded because git is better for the low
| trust model and GitHub has been hosting infra for free that no
| one in their right mind would provide for the average DB.
|
| If it didn't work we would not have these massive ecosystems
| upsetting GitHub's freemium model, but anything at scale is
| naturally going to have consequences and features that aren't so
| compatible with the use case.
| ifh-hn wrote:
| So what's the answer then? That's the question I wanted answered
| after reading this article. With no experience with git or
| package management, would using a local client sqlite database
| and something similar on the server do?
| encom wrote:
| I quite like Gentoo's rsync based package manager. I believe
| they've used that since the beginning. It works well.
| MarsIronPI wrote:
| To be clear though, the rsync trees come from a central Git
| repo (though it's not hosted on GitHub). And syncing from Git
| actually makes syncing faster.
| AaronFriel wrote:
| OCI artifacts, using the same protocol as container registries.
| It's a protocol designed for versioning (tagging) content
| addressable blobs, associating metadata with them, and it's CDN
| friendly.
|
| Homebrew uses OCI as its backend now, and I think every package
| manager should. It has the right primitives you expect from a
| registry to scale.
| aniou wrote:
| As side note. Maybe someone knows, why rust devs chose an already
| used name for language changes proposal? "RFC" was already taken
| and well-established and I simply refuse to accept that someone
| wasn't aware about Request For Comments - and if it was true and
| clash was created deliberately, then it was rude and arrogant.
|
| Every, ...king time, when I read something like "RFC 2789
| introduced a sparse HTTP protocol." my brain suffers from a
| short-circuit. BTW: RFC 2789 is a "Mail Monitoring MIB".
| adastra22 wrote:
| There are many, many RFC collections. Including many that
| predate the IETF. Some even predate computers.
| aniou wrote:
| But they were in different domains. Here, we have a strong
| clash because Rust is positioning itself as secure system and
| internet language and computer and internet standard are
| already defined by RFC-s. So, it may be not uncommon, when
| someone would tell about Rust mechanisms, defined by
| particular RFC in context of handling particular protocol,
| defined by... well... RFC too. But not by rust-one.
|
| Not so smart, when we realize, that one of aspects of secure
| and reliable system is elimination of ambiguities.
| frumplestlatz wrote:
| Since ~2002, Macports has used svn or git, but _users_ , by
| default, rsync the complete port definitions + a server-generated
| index + a signature.
|
| The index is used for all lookups; it can also be generated or
| incrementally updated client-side to accommodate local changes.
|
| This has worked fine for literally decades, starting back when
| bandwidth and CPU power was far more limited.
|
| The problem isn't using SCM, and the solutions have been known
| for a very long time.
| gethly wrote:
| If we stopped using VCS to fetch source files, we would lose the
| ability to get the exact commit(understand as version that has
| nothing to do with the underlying VCS) of these files. Git,
| Mercurial, SVN.., github, bitbucket...it does not matter.
| Absolutely nobody will be building downloadable versions of their
| source files, hosted on who knows how "prestigious" domains, by
| copying them to another location just to serve the --->exact same
| content<--- that github and alike already provide.
|
| This entire blog is just a waste of time for anyone reading it.
| throwway120385 wrote:
| Or you could just ship a tarball and an sha checksum.
| gethly wrote:
| you could, in case you want to make only certain releases
| publicly available. but then, who wants to do that manual
| labour? we're talking mainstream here, not specific use
| cases.
| layer8 wrote:
| And yet, that's pretty much how the Java world works (Maven
| repositories).
| forrestthewoods wrote:
| > This entire blog is just a waste of time for anyone reading
| it.
|
| Well that's an extremely rude thing to say.
|
| Personally I thought it was really interesting to read about a
| bunch of different projects all running into the same wall with
| Git.
|
| I also didn't realize that Git had issues with sparse
| checkouts. Or maybe author meant shallow? I forget.
| encom wrote:
| >[Homebrew] Auto-updates now run every 24 hours instead of every
| 5 minutes[...]
|
| That is such an insane default, I'm at a loss for words.
| croemer wrote:
| You mean the 5 minutes is insane, right?
| dboon wrote:
| I'm building Cargo/UV for C. Good article. I thought about this
| problem very deeply.
|
| Unfortunately, when you're starting out, the idea of running a
| registry is a really tough sell. Now, on top of the very hard
| engineering problem of writing the code and making a world class
| tool, plus the social one of getting it adopted, I need to worry
| about funding and maintaining something that serves potentially a
| world of traffic? The git solution is intoxicating through this
| lense.
|
| Fundamentally, the issue is the sparse checkouts mentioned by the
| author. You'd really like to use git to version package
| manifests, so that anyone with any package version can get the
| EXACT package they built with.
|
| But this doesn't work, because you need arbitrary commits. You
| either need a full checkout, or you need to somehow track the
| commit a package version is in without knowing what hash git will
| generate before you do it. You have to push the package update
| and then push a second commit recording that. Obviously
| infeasible, obviously a nightmare.
|
| Conan's solution is I think just about the only way. It trades
| the perfect reproduction for conditional logic in the manifest.
| Instead of 3.12 pointing to a commit, every 3.x points to the
| same manifest, and there's just a little logic to set that
| specific config field added in 3.12. If the logic gets too much,
| they let you map version ranges to manifests for a package. So if
| 3.13 rewrites the entire manifest, just remap it.
|
| I have not found another package manager that uses git as a
| backend that isn't a terrible and slow tool. Conan may not be as
| rigorous as Nix because of this decision but it is quite
| pragmatic and useful. The real solution is to use a database, of
| course, but unless someone wants to wire me ten thousand dollars
| plus server costs in perpetuity, what's a guy supposed to do?
| adrianN wrote:
| Before you managed to build a popular tool it is unlikely that
| you need to serve many users. Directly going for something that
| can serve the world is probably premature
| dboon wrote:
| For most software, yes. But the value of a package manager is
| in its adoption. A package manager that doesn't run up
| against these problems is probably a failure anyway.
| EPWN3D wrote:
| The point is not "design to serve the world". The point is
| "use the right technology for your problem space".
| ambicapter wrote:
| > Unfortunately, when you're starting out, the idea of running
| a registry is a really tough sell. Now, on top of the very hard
| engineering problem of writing the code and making a world
| class tool, plus the social one of getting it adopted, I need
| to worry about funding and maintaining something that serves
| potentially a world of traffic? The git solution is
| intoxicating through this lense.
|
| So you need a decentralized database? Those exist (or you can
| make your own, if you're feeling ambitious), probably ones that
| scale in different ways than git does.
| dboon wrote:
| Please share. I'm interested in anything that's roughly as
| simple as implementing a centralized registry, is easily
| inspected by users (preferably with no external tooling), and
| is very fast.
|
| It's really important that someone is able to search for the
| manifest one of their dependencies uses for when stuff
| doesn't work out of the box. That should be as simple as
| possible.
|
| I'm all ears, though! Would love to find something as simple
| and good as a git registry but decentralized
| strbean wrote:
| Distributed ledger! /s... ?
| jopsen wrote:
| You don't need fully distributed database, do you?
|
| You could just make a registry hosted as plain HTTP, with
| everything signed. And a special file that contains a list
| of mirrors.
|
| Clients request the mirror list and the signed hash of the
| last entry in the Merkel tree. Then they go talk to a
| random mirror.
|
| Maybe, you central service requires user sign-in for
| publishing and reading, while mirrors can't publish, but
| mirrors don't require sign-in.
|
| Obviously, you'd have to validate that mirrors are up and
| populated. But that's it.
|
| You can start by self hosting a mirror.
|
| One could go with signing schemes inspired by:
| https://theupdateframework.io/
|
| Or one could omit signing all together, so long as you have
| a Merkel tree with hashes for all publishing events. And
| the latest hash entry is always fetched from your server
| along with the mirror list.
|
| Having all publishing go through a single service is
| probably desirable. You'll eventually need to do
| moderation, etc. And hosting your service or a mirror
| becomes a legal nightmare if there is not moderation.
|
| Disclaimer: opinions are my own.
| k8ssskhltl wrote:
| Blockchain.
| krautsauer wrote:
| I wonder how meson wraps' story fits with this. They used not
| to, but now they're throwing everything into a single
| repository [0]. I wonder about the motivation and how it
| compares to your project.
|
| 0: https://github.com/mesonbuild/wrapdb/tree/master/subprojects
| mook wrote:
| Is there a reason the users must see all of the historic data
| too? Why not just have a post-commit hook render the current
| HEAD to static files, into something like GitHub Pages?
|
| That can be moved elsewhere / mirrored later if needed, of
| course. And the underlying data is still in git, just not
| actively used for the API calls.
|
| It might also be interesting to look at what Linux distros do,
| like Debian (salsa), Fedora (Pagure), and openSUSE (OBS).
| They're good for this because their historic model is free
| mirrors hosted by unpaid people, so they don't have the compute
| resources.
| jarofgreen wrote:
| I'm not OP but I'll guess .... lock files with old versions
| of libs in. The latest version of a library may be v2 but if
| most users are locked to v1.267.34 you need all the old
| versions too.
|
| However a lot of the "data in git repositories" projects I
| see don't have any such need, and then ...
|
| > Why not just have a post-commit hook render the current
| HEAD to static files, into something like GitHub Pages?
|
| ... is a good plan. Usually they make a nice static website
| with the data that's easy for humans to read though.
| dkarl wrote:
| Think about the article from a different perspective: several
| of the most successful and widely used package managers of all
| time started out using Git, and they successfully transitioned
| to a more efficient solution when they needed to.
| zephen wrote:
| Not only this, but (if I understand the article correctly) at
| least some of them still use git on the backend.
| baobun wrote:
| How about the Arch Linux AUR approach?
|
| Every package has its own git repository which for binary
| packages contains mostly only the manifest. Sources and assets,
| if in git, are usually in separate repos.
|
| This seems to not have the issues in the examples given so far,
| which come from using "monorepos" or colocating. It also avoids
| the "nightmare" you mention since any references would be in
| separate repos.
|
| The problematic examples either have their assets and manifests
| colocated, or use a monorepo approach (colocating manifests and
| the global index).
| jopsen wrote:
| The alluring thing is storing the repository on S3 (or
| similar). Recall early docker registries making requests so
| complicated that backing image storage with S3 was unfeasible,
| without a proxy service.
|
| The thing that scales is dumb HTTP that can be backed by
| something like S3.
|
| You don't have to use a cloud, just go with a big single
| server. And if you become popular, find a sponsor and move to
| cloud.
|
| If money and sponsor independence is a huge concern the
| alternative would be: peer-to-peer.
|
| I haven't seen many package managers do it, but it feels like a
| huge missed opportunity. You don't need that many volunteers to
| peer inorder to have a lot of bandwidth available.
|
| Granted, the real problem that'll drive up hosting cost is CI.
| Or rather careless CI without caching. Unless you require a
| user login, or limit downloads for IPs without a login, caching
| is hard to enforce.
|
| For popular package repositories you'll likely see extremely
| degenerate CI systems eating bandwidth as if it was free.
|
| Disclaimer: opinions are my own.
| kibwen wrote:
| I think there's a form of survivorship bias at work here. To use
| the example of Cargo, if Rust had never caught on, and thereby
| gotten popular enough to inflate the git-based index beyond
| reason, then it would never have been a problem to use git as the
| backing protocol for the index. Likewise, we can imagine
| innumerable smaller projects that successfully use git as a
| distributed delta-updating data distribution protocol, and never
| happen to outgrow it.
|
| The point being, if you're not sure whether your project will
| ever need to scale, then it may not make sense to reinvent the
| wheel when git is right there (and then invent the solution for
| hosting that git repo, when Github is right there), letting you
| spend time instead on other, more immediate problems.
| stickfigure wrote:
| Right, this post may encourage premature optimization. Cargo,
| Homebrew, et al chose an easy, good-enough solution which
| allowed them to grow until they hit scaling limits. This is a
| good problem to have.
|
| I am sure there's value having a vision for what your scaling
| path might be in the future, so this discussion is a good one.
| But it doesn't automatically mean that git is a bad place to
| start.
| nacozarina wrote:
| successful things often have humble origins, it's a feature not a
| bug
|
| for every project that managed to out-grow ext4/git there were a
| hundred that were well-served and never needed to over-invest in
| something else
| PunchyHamster wrote:
| The article conclusion is just... not good. There are many
| benefits to using Git as backend, you can point your project to
| every single commit as a version which makes testing any fixes or
| changes in libs super easy, it has built in integrity control and
| _technically_ (sadly not in practice) you could just sign commits
| and use that to verify whether package is authentic.
|
| It being unoptimal bandwidth wise is frankly just a technical
| hurdle to get over it, with benefits well worth the drawback
| 0xbadcafebee wrote:
| YOLO software engineering, the hallmark of the 21st century
| cesarb wrote:
| One of these is not like the others...
|
| > The problem was that go get needed to fetch each dependency's
| source code just to read its go.mod file and resolve transitive
| dependencies.
|
| This article is mixing two separate issues. One is using git as
| the master database storing the index of packages and their
| versions. The other is fetching the code of each package through
| git. They are orthogonal; you can have a package index using git
| but the packages being zip/tar/etc archives, you can have a
| package index not using git but each package is cloned from a git
| repository, you can have both the index and the packages being
| git repositories, you can have neither using git, you can even
| not have a package index at all (AFAIK that's the case for Go).
| bobpaw wrote:
| I think the article takes issue not with fetching the code, but
| with fetching the go.mod file that contains index and
| dependency information. That's why part of the solution was to
| host go.mod files separately.
| jayd16 wrote:
| Even with git, it should be possible to grab the single file
| needed without the rest of the repo, but i'ts still trying to
| round a square peg.
| skywhopper wrote:
| Honestly I think the article is a bit ahistorical on this
| one. 'go get' pulls the source code into a local cache so it
| can build it, not just to fetch the go.mod file. If they were
| having slow CI builds because they didn't or couldn't
| maintain a filesystem cache, that's annoying, but not really
| a fault in the design. Anyway, Go improved the design and
| added an easy way to do faster, local proxies. Not sure what
| the critique is here. The Go community hit a pain point and
| the Go team created an elegant solution for it.
| sghiassy wrote:
| Use the git clone --shallow option and you'll only download the
| most recent commits. Yeesh
| dleslie wrote:
| GitHub is intoxicatingly free hosting, but Git itself is a
| terrible database. Why not maintain an _actual_ database on
| GitHub, with tagged releases?
|
| Sqlite data is paged and so you can get away with only fetching
| the pages you need to resolve your query.
|
| https://phiresky.github.io/blog/2021/hosting-sqlite-database...
| jarofgreen wrote:
| This seems to be about hosting an Sqlite database on a static
| website like GitHub Pages - this can be a great plan, there is
| also Datasette in a browser now:
| https://github.com/simonw/datasette-lite
|
| But that's different from how you collect the data in a git
| repository in the first place - or are you suggesting just
| putting a Sqlite file in a git repository? If so I can think of
| one big reason against that.
| dleslie wrote:
| Yes, I'm suggesting hosting it on GitHub, leveraging their
| git lfs support. Just treat it like a binary blob and
| periodically update with a tagged release.
| jarofgreen wrote:
| It's not clear if you are suggesting accepting
| contributions to the SQLite file via PR from people (but
| accepting contributions is generally the point of why
| people put these on projects on GitHub).
|
| But if you are I wouldn't recommend it.
|
| PR's won't be able to show diff's. Worse, as soon as
| multiple people send a PR at once you'll have a really
| painful merge to resolve, and GitHub's tools won't help you
| at all. And you can't edit the files in GitHub's web UI.
|
| I recommend one file per record, JSON, YAML, whatever non-
| binary format you want. But then you get:
|
| * PR's with diff's that show you what's being changed
|
| * Files that technical people can edit directly in GitHub's
| web editor
|
| * If 2 people make PR's on different records at once it's
| an easy merge with no conflicts
|
| * If 2 people make PR's on the same record at once ... ok,
| you might now have a merge conflict to resolve but it's in
| an easy text file and GitHub UI will let you see what it
| is.
|
| You can of course then compile these data files into a
| SQLite file that can be served in a static website nicely -
| in fact if you see my other comments on this post I have a
| tool that does this. And on that note, sorry, I've done a
| few projects in this space so I have views :-)
| xpressvideoz wrote:
| The article lists Git-based wiki engines as a bad usage of Git.
| Can anybody recommend alternatives? I want something that can be
| self-hosted, is easily modified by text editors, and has
| individual page history, preferably with Markdown.
| cbondurant wrote:
| Admittedly, I try and stay away from database design whenever
| possible at work. (Everything database is legacy for us) But the
| way the term is being used here kinda makes me wonder, do modern
| sql databases have enough security features and permissions
| management systems in place that you could just directly expose
| your database to the world with a "guest" user that can only make
| incredibly specific queries?
|
| Cut out the middle man, directly serve the query response to the
| package manager client.
|
| (I do immediately see issues stemming from the fact that you cant
| leverage features like edge caching this way, but I'm not really
| asking if its a good solution, im more asking if its possible at
| all)
| brendoncarroll wrote:
| I personally think that this is the future, especially since
| such an architecture allows for E2E encryption of the entire
| database. The protocol should just be a transaction layer for
| coordinating changes of opaque blobs.
|
| All of the complexity lives on the client. That makes a lot of
| sense for a package manager because it's something lots of
| people want to run, but no one really wants to host.
| bob1029 wrote:
| There are still no realistic ways to expose a hosted SQL
| solution to the public without really unhappy things occurring.
| It doesn't matter which vendor you pick.
|
| Anything where you are opening a TCP connection to a hosted SQL
| server is a non-starter. You could hypothetically have so many
| read replicas that no one could blow anyone else up, but this
| would get to be very expensive at scale.
|
| Something involving SQLite is probably the most viable option.
| IshKebab wrote:
| Feels like there's an opening in the market there. Why can't
| you expose an SQL server to the public?
|
| Also Stackoverflow exposes a SQL interface so it isn't
| totally impossible.
| mirekrusin wrote:
| You can use fossil [0]
|
| [0] https://fossil-scm.org
| zX41ZdbW wrote:
| ClickHouse can do it. Examples:
| https://play.clickhouse.com/ clickhouse-client
| --host play.clickhouse.com --user play --secure
| ssh play.clickhouse.com
| baobun wrote:
| Yes but CH is not SQL.
| dromologist wrote:
| We wanted to pull updated code in our undockerized instances when
| they were instantiated, so we decided to pull the code from
| GitHub. Worked out pretty well though after a thousand trials we
| got a 502 and now we're one step closer to being forced into a CD
| pipeline.
| keithgroves wrote:
| When building https:/enact.tools we considered this. I'm glad we
| didn't go this route.
| jarofgreen wrote:
| It's not just package manager who do this - a lot of smaller
| projects crowd source data in git repositories. Most of these
| don't reach the scale where the technical limitations become a
| problem.
|
| Personally my view is that the main problem when they do this is
| that it gets much harder for non-technical people to contribute.
| At least that doesn't apply to package managers, where it's all
| technical people contributing.
|
| There are a few other small problems - but it's interesting to
| see that so many other projects do this.
|
| I ended up working on an open source software library to help in
| these cases: https://www.datatig.com/
|
| Here's a write up of an introduction talk about it:
| https://www.datatig.com/2024/12/24/talk.html I'll add the scale
| point to future versions of this talk with a link to this post.
| pizlonator wrote:
| What is the alternative?
|
| "Use a database" isn't actionable advice because it's not
| specific enough
| holyknight wrote:
| It's basically the same thing that always happens when you choose
| a technology because it's convenient rather than a great fit for
| your problem. Sooner or later, you'll hit a wall. Just because
| you can cook a salmon in your dishwasher doesn't mean you should.
| BlueTemplar wrote:
| Wait, isn't fossil based on sqlite ?
|
| Or does fossil itself still have the same issues ?
| dwardu wrote:
| Worst thing is when you're in a an office and your pc along with
| other pcs pulls from git unauthenticated, then you get hit with
| api limits
| Ericson2314 wrote:
| The Nixpkgs example is not like the others, because it _is_
| source code.
|
| I don't get what is so bad about shallow clones either. Why
| should they be so performance sensative?
| ajb wrote:
| In a compressed format, later commits would be added as a delta
| of some kind, to avoid increasing the size by the whole tree
| size each time. To make shallow clones efficient you'd need to
| rewrite the compressed form such that earlier commits are
| instead deltas on later ones, or something equivalent.
| __MatrixMan__ wrote:
| It also seems like it's not git that's emitting scary creaks
| and groans, but rather GitHub. As much as it would be a bummer
| to forgo some of GitHub's nice-to-have features, I expect we
| could survive without some of it.
| mindslight wrote:
| Furthermore, the issues given for nixpkgs are actually
| demonstrating _the success_ of using git as the database!
| Those 20k forks are all people maintaining their own version
| of nixpkgs on Github, right? Each their own independent tree
| that users can just go ahead and modify for their own whims
| and purposes, without having to overcome the activation
| energy of creating their own package repository.
|
| If 83GB (4MB/fork) is "too big" then responsibility for that
| rests solely on the elective centralization encouraged by
| Github. I suspect if you could go and total up the cumulative
| storage used by the nixpkgs source tree _distributed on
| computers spread throughout the world_ , that is many orders
| of magnitude larger.
| MarsIronPI wrote:
| Exactly. Gentoo's main package repo is hosted in Git (but not
| GitHub, except as a mirror). Now, most users fetch it via
| rsync, but actually using the Git repo IME makes syncing
| _faster_ , not slower. Though it does make the initial fetch
| slower.
| teiferer wrote:
| And this my friends is the reason why (only) focusing on CPU
| cycles and memory hierarchies is insufficient when thinking of
| the performance of a system. Yes they are important. But no level
| of low-level optimization will get you out of the hole that a
| wrong choice of algorithm and/or data structure may have dug you
| into.
| iamwil wrote:
| This sounds like a missing piece of software in the OSS world. If
| you have the inclination, you should write it.
| weiwenhao wrote:
| For package management software that is rarely used, free is the
| biggest motivation.
| mukundesh wrote:
| Though not Github, worth mentioning Huggingface, which is also
| using git, but managing large files with their(?) xet protocol.
| https://huggingface.co/docs/hub/en/xet/index
| drzaiusx11 wrote:
| One of the first things I did at my current place of employment
| was to detangle the mess of gemfile git dependencies and get them
| to adopt semver and an actual package repo. There were so many
| footguns with git dependencies in ruby we were getting taken down
| by friendly fire on the daily...
| drzaiusx11 wrote:
| I'd add git gemfile dependencies to the list of languages called
| out here as well. It supports git repos, but in general it's a
| bad idea unless you are diligent with git tag use and disallow
| git tag mutability, which also assumes you have complete control
| of your git dependencies...
| newswangerd wrote:
| It's always humbling when you go on the front page of HN and see
| an article titled "the thing you're doing right now is a bad idea
| and here's why"
|
| This has happened to me a few times now. The last one was a
| fantastic article about how PG Notify locks the whole database.
|
| In this particular case it just doesn't make a ton of sense to
| change course. Im a solo dev building a thing that may never take
| off, so using git for plug-in distribution is just a no brainer
| right now. That said, I'll hold on to this article in case I'm
| lucky enough to be in a position where scale becomes an issue for
| me.
| baobun wrote:
| The good news is you can easier avoid some of the pitfalls now
| even as you stick with it. Some good points in comments.
|
| I don't know if you rely on github.com but IMO vendor lock-in
| there might be a bigger issue which you can avoid.
| ekjhgkejhgk wrote:
| Uncertain if this is OT, but given that the CCC is politically
| inspired organization, I hope not:
|
| One thing that still seems absent is awareness of the complete
| takeover of "gadgets" in schools. Schools these days, as early as
| primary school, shove screens in front of children. They're
| expected to look at them, and "use" them for various activities,
| including practicing handwriting. I wish I was joking [1].
|
| I see two problems with this.
|
| First is that these devices are engineered to be addictive by way
| of constant notifications/distractions, and learning is something
| that requires long sustained focus. There's a lot of data showing
| that under certain common circumstances, you do _worse_ learning
| from a screen than from paper.
|
| Second is implicitly it trains children to expect that anything
| has to be done through a screen connected to a closed point-and-
| click platform. (Uninformed) people will say "people who work
| with computers make money, so I want my child to have an ipad".
| But interacting with a closed platform like an ipad is removing
| the possibilities and putting the interaction "on rails". You
| don't learn to think, explore and learn from mistakes, instead
| you learn to use the app that's put in front of you. This in turn
| reinforces the "computer says no" [2] approach to understanding
| the world.
|
| I think this is a matter of civil rights and freedom, but sadly I
| don't often see "civil rights" organizations talk about this. I
| _think_ I heard Stallman say something along these lines once,
| but other than that I don 't see campaigns anywhere.
|
| [1] https://www.letterjoin.co.uk/
|
| [2] https://youtu.be/eE9vO-DTNZc
| AceJohnny2 wrote:
| It looks like you commented on the wrong post, although I don't
| immediately see a front-page post about the ongoing Chaos
| Computer Congress.
| kzrdude wrote:
| it's here https://news.ycombinator.com/item?id=46386211 (and
| it was last on the front page at the moment)
| ekjhgkejhgk wrote:
| ty
| ekjhgkejhgk wrote:
| LOL sorry. You're right. I'll copy paste over there.
| grumbel wrote:
| Do we have distributed databases that regular users can clone,
| modify and merge?
| stephenlf wrote:
| Omarchy
| jama211 wrote:
| "It never works out" - hmm, seems like it worked out just fine,
| worked great to get the operation of the ground and when scale
| became an issue it was solvable by moving to something else. It
| served its purpose, sounds like it worked out to me.
| swiftcoder wrote:
| You appear to have glossed over the two projects in the list
| that are stuck due to architectural decisions, and don't have
| any route to migrate off of git-as-database?
| hombre_fatal wrote:
| Be more specific because I just see a list of workarounds
| deployed once they had the scale to warrant them, supporting
| the OP's claim.
| swiftcoder wrote:
| Read the vcpkg section, it explicitly states that they have
| no horizontal on a solution. The nix section also doesn't
| explain any potential solution.
| baobun wrote:
| The issues with nixpkgs stem from that it is a monorepo for
| all packages and doubling as an index.
|
| The issues are only fundamental with that architecture. Using
| a separate repo for each package, like the Arch User Repos,
| does not have the same problems.
|
| Nixpkgs certainly could be architected like that and
| submodules would be a graceful migration path. I'm not aware
| of discussion of this but guess that what's preventing it
| might be that github.com tooling makes it very painful to
| manage thousands of repos for a single project.
|
| So I think it can be a lesson not to that using git as a
| database is bad but that using github.com as a database is.
| PRs as database transactions is clunky and GitHub Actions
| isn't really ACID.
| lijok wrote:
| Nooo you don't get it - it didn't scale from 0 to a trillion
| users so it's a garbage worthless system that "doesn't scale".
| zephen wrote:
| ^^^ Poe's Law may or may not apply to the above comment.
| efitz wrote:
| When you start out with a store like git, with file system
| semantics and a client that has to be smart to handle all the
| compare and merge operations, then it's practically impossible
| to migrate a large client base to a new protocol. Takes years
| lots of user complaints to and random breakage.
|
| Much better to start with an API. Then you can have the server
| abstract the store and the operations - use git or whatever -
| but you can change the store later without disrupting your
| clients.
| leoh wrote:
| I couldn't agree more strongly. There is a huge opportunity to
| make git more effective for this kind of use-case, not to
| abandon it. The essay in question provides no compelling
| alternative; it therefore reaches an entirely half-baked
| conclusion.
| mikepurvis wrote:
| The nix cli almost exclusively pulls GitHub as zipballs. Not
| perfect but certainly far faster than a real git clone.
| pxc wrote:
| That it supports fetching via Git as well as various via forge-
| specific tarballs, even for flakes, is pretty nice. It means
| that if your org uses Nix, you can fall back to distribution
| via Git as a solution that doesn't require you to stand up any
| new infra or tie you to any particular vendor, but once you get
| rolling it's an easy optimization to switch to downloading
| snapshots.
|
| The most pain probably just becomes from the hugeness of
| Nixpkgs, but I remain an advocate for the huge monorepo of
| build recipes.
| mikepurvis wrote:
| Yes agreed. It's possible to imagine some kind of cached-
| deltas scheme to get faster/smaller updates, but I suspect
| the folks who would have to build and maintain that are all
| on gigabit internet connections and don't feel the complexity
| is worth it.
| pxc wrote:
| > It's possible to imagine some kind of cached-deltas
| scheme to get faster/smaller updates
|
| I think the snix1 folks are working on something like this
| for the _binary caches_ -- the greater granularity of the
| content-addressing offers morally the same kind of
| optimization as delta RPMs: you can download less of what
| you don't need to re-download.
|
| But I'm not aware of any current efforts to let people
| download the Nixpkgs tree itself more efficiently. Somehow
| caching Git deltas would be cool. But I'd expect that kind
| of optimization to come from a company that runs a Git
| forge, if it's generally viable, and to benefit many
| projects other than Nix and Nixpkgs.
|
| --
|
| 1: https://snix.dev/
| didip wrote:
| So... What we need is a globally distributed git seeders of all
| open source github content, then?
|
| Seems possible if every git client is also a torrent client.
| the__alchemist wrote:
| The Cargo example at the top is striking. Whenever I publish a
| crate, and it blocks me until I write `--allow-dirty`, I am
| reminded that there is a conflation between Cargo/crates.io and
| Git that should not exist. I will write `--allow-dirty` because I
| think these are two separate functionalities that should not be
| coupled. Crates.io should not know about or care about my
| project's Git usage or lack thereof.
| aidenn0 wrote:
| As far as I know, Nixpkgs doesn't use git as a package database.
| The packages definitions are stored and developed in git, but the
| channels certainly are not.
| mcny wrote:
| I want to take a quick detour here if anyone is knowledgeable
| about this topic.
|
| > The hosting problems are symptoms. The underlying issue is that
| git inherits filesystem limitations, and filesystems make
| terrible databases.
|
| Does this mean mbox is inherently superior to maildir? I really
| like the idea of maildir because there is nothing to compact but
| if we assume we never delete emails (on the local machine
| anyways), does that mean mbox or similar is preferable over
| maildir?
| juped wrote:
| No, of course not.
| notorandit wrote:
| Repsy
| pxc wrote:
| Loved this article. Just enough detail to make the broad scope
| compatible with a reasonable length, and well-argued.
|
| I feel sometimes like package management is a relatively second-
| class topic in computer science (or at least among many working
| programmers). But a package manager's behavior can be the
| difference between a grotesque, repulsive experience and a
| delightful, beautiful one. And there aren't quite yet any package
| managers that do well everything that we collectively have
| learned how to do well, which makes it an interesting space imo.
|
| Re: Nixpkgs, interestingly, pre-flakes Nix distributes all of the
| needed Nix expressions as tarballs, which does play nice with
| CDNs. It also distributes an index of the tree as a SQLite
| database to obviate some of the "too many files/directories"
| problem with enumerating files. (In the meantime, Nixpkgs has
| also started bucketing package directories by name prefix, too.)
| So maybe there was a lesson learned here that would be useful to
| re-learn.
|
| On the other hand, IIRC if you use the GitHub fetcher rather than
| the Git one, including for fetching flakes, Nix will download
| tarballs from GitHub instead of doing clones. Regardless,
| downloading and unpacking Nixpkgs has become kinda slow. :-\
| themk wrote:
| I think git is overkill, and probably a database is as well.
|
| I quite like the hackage index, which is an append-only tar file.
| Incremental updates are trivial using HTTP range requests making
| hosting it trivial as well.
| leoh wrote:
| The conclusion reached in this essay is 100% wrong. See " The
| reftable backend What it is, where it's headed, and why should
| you care?"
|
| >With release 2.45, Git has gained support for the "reftable"
| backend to read and write references in a Git repository. While
| this was a significant milestone for Git, it wasn't the end of
| GitLab's journey to improve scalability in repositories with many
| references. In this talk you will learn what the reftable backend
| is, what work we did to improve it even further and why you
| should care.
|
| https://www.youtube.com/watch?v=0UkonBcLeAo
|
| Also see Scalar, which Microsoft used to scale their 300GiB
| Windows repository, https://github.com/microsoft/scalar.
| skywhopper wrote:
| Not sure I can agree with the takeaway. It works well at first,
| but doesn't scale, so folks found workarounds. That's how
| literally every working system grows. There are always
| bottlenecks eventually. And you address them when they become an
| issue, not five years earlier.
| juped wrote:
| These are actually all problems with using Github as an ersatz
| CDN.
| bandrami wrote:
| Maybe I'm misreading the article but isn't every example about
| the downside of using _github_ as a database host, not the
| downside of using git as a database?
|
| Like, yes, you should host your own database. This doesn't seem
| like an argument against that database being git.
| khc wrote:
| seems like the issue isn't with using git as a database, but
| using github as a distribution mechanism?
| zzo38computer wrote:
| Git commits will have a hash and each file will have a hash,
| which means that locking is unnecessary for read access. (This is
| also true of fossil, although fossil does have locking since it
| uses SQLite.)
|
| The other stuff mentioned in the article seems to be valid
| criticisms.
___________________________________________________________________
(page generated 2025-12-26 23:00 UTC)