Post AdRLwHeBI9bm7cLbw8 by mnl@hachyderm.io
(DIR) More posts by mnl@hachyderm.io
(DIR) Post #AdPl7GUK3Be2Fb9KCW by freakazoid@retro.social
2024-01-01T16:02:59Z
0 likes, 0 repeats
That anyone thinks LLMs can be useful for generating code isn't an indication of how good LLMs are but how bad programmers are. We've turned them into typists with our simplistic languages, bureaucratic development processes, and CS programs aimed at getting them ready to be typists.
(DIR) Post #AdPl7HSESRZDFPBAa8 by mnl@hachyderm.io
2024-01-01T17:08:09Z
0 likes, 0 repeats
@freakazoid what would be an alternative educational approach?I find most value in LLMs in writing tests / fuzzers / logging / instrumentatjon, rich tooling, informational error messages / Help strings / readmes. Indeed rote stuff, but extremely valuable.I think it is doing programmers and our field a disservice to sneer at this kind of programming.
(DIR) Post #AdPl7IMF6CN037Ntsu by freakazoid@retro.social
2024-01-01T17:13:20Z
0 likes, 0 repeats
@mnl There isn't one. Capitalism will always ruin programming.
(DIR) Post #AdPl7J86EGeQRXm7Tk by freakazoid@retro.social
2024-01-01T17:15:33Z
0 likes, 0 repeats
@mnl "Our field" isn't one field; it's two: one being the software tradespeople, who are as I said "typists", or construction workers. Most university "Computer Science" programs are actually overpriced trade schools. The other is actual computer science, which also contains actual software engineering, which is nearly absent from industry.
(DIR) Post #AdPl7K1Ouet3D3eHg0 by freakazoid@retro.social
2024-01-01T17:30:41Z
0 likes, 0 repeats
@mnl I guess we could improve the situation with actual trade schools, but I doubt most of industry would go for it since they have deluded themselves into thinking universities actually teach in accordance with their cost. With that we would need companies to realize that you can't build a 10 story building by hiring 100 construction workers. And that "scrum master" is not a job.
(DIR) Post #AdPl7Kmu42staNsDia by freakazoid@retro.social
2024-01-01T17:32:40Z
0 likes, 0 repeats
@mnl If I were trying to hire programmers, I'd probably filter out anyone with a 4 year CS degree. Too expensive for what you get. I don't really know how to find actual software engineers though.
(DIR) Post #AdPl7LVDPIKVnobbmq by mnl@hachyderm.io
2024-01-01T17:35:13Z
0 likes, 0 repeats
@freakazoid I'm going to strongly disagree with the fact that "software tradespeople" are just typists that don't software architecture. That is quite dismissive and doesn't reflect reality, or that scrum master (as a stand-in for person focused on process, or maybe software foreman) isn't a real job.
(DIR) Post #AdPl7MEwfGuS5e0848 by freakazoid@retro.social
2024-01-01T17:40:29Z
0 likes, 0 repeats
@mnl A lot of people strongly disagree with that idea, or we wouldn't be in the situation we're in today.Scrum master is by definition not a job, because Scrum says it's supposed to be one of the developers on the team. Therefore, if you have a dedicated scrum master job, you're not doing Scrum, thus they're not a Scrum master. I'm a CSM and therefore can say that with authority.
(DIR) Post #AdPl7NANDkqYxkrzZw by mnl@hachyderm.io
2024-01-01T17:47:35Z
0 likes, 0 repeats
@freakazoid what situation are we in today? was it better in the past? and how are LLMs related to any of this?IMO software quality has never been higher, thanks to a vibrant opensource ecosystem and millions of programmers of varying levels of competence. I don't understand why you can say that people without a CS background are just typists and people with a CS degree not worth their price. Who does that leave?
(DIR) Post #AdPl7NpUkrjx1I6pfs by freakazoid@retro.social
2024-01-01T17:42:41Z
0 likes, 0 repeats
@mnl In my experience, companies that say they're doing "Scrum" are really just trying to make up for poor execution by adding process. They don't do Scrum to serve the team; they do it to serve management. This is why that job exists in the first place: they're trying to make up for an absence of leadership by adding process. Which ironically tends to slow down progress, but it can make management think it's sped up the process by increasing how much visibility they have.
(DIR) Post #AdPl7O1u0jfHdluk0u by freakazoid@retro.social
2024-01-01T17:55:33Z
0 likes, 0 repeats
@mnl Well, I was being a bit hyperbolic with the 4 year CS degree. Someone could have a 4 year CS degree and show their ability to actually build software through their experience.The situation that we're in today is that we're buried under a giant pile of garbage. To the extent it's worse it's not really because of bad programming practice, just volume. The real damage to computing has been "enshittification" and the war on general purpose computing.I'm not sure the problem is solvable within the context of current platforms and software distribution. To the extent that open source benefits everyday people, it is because corporations use the open source community as free labor, but for the most part the corporations keep most of the surplus from that anyway. So it's just a small number of nerds like myself who get to benefit from it.The "wrong turn" has nothing to do with education IMO; education has just followed the industry. The "wrong turn" was when computers shifted from being devices users programmed to do what they want to being nothing but platforms for proprietary applications. The iPhone is the epitome of that wrong turn. People buy them because they've been taught that computers are scary and learning things is bad. And the developers want a slice of that huge pool of money so they allow themselves to be beholden to Apple.You know, I say it's not really "worse", but honestly the early days of the PC where there was actual competition and the software developers weren't dependent on the OS or computer makers for distribution were far better when you account for the much lower volume.
(DIR) Post #AdPl7PM94xS3kqOQCm by freakazoid@retro.social
2024-01-01T18:01:10Z
0 likes, 0 repeats
@mnl My mother in law was a programmer. At that time, programming was about 50% women. Many of them came from the typing pool, though my mother in law wasn't a typist (I don't remember what her actual job was).So we've gone from something that people used to learn on the job to something it takes them 4 years to learn and somehow some people just can't learn? I doubt it. Seems a lot more likely it's gatekeeping, much of it unintentional feedback loops.
(DIR) Post #AdPl7PQOp8r1y2Nops by freakazoid@retro.social
2024-01-01T17:47:47Z
0 likes, 0 repeats
@mnl I've been programming professionally since 1993. I've manged both SWEs and SREs. I've worked at every size company from startup to over 100k employees. I've worked on teams that have done actual scrum and teams that did fake top-down scrum. Aside from the occasional well-run waterfall, the only process I've seen actually work is one that the team comes up with themselves. All top-down process fails.
(DIR) Post #AdPl7QBXzqZIKGRTKC by mnl@hachyderm.io
2024-01-01T18:05:13Z
0 likes, 0 repeats
@freakazoid do you think software engineering is being gatekept? by whom? if anything, the amount of freely available resources hinder learning because they are so omnipresent. Again, something LLMs can already help tremendously with (streamlining curriculums, mixing theory with practice), and that's with some barebones RAG and chatbot interfaces.
(DIR) Post #AdPl7RFq1NavdrSPeS by freakazoid@retro.social
2024-01-01T18:09:43Z
0 likes, 0 repeats
@mnl As I said, a lot of it is unintentional feedback loops. Unwillingness to train, or to allow people to learn on the job, is a form of gatekeeping. It allows universities to charge a huge premium without actually teaching much of anything.Come to think of it, my dad also learned to program on the job. He went from repairing cryptographic machines in the air force to repairing typewriters and then printers for IBM. He moved on to the rest of the mainframe and started finding bugs for the programmers to fix. Eventually he got tired of waiting for fixes and started fixing the bugs himself. He retired from IBM at the highest non-management programmer level.
(DIR) Post #AdPl7S6eqzqUHgAayu by freakazoid@retro.social
2024-01-01T18:17:40Z
0 likes, 0 repeats
@mnl Frankly this boils down to poor leadership as well, now that I think of it. One phenomenon I've observed is companies hiring a large number of senior folks in an attempt to make up for a lack of leadership. Those folks usually end up leaving, because it's actually pretty hard to have impact at senior levels without good leadership, unless you manage to find a good niche. But you end up with the "too many cooks spoil the broth" problem.Poor leadership means companies have low confidence they can actually develop junior people.Hmm, which reminds me of the origin of Facebook's and Google's (I've managed at both companies) "up or out" philosophy for levels 3 and 4: they felt that people at those levels were only worth their salaries as potential 5s. Once it became clear they weren't going to make it to 5 in a timely manner, they'd get PIPed out.But why should someone at level 3 be too expensive? Shouldn't the market take care of that? No, because the only way you end up as level 3 is by having gone to an overpriced 4 year university, and people who do that expect high starting salaries.Google experimented with a "resident" program, but it still required a degree. They only had it because not enough people were coming out of universities able to pass the level 3 "bar" (scare quotes because it implies objectivity which isn't there).
(DIR) Post #AdPl7T11TQvr6UXbpw by freakazoid@retro.social
2024-01-01T18:18:43Z
0 likes, 0 repeats
@mnl So perhaps the problem isn't with programming at all but with tech leadership. Given that tech is run by psychopaths, I'm surprised it's taken me this long to figure it out.
(DIR) Post #AdPl7TrUKMtpjD5Vc8 by mnl@hachyderm.io
2024-01-01T18:19:22Z
0 likes, 0 repeats
@freakazoid what do you mean by "tech is run by psychopaths"? What does running tech mean? There's a whole world outside of google and facebook.
(DIR) Post #AdPl7Uk53OZISWd6hs by freakazoid@retro.social
2024-01-01T18:23:40Z
0 likes, 0 repeats
@mnl I've worked for 13 tech companies now. I've been in tech since I was illegal child labor in 1984.But the situation appears to have become much worse after the low interest rates post 2008 drove a high appetite for risk among investors, thus huge amounts of money going into VC. And the VC pattern is to build companies that survive just long enough to dump them on the public.
(DIR) Post #AdPl7VmbBWB1gcodGq by mnl@hachyderm.io
2024-01-01T18:24:25Z
0 likes, 0 repeats
@freakazoid but not everything in tech is VC backed, though?
(DIR) Post #AdPl7WWgQB2XzYNR6O by freakazoid@retro.social
2024-01-01T18:25:14Z
0 likes, 0 repeats
@mnl I'm not sure why you think that matters; that's where the money is, thus universities and the field of programming orient around that.
(DIR) Post #AdPl7XLjMNsCXsGCfY by mnl@hachyderm.io
2024-01-01T18:30:56Z
0 likes, 0 repeats
@freakazoid I've only worked for one VC backed company in my career, and I still am getting paid. I don't understand what you mean by "the field of programming" orients around [VC-backed companies]? Most of the things taught in CS curriculum are pretty separated from what your day to day in industry looks like?
(DIR) Post #AdPl7YUz5Sry6rb6jQ by jalcine@todon.eu
2024-01-01T18:32:22Z
0 likes, 0 repeats
@mnl @freakazoid right but when you leave that, you end up picking up Microsoft, Amazon, Google, Oracle or Apple's means of "application development and deployment".OpenStack or even more Apache centric tooling don't help when the majority of roles and VC places _run_ to corporations to define how things work. Even popular programming languages are funded heavily by them.
(DIR) Post #AdPl7Zvxke2mYpEAQC by mnl@hachyderm.io
2024-01-01T18:35:50Z
0 likes, 0 repeats
@jalcine @freakazoid I use go and kubernetes and terraform and linux and php and react and elasticsearch. There's certainly arguments to be made around companies shaping those pieces of software, but honestly I don't really feel too encumbered by it. Some things I like, some I don't, many things I can fix/hack myself, nowhere is my freedom of architecture really impacted, or at least not more than by the complexity of the problems I want to solve....
(DIR) Post #AdPl7ave3JNre85QZ6 by mnl@hachyderm.io
2024-01-01T18:35:50Z
0 likes, 0 repeats
@jalcine @freakazoid If anything, the fact that the processes these pieces of software prescribe are so unified and streamlined makes it easier for me to use and adapt that software than if it was some idiosyncratic GNU-pilled ivory-tower CS stuff (from my days in CL land, for example).
(DIR) Post #AdPl7cRaQ2WoLU2RzU by freakazoid@retro.social
2024-01-01T18:38:08Z
0 likes, 0 repeats
@mnl @jalcine So you're using hammer and nails and wood which are explicitly designed so that organizations of relatively low-skilled folks can put them together into something that won't just fall down. It's nothing to be ashamed of, but it's also not engineering, or architecture.
(DIR) Post #AdPl7dCNc3xUgbvovY by mnl@hachyderm.io
2024-01-01T18:40:36Z
0 likes, 0 repeats
@freakazoid @jalcine are you telling me that i'm not doing engineering or architecture? because I then indeed think this discussion won't really lead to useful insights. I certainly think otherwise, just as I think that there is more to say, go, kubernetes and react than "materials for relatively low-skilled folks to put them together". In fact I think that framing them as such is an insult to the vibrant communities building and using those pieces of software.
(DIR) Post #AdPl7dzIgB5f8KotBA by freakazoid@retro.social
2024-01-01T18:42:07Z
0 likes, 0 repeats
@mnl @jalcine I don't know if you're doing engineering or architecture or not. There was a lot of very impressive engineering at Facebook. It just wasn't done by and large by people for whom PHP was their primary or preferred language, even though a lot of it was *in* PHP (or at least their modified version of it, later Hack).
(DIR) Post #AdPl7elrlbwFYxXfsW by mnl@hachyderm.io
2024-01-01T18:44:56Z
0 likes, 0 repeats
@freakazoid @jalcine you're telling me that my choice of tools means I'm not. or at least it reads that way.
(DIR) Post #AdPl7fLfcUZvM0IGga by freakazoid@retro.social
2024-01-01T18:43:31Z
0 likes, 0 repeats
@mnl @jalcine And as someone who has used Terraform extensively, I can say with certainty that any actual engineering that's done with Terraform is done despite the language and while constantly fighting it.
(DIR) Post #AdPl7fSPDRxxgtReBU by freakazoid@retro.social
2024-01-01T18:49:27Z
0 likes, 0 repeats
@mnl @jalcine At least in the case of PHP and Terraform, they're not tools that lend themselves well to engineering. One ends up spending a lot of time fighting them.Terraform is mostly a hack around crappy cloud APIs. And the different cloud providers spend their time trying to differentiate themselves from one another. Kubernetes is the one area where they've actually standardized. And most organizations don't use Terraform to create resources within Kubernetes. We need some analog to Kubernetes for other cloud resources.
(DIR) Post #AdPl7h07TaWoTkE5NA by freakazoid@retro.social
2024-01-01T18:51:36Z
0 likes, 0 repeats
@mnl @jalcine My preferred alternative to k8s/cloud is actual distributed operating systems that are sufficiently secure not to need a layer of virtualization either underneath or on top. We have a multi-user, multiprocess operating system being used essentially as a single-user, single-process operating system, and even the containerization system isn't considered secure enough unless you virtualize system calls using something like gVisor, and the OS underneath k8s isn't considered secure enough if it's not running on top of a hypervisor.
(DIR) Post #AdPl7hoSSQnIzrmHpo by mnl@hachyderm.io
2024-01-01T18:55:26Z
0 likes, 0 repeats
@freakazoid @jalcine but that's not an alternative to terraform. If I want to deploy 80 copies of my web application with different credentials and assets, I can do that in a reusable fashion in a couple of lines of terraform. That's not just random typing, that implies architecture and engineering.(I'm not interested in playing a recursive game of "why do we need 80 copies of a web application with different credentials in the first place", i.e. "the premise of everything is wrong").
(DIR) Post #AdPl7iYXh5epInL5fM by freakazoid@retro.social
2024-01-01T18:58:28Z
0 likes, 0 repeats
@mnl @jalcine If you're actually deploying applications with Terraform, I don't really have much to say about that. That seems like the very definition of fighting with a language that's not even designed for the thing you're using it for.I generally use Kubernetes operators for databases and users. Not for assets, but TF for assets seems like a very strange choice to me.
(DIR) Post #AdPl7jCbI9hTJ2556W by mnl@hachyderm.io
2024-01-01T19:02:11Z
0 likes, 0 repeats
@freakazoid @jalcine I configure the S3 paths for assets as module variables. I find it works quite well. I am not using kubernetes in my current job.
(DIR) Post #AdPl7jr0ru1hKMzM5w by freakazoid@retro.social
2024-01-01T19:06:12Z
0 likes, 0 repeats
@mnl @jalcine That's a good use of Terraform, especially when you're not using Kubernetes.I think Kubernetes's approach to managing resources is interesting enough on its own that it'll probably start encompassing more and more resources, including object storage. It's a little awkward now to have your container orchestration system manage external resources. But having a generic "resource API" that has a plugin that supports containers makes a lot of sense.
(DIR) Post #AdPl7kUMVbVBIPOmQa by mnl@hachyderm.io
2024-01-01T19:17:25Z
0 likes, 0 repeats
@freakazoid @jalcine I think of kubernetes as "blackboard system", tbh. Write something into a database, have little daemons subscribe to an API to receive events when that happens, and those little daemons reconcile some external state to whatever is in the database.I can build the same architecture in 200 lines of go if I need, of course with less affordances.Using kubernetes allows me to slap a ready made argocd on top and bam, now I have git backed resource management.
(DIR) Post #AdPl7kl1VepU85C5Oi by mnl@hachyderm.io
2024-01-01T18:56:50Z
0 likes, 0 repeats
@freakazoid @jalcine To me engineering and architecture is finding a functional, reasonably long-lived robust solution to a given problem. Which IMO a lot of software out there does pretty well. I'm not going to reinvent operating system virtualization because I have to deploy 5 e-commerce portals next week, and yet my work involves engineering, because you can do that well and you can do that poorly.
(DIR) Post #AdPl7lERkGMhbKxaG8 by freakazoid@retro.social
2024-01-01T19:20:50Z
0 likes, 0 repeats
@mnl @jalcine Yes exactly! A lot of people hate Kubernetes, but the main thing that bugs me is the platform it sits on top of, i.e. Linux and perhaps Xen. It's not something you would ever end up with if you were building the whole thing from scratch.ArgoCD is my deployment tool of choice as well. And I think Argo's architecture overall may be a glimpse into the future of cloud APIs generally. But perhaps not as long as Kubernetes is "just" for containers.
(DIR) Post #AdPl7m10phDI1xgMxU by mnl@hachyderm.io
2024-01-01T19:28:17Z
0 likes, 0 repeats
@freakazoid @jalcine that's what I mean by architecture/engineering being eminently possible with todays tools. I can think "versioned immutable resource management with distributed blackboard system", write a tiny dockerfile or terraform file, and make it real, and have it actually be pretty and monitored and logged and what not.
(DIR) Post #AdPl7n2735ghBfClJQ by mnl@hachyderm.io
2024-01-01T19:28:18Z
0 likes, 0 repeats
@freakazoid @jalcine Sure it would be fun to write my own versioned immutable distributed resource management system in 500 lines of erlang, but it wouldn't have the sophistication of the ArgoCD ui, the fact I can yeet it into github and reuse whatever review workflow half of the planet uses, that I can use opentelemetry to use any number of the log management products out there. And those to me are eminently important engineering decisions.
(DIR) Post #AdPl7nXJB6dokPnfw8 by mnl@hachyderm.io
2024-01-01T19:17:26Z
0 likes, 0 repeats
@freakazoid @jalcine In both cases I thought "architecturally", and k8s or go or not k8s or not go don't really play much of a role. The engineering decision is then, is slapping argocd conducive to getting the project done in time and leaving it in such a state that it won't be a pain down the line. And I find that in my line of work (retail, web stuff), more esoteric yet clever and elegant stuff rarely makes sense, engineering wise.
(DIR) Post #AdPl7nw7gqUTzNPUcC by mnl@hachyderm.io
2024-01-01T19:28:18Z
0 likes, 0 repeats
@freakazoid @jalcine Do I wish we could actually build 500 line solutions with the same amount of comfort and affordances? Yes, and I think LLMs are actually a way to do that.
(DIR) Post #AdPl7olWbjbiYnSXjc by freakazoid@retro.social
2024-01-01T19:31:28Z
0 likes, 0 repeats
@mnl @jalcine I doubt it, TBH. The hallucination problem is very difficult to solve, because LLMs function through hallucination. It's just that most of their hallucinations tend to be right. But as soon as you get too far from the training data, they just start making stuff up, and there's no way to tell when that's happening.
(DIR) Post #AdPl7pX1l7bYw7gTmC by freakazoid@retro.social
2024-01-01T19:35:56Z
0 likes, 0 repeats
@mnl @jalcine I think LLMs will run into a problem analogous to quantum computing technology is running into today and which will turn out to be insurmountable: anything that can be accomplished with them that doesn't fundamentally rely on hallucinating will be accomplished better through a determinstic algorithm, in particular improvements to languages, compilers, and IDEs.The success of AlphaFold is interesting to think about, though. I don't know how it applies to this problem, but it seems like it might. I think AlphaFold is successful because we have such a limited understanding of the underlying problem. Eventually it'll be surpassed as we learn more about the problem, and it can help us learn about the underlying problem. And today it's able to help us do real work.
(DIR) Post #AdPl7qHox92FHFZqiG by mnl@hachyderm.io
2024-01-01T19:40:32Z
0 likes, 0 repeats
@freakazoid Here's the magic though: The output of an LLM is deterministic. As in, if I use an LLM to write some kind of code, and maybe I discard it or have to slightly fix it, I still end up with totally normal, deterministic code.Or maybe I use an LLM to sketch out a variety of architectures (what if we had an erlang blackboard? or maybe we could use some PHP event queue thing? On AWS maybe?), generate some code that might or might not work, I mostly care about the "vibes" and looks.
(DIR) Post #AdPl7r0qFl31Wsdnt2 by aud@fire.asta.lgbt
2024-01-01T20:14:07.483Z
0 likes, 0 repeats
@mnl@hachyderm.io @freakazoid@retro.social the output of an LLM is not deterministic.EDIT: I see what you’re saying in better context now, but I would also say there’s no guarantee you’re getting “deterministic” results from the code the LLM spits out in any meaningful sense of the word. I’d be… cautious about this framing.
(DIR) Post #AdPl7sARxWKN6y8zVA by mnl@hachyderm.io
2024-01-01T20:14:41Z
0 likes, 0 repeats
@aud @freakazoid if it outputs "printf('hello world')", and i run it 100 times, i will get hello world 100 times.
(DIR) Post #AdPl7t0Ypm0liaWbj6 by aud@fire.asta.lgbt
2024-01-01T20:19:23.058Z
0 likes, 0 repeats
@mnl@hachyderm.io @freakazoid@retro.social I went ahead and edited my response to better reflect what you meant (as I had missed the part where you were talking about one specific result of an LLM, not the LLM output in general). But I still would not suggest that any code that is generated is deterministic in general.
(DIR) Post #AdPl7tKPdxtIi9oSfY by mnl@hachyderm.io
2024-01-01T19:40:32Z
0 likes, 0 repeats
@freakazoid @jalcine Then you end up with a pretty solid transcripts which can be reworked (with LLM assistance) into a solid RFC. Doesn't mean the final product will be written by an LLM (usually I just write it myself, because they suck at coding), but the LLM was instrumental to coming up with a good architecture and (importantly) communicating it clearly.Similarly, they are great at "here's 3 commands I ran while debugging, now turn this into a tutorial for my colleagues."
(DIR) Post #AdPl7tr1ghykLJ4VVI by mnl@hachyderm.io
2024-01-01T20:22:46Z
0 likes, 0 repeats
@aud @freakazoid yeah i wasn't very clear. What I mean is that you get code out of it, and that code is just code. It won't change if you're not looking, it's just... code as we know it. Now if you reduce this code to be something that can almost not go wrong, or if so, in very catchable and previewable and validatable ways, for example by creating ad-hoc DSLs, then you can generate away pretty freely.
(DIR) Post #AdPl7utXopaTZPG24G by mnl@hachyderm.io
2024-01-01T20:22:46Z
0 likes, 0 repeats
@aud @freakazoid LLMs are actually really quite good at generating consistent / idiomatic little languages (in fact, mixtral is quite better at generating interesting languages than GPT4). And generating at least the scaffold of an interpreter for the language, guided maybe with a few pattern names, is pretty straightforward as well, so that the LLM output is actually useful. And then you're kind of done...
(DIR) Post #AdPl7wLaQ3c24fNwPo by freakazoid@retro.social
2024-01-01T20:32:07Z
0 likes, 0 repeats
@mnl @aud I think this example illustrates a lot of what I see as problems with the field/industry.YAML is a terrible base for a DSL. And why do people use it? So they don't have to write a parser. But you still have to write a tree walker. Modern parsing tools will generate both from a single language specification. And there's no reason that specification shouldn't give you highlighting and completion as well, which you won't get from YAML beyond the basic YAML syntax.It also occurs to me that writing new code is a tiny part of programming. The vast majority of programming is maintaining existing code. LLMs don't help very much with that. So overall your productivity is not much improved unless you're one of those annoying programmers who moves on the moment the bulk of project work shifts to maintenance and bug fixes. Which is precisely why the "10x programmer" is a myth: they actually harm productivity overall by cranking out unmaintainable code that others end up having to replace, or can only change very slowly.I would also say LLM-written unit tests are pretty likely to be useless. Sure, they'll test your low level logic, but writing *good* unit tests that are actually maintainable takes a tremendous amount of thought. You gave a good example with the use of an LLM to "bounce ideas off of" or whatever, but using it to make a DSL? I'm not so sure about that, unless it's generating ANTLR or something.
(DIR) Post #AdPl7xKYlMNx7ludSC by freakazoid@retro.social
2024-01-01T20:34:32Z
0 likes, 0 repeats
@mnl @aud Also, by creating a bunch of ad-hoc DSLs, aren't you essentially reinventing CL? Or more modern languages like Clojure or R7RS that have more powerful macro systems. Which you then get code completion and syntax highlighting for for free.
(DIR) Post #AdPl7yH3FtAo3BHLcm by freakazoid@retro.social
2024-01-01T20:36:32Z
0 likes, 0 repeats
@mnl @aud Honestly, what it's sounding like to me is that you encountered CL early in your career, struggled with it, and developed a strong bias against it. Now you're doing all the stuff you would have done had you gotten good with CL using a much more complicated and expensive system.
(DIR) Post #AdPl7z2CQat4PPL076 by freakazoid@retro.social
2024-01-01T20:39:52Z
1 likes, 0 repeats
@mnl @aud In fact I'm guessing that whole system (and some of the attitudes around it) pissed you off. But I think Clojure and Racket are both excellent illustrations of what a modern LISP can look like. Clojure has managed to get some uptake in the industry because it interoperates well with Java. Racket less so.
(DIR) Post #AdQ5xgDZdA5iBxaOvY by mnl@hachyderm.io
2024-01-01T18:02:35Z
0 likes, 0 repeats
@freakazoid and yet plenty of people use their iphone to keep their budget, grocery lists, look up manuals for their devices, read books in software that I personally find quite decent. On the other side of the equation, I can buy a microcontroller for $1 that is more powerful than my PC in the nineties and run any number of opensource programs and modify them at will.I definitely don't feel buried under a giant pile of garbage, most third party solutions I use are pretty decent.
(DIR) Post #AdQ5xh3KWjUWmTnjbE by mnl@hachyderm.io
2024-01-01T18:02:35Z
0 likes, 0 repeats
@freakazoid What would you say is a good platform / good software these days? What would you build to make things better?One of the big promises of LLMs that I see (and we really don't know very much about how to program with them) is the possibility to make new operating systems / base platforms that actually get adopted feasible. ...
(DIR) Post #AdQ5xi3Mo57BsspHIO by screwtape@mastodon.sdf.org
2024-01-01T21:32:11Z
0 likes, 0 repeats
@mnl on this topic basically, what do you think about the maintainability of LLM derived code, even for an operating system say?@freakazoid
(DIR) Post #AdQ5xizrIbu2oIBzSy by mnl@hachyderm.io
2024-01-01T21:46:49Z
0 likes, 0 repeats
@screwtape @freakazoid I gave a short overview of what I have in mind here: https://hachyderm.io/@mnl/111682847572555354The OS itself would be a collection of "building blocks" with documentation that makes them amenable to be easily composed using LLMs (which also makes them very readable to humans), so that I could take the "struct to linear storage" and the "flash DMA control" and the "CPS compiler" modules and say "create an interpreter for this language, using these modules."
(DIR) Post #AdQ5xjt9z08fZo49fE by mnl@hachyderm.io
2024-01-01T21:46:50Z
0 likes, 0 repeats
@screwtape @freakazoid But the first step in that direction is what I'm building right now, which is a set of building blocks for building logistics / retail backends, or what I call "forms and tables software". You can draw similar parallels here....
(DIR) Post #AdQ5xkRXv9e1IS9cGG by mnl@hachyderm.io
2024-01-01T18:02:35Z
0 likes, 0 repeats
@freakazoid While my laptop offers a thousand X the same processing power my 90ies 8086 did for the same power consumption, I wish I could run a similar set of software on the back of a coin battery cell, but linux/windows OS abstractions don't make it possible (or at least not easily).Since most of what makes a OS usable is "glueing boilerplate" type of work, I can see LLMs making a big impact, and that's just at the code generation level.
(DIR) Post #AdQ5xkh8zA7a4pS4Zc by mnl@hachyderm.io
2024-01-01T21:46:50Z
0 likes, 0 repeats
@screwtape @freakazoid There is no reason that the checkbox that marks a product as out of stock has to be a CMS flag configured in the depth of wordpress and then ends up being indexed by the search indexer and triggers some event notifications and what not, for a mom and pop shop. I want it to work like this:- mom and pop say "I want a checkbox to set products in stock"- out comes a DSL they can preview and futz with until they like...
(DIR) Post #AdQ5xldHV0cqz8eVBw by mnl@hachyderm.io
2024-01-01T21:46:50Z
0 likes, 0 repeats
@screwtape @freakazoid - a software engineer can take the sqlite + sqlite search API + mailchimp module and say "create the interpreter for the DSL using these modules", and out comes a minimal interpreter that implements the mom and pop webshop with a bit of glue code and runs in a 128 MB RAM container or as a lambda.That I'm actually building right now.
(DIR) Post #AdQ5xmWECiZtjYMNpw by freakazoid@retro.social
2024-01-01T22:00:51Z
0 likes, 0 repeats
@mnl @screwtape It's cool that you can do that, but I would submit that it indicates a pretty huge hole in the tooling available that you need to use an LLM to do that.
(DIR) Post #AdQ5xnI5KmrK7ykbQm by mnl@hachyderm.io
2024-01-01T22:02:37Z
0 likes, 0 repeats
@freakazoid @screwtape you can do it with an LLM, it's just tedious. Tedious means that it might take a team to do it, which means the mom and pop shop won't be able to afford it. If the 18 year old nephew and LLM can do it, then the LLM is a game changer. The result would end up being pretty much the same in both cases.
(DIR) Post #AdQ5xo4IRXQKXVJ6Zs by freakazoid@retro.social
2024-01-01T22:05:26Z
0 likes, 0 repeats
@mnl @screwtape Right, but why is it tedious? Is tediousness inherent to the task? Why are you able to communicate it to an LLM while nobody has been able to make a concise language to communicate about the same problem?
(DIR) Post #AdQ5xoo1hW0GpKhcrA by mnl@hachyderm.io
2024-01-01T22:08:33Z
0 likes, 0 repeats
@freakazoid @screwtape because building that concise language to build all possible things with all possible technologies is pretty hard to build.Something like "make me a $whatever, using these prepackaged building blocks". is pretty concise and LLMs make it possible. I'm not sure what to answer otherwise. It feels I'm saying "here's a pretty cool solution" and the answer is "no, not like this".
(DIR) Post #AdQ5xpqBqxKQ2Kirrs by freakazoid@retro.social
2024-01-01T22:12:41Z
0 likes, 0 repeats
@mnl @screwtape But we're not talking about "all possible things". We're talking about things that have been glued together often enough that an LLM can take their specs and spew out code.I'm not saying LLMs don't work, just that the reason they are so useful is that there has been very little work to make aspects of programming less tedious. That is, by and large, an intuitive argument on my part from my observation of the incentives at play.We're also making our tooling many orders of magnitude more complex by inserting an LLM there. Meanwhile, you're sending everything you're doing off to a third party, because presumably you're not actually using a local LLM to do the work you're talking about.
(DIR) Post #AdQ5xsEiwi8pT0DUO0 by mnl@hachyderm.io
2024-01-01T22:19:22Z
0 likes, 0 repeats
@freakazoid @screwtape If it was easy to make things less tedious I would have done so, this is kind of a circular discussion I feel. And the things I have done to make my work less tedious I do indeed not need an LLM for.I currently mostly use GPT-4, I send everything I'm doing to github and others already. I'm pretty sure local models will work plenty well, if they aren't already.
(DIR) Post #AdQ5xt0w3ShpsWlzX6 by mnl@hachyderm.io
2024-01-01T22:19:22Z
0 likes, 0 repeats
@freakazoid @screwtape I haven't built a "natural language to working e-commerce system" with conventional means because that's kind of impossible. I haven't built a "formal language to working e-commerce system" because that's almost just as hard and mostly pointless, since I don't know what features it would need (should I support postgres? firebase? deploy on AWS? and GCP? and local? and work with PHP templates? and import wordpress? and integrate with shopify?)
(DIR) Post #AdQ5xtpyzfXUQqel6G by mnl@hachyderm.io
2024-01-01T22:19:22Z
0 likes, 0 repeats
@freakazoid @screwtape But it's actually doable with today's LLMs and a modicum (one person's? small team's?) of work, from scratch. Once we build better tools, languages, conventions and patterns around it, these things will shrink in cost and effort even further.
(DIR) Post #AdQ5xuWWRVZCYmYjPE by freakazoid@retro.social
2024-01-01T22:25:25Z
0 likes, 0 repeats
@mnl @screwtape I seriously doubt it. The incentives are all wrong. The AI companies want us dependent on them, so I strongly suspect it'll end up looking a lot like the cloud ecosystem: heavily fragmented, heavy on the vendor lock-in.
(DIR) Post #AdQ5xvLDP27H60HDQ8 by mnl@hachyderm.io
2024-01-01T22:29:21Z
0 likes, 0 repeats
@freakazoid @screwtape not sure what you mean? how can you do vendor lock-in when the API is legit "string" -> "string"? There's plenty of "opensource" models out there and I don't see that stopping, and new methods of low-cost training or finetuning are being published by the day. I'm also reasonably sure that mixtral does most of what I need an LLM to do, in its simple untuned instruct version.That said I'm not very knowledgeable about the tech itself.
(DIR) Post #AdQ5xw5ecNGNQ20Inw by freakazoid@retro.social
2024-01-01T22:32:10Z
0 likes, 0 repeats
@mnl That's my hope. But I'm sure that's not the future OpenAI envisions. Or even Mistral since they're keeping their largest model proprietary.Ideally open source LLMs will become "good enough" that there's just no way to make money on proprietary models anymore, kind of like we've seen with web servers, operating systems, etc.But doesn't language development essentially freeze in place at that point? Everything from then on gets built on LLMs. I'm not super comfortable with that idea.@screwtape
(DIR) Post #AdQ5xwrVkRXnoSOWOm by mnl@hachyderm.io
2024-01-01T22:36:20Z
0 likes, 0 repeats
@freakazoid @screwtape Indeed I hope that opensource models (or academia funded whatever) become good enough that there is no need to rely on vendors. This is for the narrow field of glueing stuff together, I'm sure the bigger companies will find something to keep themselves busy. I don't care very much.As for language development, I don't know. Does it matter? Not to me really, I'm more interested in building systems and products and making silicon do things.
(DIR) Post #AdQ5xxoMDecEkxvW7c by mnl@hachyderm.io
2024-01-01T22:36:20Z
0 likes, 0 repeats
@freakazoid @screwtape If the need for most programming is eliminated, doesn't that solve your problem?
(DIR) Post #AdQ5xyWJaDmGxIUcdc by freakazoid@retro.social
2024-01-01T22:38:01Z
0 likes, 0 repeats
@mnl I'm not sure if I understand what you mean. Outsourcing programming to LLMs isn't eliminating it. It's just adding a bunch more complexity and making systems even more opaque than they already are. But I dunno. Maybe that's OK?@screwtape
(DIR) Post #AdQ5xzJad1C1Q7XyRU by mnl@hachyderm.io
2024-01-01T22:42:42Z
0 likes, 0 repeats
@freakazoid @screwtape But if no one needs to look at what goes on behind the scenes, then do we care? I usually don't care about all the little machine instructions that the CPU actually runs when it runs my program. Likewise, if most programming is done by LLMs generating whatever LLMs generate to make things work, and a few specialists know how to make that better, then does it really matter?
(DIR) Post #AdQ5xzsKXqyx9rniam by freakazoid@retro.social
2024-01-01T22:38:59Z
0 likes, 0 repeats
@mnl I suspect we're going to end up with whole new classes of problems that we don't have today. How do you debug such a system? What kind of security vulnerabilities will it have? It's not obvious to me that an LLM can debug a system just because it can create it.Why do we still have buffer overflows?@screwtape
(DIR) Post #AdQ5y09hVGsQ1jvafQ by mnl@hachyderm.io
2024-01-01T22:42:42Z
0 likes, 0 repeats
@freakazoid @screwtape Or maybe more aptly, do I care about the software that squarespace runs when I make a website? No, I'm just building a webpage. Do I care about how mp3 and streaming and what not works when I listen to a song on spotify? No, I just listen to a song on spotify.Do I care about what the machines do when I ask them to build me an app to feed my cat? No, I'm just building an app to feed my cat.
(DIR) Post #AdQ5y0op2Nlo5HAQlM by freakazoid@retro.social
2024-01-01T22:44:55Z
0 likes, 0 repeats
@mnl Why do you need SquareSpace to make a web site, one of the simplest possible things you can do?And this is the approach we've taken at every turn: instead of simplifying, we monetize.@screwtape
(DIR) Post #AdQ5y1NYxDYjp1QAue by freakazoid@retro.social
2024-01-01T22:42:12Z
0 likes, 0 repeats
@mnl Or, to put it another way, LLMs can handle the current stack. But they won't be able to handle a stack that has LLMs in it. We'll just wind up with a different version of the same problem, and I suspect the actual capabilities of the systems we produce won't increase in line with their complexity. So we end up with a system that's, say, 10x as complex but only 2x as capable.Which is kind of the track we were already on, isn't it? We're getting into diminishing returns with complexity in computing generally.@screwtape
(DIR) Post #AdQ5y1ZGFiuuPItW9A by mnl@hachyderm.io
2024-01-01T22:48:49Z
0 likes, 0 repeats
@freakazoid @screwtape What would you suggest to make a website? I'm fine paying someone who makes it super easy to make a website and takes care of all the junk a couple of dollars per month, imo they earned it. I personally have my blog on bearblog because I enjoy not having the hassle, and I am perfectly happy giving them $9 per month for it.
(DIR) Post #AdQ5y2SuunR7Buvxtg by mnl@hachyderm.io
2024-01-01T22:48:49Z
0 likes, 0 repeats
@freakazoid @screwtape What's the alternative? hosting a server at home (since paying AWS would be monetizing?), taking care of backups, connectivity, bandwidth, DDOS protection, graphic design, a UI for administration and metrics and logs?
(DIR) Post #AdQ5y3f0TKhWthb8Nc by freakazoid@retro.social
2024-01-01T22:54:38Z
0 likes, 0 repeats
@mnl We've made every single step of that harder because of the incentives at play. Hosting a server at home is hard because my ISP won't give me a static IP without charging me an arm and a leg. They also deliberately throttle my upstream traffic. Back in the 90s we were talking about caching at every layer, which would have substantially reduced the burden of operating a web site. And there's dat and ipfs, but we don't use those because it's pointless to use anything Google won't add to Chrome. And why is it so ludicrously complicated to even render a web page?@screwtape
(DIR) Post #AdQ5y4OjjJHTBWzeeu by mnl@hachyderm.io
2024-01-01T22:57:13Z
0 likes, 0 repeats
@freakazoid @screwtape or I can just pay someone $5 and have a website right now, and so can my mom for that matter. A website that is way more reliable and feature rich than anything I'd ever built. I'm not going back to hosting my webpage like in the nineties.
(DIR) Post #AdQ5y54ZDmk1HGZ3rM by freakazoid@retro.social
2024-01-01T22:58:59Z
0 likes, 0 repeats
@mnl Nor should you. But the fact that you need to pay someone else $5 and then risk them shutting down or kicking you off for arbitrary reasons is not an accident.You can't imagine a different way that could have gone? One that would give YOU the power instead of SquareSpace and Cloudflare and Google?@screwtape
(DIR) Post #AdQ5y5sYDwivmHwylk by mnl@hachyderm.io
2024-01-01T23:01:20Z
0 likes, 0 repeats
@freakazoid @screwtape I can imagine many different ways, but I live in a society, and I can also move my webpage to another provider if those things happen. But the thing is that my mom will not be able to do any of those things without paying someone $5, and if in the future people will be able to build the app of their dreams by paying someone $5, and noone ever sees the code behind that app, that's a pretty interesting future to me.
(DIR) Post #AdQ5y6gXE6hqHJKtg8 by lispi314@udongein.xyz
2024-01-02T00:42:17.659482Z
0 likes, 0 repeats
@mnl @screwtape @freakazoid A future where no one knows how to do anything and they all just pay opaque systems that remove all non-surface user agency doesn't sound like a very nice future to me.It is also a future that excludes whoever ended up poor by misfortune or to satisfy the perverse incentives of many of those systems.
(DIR) Post #AdQ5y7bFpE4n7DsC5Q by iska@catposter.club
2024-01-02T01:23:50.303Z
0 likes, 0 repeats
@lispi314@udongein.xyz @mnl@hachyderm.io @screwtape@mastodon.sdf.org Hey @freakazoid@retro.social long time no see :hicat:Quite a long thread, all I want to tell is that if you're programming using an LLM, the prompt becomes the programming language, and it means your existing tools were bad. Same with UIs and all.I can give countless examples of all this.Most tech people haven't even seen what people at Xerox, MIT etc. have been doing before they were born, so they fall for it :acat_chew:
(DIR) Post #AdQBm5VaNIVUsYrnlY by freakazoid@retro.social
2024-01-02T01:27:11Z
0 likes, 0 repeats
@iska This is my intuition as well, TBH. I'm not firmly attached to it, but I'm also not yet convinced otherwise. The main thing I'm convinced of is that LLMs do have some utility, maybe a lot. The question of whether the tools are bad could be orthogonal. Maybe we can even use LLMs to help improve the tools.@screwtape @mnl @lispi314
(DIR) Post #AdQBm6MlBb2dXTkGeG by screwtape@mastodon.sdf.org
2024-01-02T01:29:25Z
1 likes, 0 repeats
@freakazoid @iska yes this is my ongoing obsession with that 1984 commentated anthology Interactive Programming Environments about what some norms actually were before the mid 80s.@mnl @lispi314
(DIR) Post #AdQE4WJtd3Nl9cJf60 by iska@catposter.club
2024-01-02T02:54:52.967Z
0 likes, 0 repeats
@freakazoid@retro.social @lispi314@udongein.xyz @mnl@hachyderm.io @screwtape@mastodon.sdf.org do you have any idea of these uses for LLMs? vague theories aren't interesting..Not talking about AI in general, it already found many uses
(DIR) Post #AdQpfqPxZ6YkVMoT0S by screwtape@mastodon.sdf.org
2024-01-02T02:54:07Z
0 likes, 1 repeats
@freakazoid @iska @mnl @lispi314 I wrote this polemic in a moment's passion. gopher://tilde.institute/0/~screwtape/189293024-the-llm-chatbots.txthttps://gopher.tildeverse.org/tilde.institute/0/~screwtape/189293024-the-llm-chatbots.txt@ksaj did you want to start a company that Does Problem 100 and call it "Dynamic SEO" ?#phloggersgarage #LLM #problems Edit: I guess I wandered between different AI things while saying the specific cwords LLM a bit.
(DIR) Post #AdQqodA6JqjzxA9H8K by iska@catposter.club
2024-01-02T10:08:55.526Z
0 likes, 0 repeats
@screwtape@mastodon.sdf.org @freakazoid@retro.social @lispi314@udongein.xyz @mnl@hachyderm.io @ksaj@infosec.exchange I don't think you stressed enough the cost of training LLMs.Mixtral dolphin, an uncensored finetune of Mixtral, allegedly took thousands of dollars to make. (the author rented 2 A100s for a couple of days)I think the environment should let users fine-tune the given models as they wish, and needing $10k accelerators would be a big barrier.
(DIR) Post #AdQrAk2jPqTDfKFFUO by screwtape@mastodon.sdf.org
2024-01-02T03:05:38Z
0 likes, 0 repeats
@iska I guess I'm familiar with a reasonable amount about what has been done while explicitly being against LLMs. You're welcome to draw your own conclusions.@freakazoid @mnl @lispi314
(DIR) Post #AdQrAlADFW358okjmy by pkw@mastodon.sdf.org
2024-01-02T09:17:06Z
1 likes, 0 repeats
@screwtape @iska @freakazoid @mnl @lispi314 I am assuming that LLMs are being subsidized with fresh water and power in ways that aren't truly sustainable or being paid for. Like how we subsidize the cost of producing garbage by ignoring it.I want to live in a world that there are programmers like there are plumbers and carpenters.What is your line?LLMs are really just the stuff that was in our minds the whole time?(forcing that joke pattern of the friends we met along the way)
(DIR) Post #AdQrHWUaCCIzfKVQDA by gnemmi@mastodon.sdf.org
2024-01-02T03:55:02Z
0 likes, 0 repeats
@screwtape hum ..." harvested human lives as data" ..I remember someone using the term "Human Information Harvesting" in a moment of passion too .. hopefully, the concept is starting to grow some roots!Thank you for sharing such an excellent read! 🤗 @freakazoid @iska @mnl @lispi314 @ksaj
(DIR) Post #AdQrHX8Hoa43eT5864 by screwtape@mastodon.sdf.org
2024-01-02T04:27:07Z
1 likes, 0 repeats
@gnemmi after iska said something, I felt like I was mixing AI de jour together a bit much, for example because the term LLM refers to massive natural language corpora rather than even a unifying algorithmic theme like transformers (feed forward networks optimised by feed forward networks or something). I'll think out loud about it on the #lispyGopher #climate tomorrow a bit more. Did you have a piece of writing I should mention already?@freakazoid @iska @mnl @lispi314 @ksaj
(DIR) Post #AdQsAgtfDITJrUXk7U by iska@catposter.club
2024-01-02T10:24:10.956Z
0 likes, 0 repeats
@screwtape@mastodon.sdf.org @freakazoid@retro.social @lispi314@udongein.xyz @mnl@hachyderm.io should letmust*. personalization is very important, and is one of the biggest selling points of dynamic systems.QT still can't compare in these terms, while GTK devs write a page to tell you not to use custom color schemes and icons...
(DIR) Post #AdRLwHeBI9bm7cLbw8 by mnl@hachyderm.io
2024-01-02T14:25:52Z
0 likes, 0 repeats
@freakazoid @iska @screwtape @lispi314 the thing I might not even worth bringing up, but “human language as the main language” means that my programming today mostly happens on paper. I have always done this (more visually, before llms), but this is where the actual work happens. This wouldn’t be possible without llms, as I’d be much more chained to my editor to get stuff done. I can go through 3-4 pages a day this way, yet still manage my usual quota of 20-30 commits.
(DIR) Post #AdRLwIWm1BHEqvtD1s by iska@catposter.club
2024-01-02T15:57:43.545Z
0 likes, 0 repeats
@mnl@hachyderm.io @freakazoid@retro.social @screwtape@mastodon.sdf.org @lispi314@udongein.xyz This reminded me of some old mobile game where you can draw the custom levels on paper then shoot it with a camera. It was similar to Mario Maker, but came out before that.Why can't we do this with programming? Just do simple shape + text recognition, give the ability to define own clauses and shapes, and you'll have programming on paper, with added interactivity if you get a cheap drawing tablet.All this without any LLMs, so you can run it offline on a 10 year old laptop.
(DIR) Post #AdRMrqkHTA7nU3eNRg by mnl@hachyderm.io
2024-01-02T15:58:56Z
0 likes, 0 repeats
@iska @freakazoid @screwtape @lispi314 i suspect that it's hard to build...
(DIR) Post #AdRMrrRAtgR5d5idIu by iska@catposter.club
2024-01-02T16:08:07.598Z
0 likes, 0 repeats
@mnl@hachyderm.io @freakazoid@retro.social @screwtape@mastodon.sdf.org @lispi314@udongein.xyz OCR is built into your iphone, tons of shape recognition libraries exist. Codegen/AST manipulation feels natural to me.