[HN Gopher] Systems ideas that sound good but almost never work
       ___________________________________________________________________
        
       Systems ideas that sound good but almost never work
        
       Author : kaycebasques
       Score  : 380 points
       Date   : 2024-12-31 16:47 UTC (1 days ago)
        
 (HTM) web link (hardcoresoftware.learningbyshipping.com)
 (TXT) w3m dump (hardcoresoftware.learningbyshipping.com)
        
       | PaulHoule wrote:
       | (1) DSLs work great sometimes. See https://www.jooq.org/
       | 
       | (2) Elastic Load Balancer is a control loop responsive to
       | workloads, that kind of thing is a commodity
       | 
       | (3) Under-provisioning is rampant in most industries; see
       | https://erikbern.com/2018/03/27/waiting-time-load-factor-and...
       | and https://www.amazon.com/Goal-Process-Ongoing-
       | Improvement/dp/0...
       | 
       | (4) Anomaly detection is not inherently a problem of distributed
       | systems like the others, but someone facing the problems they've
       | been burned with might think they need it. Intellectually it's
       | tough. The first algorithm I saw that felt halfway smart was
       | https://scikit-learn.org/1.5/modules/outlier_detection.html#...
       | which is sometimes a miracle and I had good luck using it on text
       | with the CNN-based embeddings we had in 2018 but none at all w/
       | SBERT.
        
         | billyp-rva wrote:
         | > (1) DSLs work great sometimes.
         | 
         | I'll take a stab at fleshing this out: DSLs work great when
         | they have an IDE with autocomplete and a quick (or instant)
         | feedback loop.
        
           | diggan wrote:
           | I was gonna say something like "DSLs work great when they're
           | small, purposeful and easy to test", I guess yours kind of
           | helps when they're not what I'd suggest :)
        
             | PaulHoule wrote:
             | Maybe his problem is with those yucky distributed systems
             | as in Kube, Antilles, etc. I write plain ordinary Java
             | programs that work with _a_ cloud API and compile bash
             | scripts that colonize machines.
        
               | verdverm wrote:
               | I think of Kube as more of abstractions whereas Wasp &
               | Dark lang would be DSLs for the same concepts
        
             | tinthedev wrote:
             | This whole thread said exactly what I wanted to write.
             | Feels bad to be so pre-empted.
             | 
             | DSLs that solve a specific problem with a page or two of
             | documentation overhead are great.
             | 
             | Trying to reinvent paradigms or scope creep is where the
             | pain comes in. Seems like the post author has been burned
             | by that type of DSLs.
        
               | f1shy wrote:
               | I would say that is the line between DSL and just "L"
               | another language...
        
               | earnestinger wrote:
               | > DSLs that solve a specific problem with a page or two
               | of documentation overhead are great.
               | 
               | Do you have any example? I've heard lots of good things
               | of dsl, but never had the luck to witness it's full
               | glory.
               | 
               | (except for regex, which I love, but it has more than two
               | pages of docs)
        
           | f1shy wrote:
           | So any DSL as an embedded language inside another.
        
           | mdaniel wrote:
           | > and a quick (or instant) feedback loop.
           | 
           | And yet Terraform/Tofu continues to poison people's brains.
           | It boggles(!) the mind
        
           | crabbone wrote:
           | Counterexample: regex. In terms of how successful DSLs are, I
           | think, something like Perl's regular expression is, at least,
           | in the top ten. Most regex users don't care about there being
           | an IDE for it, I don't think there's a lot of value for regex
           | autocomplete, even if such thing existed.
        
         | pizlonator wrote:
         | I think you're responding to the tweet from Martin that Steven
         | includes at the top, not to Steven's list.
        
           | kelnos wrote:
           | That's fine. The points made in the tweet are interesting to
           | discuss too.
        
           | echelon wrote:
           | The tweet has some claims that don't fly.
           | 
           | You don't get the luxury of offline migrations or single-
           | master writes in the high-volume payments space. You simply
           | don't have the option. The money must continually flow.
        
         | stickfigure wrote:
         | jOOQ is a disaster and I would not recommend it to anyone.
         | 
         | You write some SQL queries, test them in datagrip or whatnot,
         | then spend the next several hours figuring out how to convert
         | them to the DSL. This problem is compounded when you use
         | "exotic" SQL features like json expressions. Debugging is
         | "print the generated sql, copy it into datagrip/whatnot, tune
         | the query, then figure out how to retrofit that back into the
         | DSL".
         | 
         | It's a huge waste of time.
         | 
         | The primary selling point of jOOQ is "type safe queries". That
         | became irrelevant when IntelliJ started validating SQL in
         | strings in your code against the real data. The workflow of
         | editing SQL and testing it directly against the database is
         | just better.
         | 
         | jOOQ reinforces the OP's point about DSLs.
        
           | geophile wrote:
           | This is a very specific and popular subset of the DSL point:
           | Let's just invent a language L that is better than horrible
           | standard language X but translates to X. Imagine the vast
           | cubicle farms of X programmers who will throw off their
           | chains and flock to our better language L!
           | 
           | In many scenarios (including JOOQ and all ORMs), X is SQL. I
           | should know, I spent years working on a Java-based ORM. So
           | believe me when I say: ORMs are terrible. To use SQL
           | effectively, you have to understand how databases work at the
           | physical level -- what's a B-tree lookup, what's a scan, how
           | these combine, etc. etc. You can often rely on the optimizer
           | to do a good job, but must also be able to figure out the
           | physical picture when the optimizer (or DBA) got things
           | wrong. You're using an ORM? To lift a phrase from another
           | don't-do-this context: congratulations, you now have two
           | problems. You now have to get the ORM to generate the SQL to
           | do what really needs to be done.
           | 
           | And then there are the generalizations of point made above:
           | There are lots of tools that work with SQL. Lots of
           | programmers who know SQL. Lots of careers that depend on SQL.
           | Nobody gives a shit about your ORM just because it saves you
           | the trouble of the easiest part of the data management
           | problem.
        
             | sroussey wrote:
             | > To use SQL effectively, you have to understand how
             | databases work at the physical level -- what's a B-tree
             | lookup, what's a scan, how these combine, etc.
             | 
             | This is a good reason to use an ORM. But also, as a ORM
             | designer, don't let the ORM be flexible to do any SQL. Only
             | let it do performant data access.
        
             | stickfigure wrote:
             | This is an odd take. Your programming language works with
             | objects, the data is in relational tables, you need
             | software to map the relations to objects. Thus the Object
             | Relational Mapper. There's no reason you can't write SQL
             | and let an ORM handle the result set mapping.
        
               | sgarland wrote:
               | If that's all you were doing, then _maybe_ , but it never
               | is. ORMs enable people who have no idea how RDBMS works
               | to use them, which rarely ends well.
               | 
               | I'm not suggesting that to use RDBMS you should know how
               | to administrate and tune it (though it helps), but
               | knowing their language, and understanding a single data
               | structure (B+ trees) isn't too much to ask, I think.
        
               | gedy wrote:
               | > ORMs enable people who have no idea how RDBMS works to
               | use them, which rarely ends well.
               | 
               | In some cases, but the more frequent issue I saw back in
               | the day was the DBA making some really complex schema
               | tuned for what they wanted, then an application trying to
               | use the data in a pretty reasonable OOP manner (1 to many
               | relationships, etc) and the DBA pissed they were using an
               | ORM instead of their perfect SQL queries and procedures.
        
               | sgarland wrote:
               | > the DBA pissed they were using an ORM instead of their
               | perfect SQL queries and procedures.
               | 
               | Tbh, I don't understand why this is seen as a bad thing.
               | Correction: I know why it is (any changes are obviously
               | going to be dramatically slowed down), but in the long
               | run, I don't understand why people are against it. You
               | wanted something done correctly, so you went to the SME
               | for that specific field, and had them do it for you. Then
               | you decided to throw it away?! Why are you bothering to
               | ask them in the first place?
               | 
               | > 1 to many relationships, etc
               | 
               | I know this was just an example, but 1:M is a perfectly
               | natural part of any RDBMS, and in no way requires an ORM
               | to be done.
        
               | stickfigure wrote:
               | Of course. And after you understand SQL and databases,
               | ORMs can save you a lot of typing. I've never understood
               | the either/or attitude.
        
               | sgarland wrote:
               | Sometimes it can. Other times, you have the SQL already
               | written in your head, but then you have to figure out how
               | to coerce the ORM to doing what you want.
        
               | stickfigure wrote:
               | Even Hibernate has `em.createNativeQuery("type your sql
               | here", SomeResult.class)`. I've never seen an ORM (for an
               | RDBMS) that didn't make it easy to run SQL.
        
               | crabbone wrote:
               | Then what's the point of using Hibernate? Just use the
               | ODBM driver... why are you dragging the gorilla and all
               | of the jungle with you if all you wanted was a banana?
        
               | stickfigure wrote:
               | Since we're talking about Hibernate, I assume you mean
               | the JDBC driver? Because the API is tedious and
               | unpleasant.
               | 
               | The mapping of database results to java objects with
               | Hibernate is convenient. The basic "load entity, change a
               | couple fields, let Hibernate persist it" flow is
               | convenient. In a limited set of cases, basic entity graph
               | navigation is convenient.
               | 
               | As I said, if you're working in an object-based language,
               | by definition you need something that maps relations to
               | objects. Hibernate is a competent choice. There are other
               | competent choices, but JDBC is not one of them unless
               | your app is trivial.
        
               | crabbone wrote:
               | Yeah, I confused multiple acronyms here :)
               | 
               | Anyways. Hibernate works on top of JDBC, so, if you like
               | its interface, then it means you could make your own, but
               | skipping >99% of the rest of Hibernate code that has
               | nothing to do with wrapping the driver.
               | 
               | Or, imagine there was a library Hibernate', that threw
               | away all the ORM stuff, and only offered mapping of SQL
               | results to Java objects and sending queries to the
               | database. Then, why not use Hibernate' instead of
               | Hibernate?
               | 
               | NB. About triviality. From experience: trivial apps tend
               | to work OK with ORM. Non-trivial will usually ditch the
               | ORM because of performance, missing functionality and
               | general difficulty with servicing it. So, it's the other
               | way around: if you are shooting for the stars, you are
               | probably not going to use Hibernate, Hibernate is one of
               | the variety of tools that helps losers loose less, it's
               | not a tool for the winners.
        
               | crabbone wrote:
               | Nah. That's an inconsequential part of the interaction
               | between the database and the application. The reality is
               | that your code has both, the database and the
               | application. And if you want to write good software, you
               | need to know how both work and be an expert at that.
               | 
               | It's infinitely easier and less error-prone to keep the
               | interface between the database and the application to the
               | minimum (just convert the final results of a query to the
               | application objects and embed complete queries in the
               | application code) than to try and create complex query
               | builders behind the scenes of object-to-object
               | interaction.
               | 
               | If you want to make a good product, you may start with
               | ORM, as it may, for a time, delay the need of
               | understanding the relationship between the application
               | and the database, and allow you to experiment faster at
               | the expense of lost performance. Once you know what you
               | need to do, ORM just no longer works: you will have to
               | break it at least in order to deal with performance
               | issues, but often you will also find yourself dealing
               | with the fact that a lot of what you want to express in
               | your queries is either too difficult or even impossible
               | to express in a particular ORM.
        
           | hot_gril wrote:
           | Yep, abstracting away SQL is a common and very costly
           | mistake. The article is about more general system design,
           | otherwise I would have expected to see that in the list.
        
         | corinroyal wrote:
         | I'm always baffled by hate for DSLs until I realize that what
         | people are criticizing aren't DSLs, but DSLs you have to write
         | from scratch. If you host your DSL on Lisp, then all you have
         | to write is your domain logic, not the base language. Most of
         | the work is already done, and your language is useful from day
         | one. I don't understand why people insist on creating new
         | languages from scratch just to watch them die on the vine, when
         | these langs could have been hosted DSLs on Lisp and actually
         | get used.
        
           | TimTheTinker wrote:
           | Not just Lisp, but any language that has strong support for
           | either literal in-language data expressions like JSON or
           | YAML, or meta-language support like Ruby, Elixir, JSX/TSX (or
           | both!).
           | 
           | Every time you write a React JSX expression, terraform file,
           | config.yaml, etc., you're using a DSL.
           | 
           | I once wrote a JSON DSL in Ruby that I used for a template-
           | based C# code generator. This enabled a .NET reporting web
           | app to create arbitrarily shaped reports from arbitrary rdmbs
           | tables, saving our team thousands of hours. Another team
           | would upload report data to a SQL Server instance, write a
           | JSON file in the DSL, check it against a tiny schema
           | validator website, submit it, and their reports would soon be
           | live. One of the most productive decisions I ever made.
        
           | CyberDildonics wrote:
           | This is generally a terrible way to work. Making a bunch of
           | custom syntax even in the same language is just adding more
           | stuff to memorize for no gain.
           | 
           | Even in C using the "goes to operator" of while(i --> 0) or
           | using special operator overloading like the C++ STL >> and <<
           | operators for concatenation is just making people memorize
           | nonsense so someone writing can be clever.
           | 
           | People don't give presentations with riddles and limericks
           | either. It can be clever as a puzzle but when things need to
           | get done, it is just indulging someone showing off their
           | cleverness at the expense of everyone who has to deal with
           | it.
        
             | f1shy wrote:
             | I think you misunderstood what a DSL is, or at least the
             | point of the OP?
             | 
             | We are advocating exactly to keep the syntax the same as
             | the base language, and add semantic value through the
             | abstractions of the language.
        
               | CyberDildonics wrote:
               | I didn't misunderstand anything.
               | 
               | If you're not changing any syntax and are just using
               | normal function calls, that's an API and that's direct.
               | 
               | If you're not just using normal function calls and are
               | making your own "semantic value through abstractions of
               | the language" you aren't making something that is direct
               | and are creating something that needs to be memorized.
               | 
               | The cleverness and indirection of the new stuff that
               | hides what is really going on is 99% of the time not
               | worth what it gives you, because you have to memorize
               | this new clever thing someone came up with, then you have
               | to learn what it is actually doing underneath that is
               | being hidden.
        
               | f1shy wrote:
               | > If you're not changing any syntax and are just using
               | normal function calls, that's an API and that's direct.
               | 
               | No. Sorry. Wrong. Look SICP where they explain the
               | concept of embedded DSl. Hint: you may be conflating
               | syntax and language.
        
               | CyberDildonics wrote:
               | Everyone understands the concept. Understanding why you
               | shouldn't do it is what takes experience.
               | 
               | If you look at the source code for doom it is very
               | straight forward. No fancy stuff, not cleverness, no
               | pageantry of someone else's idea of what "good
               | programming" is, just what needs to happen to make the
               | program.
               | 
               | I'll even give you an example of an exception. Most for
               | loops in C and successors are more complicated than they
               | need to be. Many loops are looping from 0 to a final
               | index and they need a variable to keep track which index
               | they are on. Instead of a verbose for loop, you can make
               | a macro to always loop from 0 and always give you an
               | index variable, so you just give it the length and what
               | symbol to use. Then you have something simplified and
               | that's useful. It's shorter, it's clear, it will save
               | bugs and be easier to read when you need nested loops
               | through arrays with multiple dimensions.
               | 
               | I already gave examples before where clever extra syntax
               | creates an exceptional situation but gains nothing.
               | 
               | The fundamental point here is that these opportunities
               | are rare. Thinking that making up new syntax is a goal of
               | programming is doing a disservice to everyone who has to
               | deal with it in the future.
        
               | f1shy wrote:
               | You are 100% right in all, except you are talking about
               | syntax extensions (the for example) and not DSL. A DSL
               | does not need a new syntax, is a collection of
               | abstractions that allow to express problems in the
               | language of the domain problem. It is not an API, because
               | is not an interface for a functionality. Is not about
               | exposing functionality, but to add semantic value to the
               | upper layers. May (not necessarily, but may) be formed by
               | a collection of functions, in that case similar to an
               | API, in that sense. Sometimes may include indeed
               | extensions to a language, but in that case by the
               | standard means of abstraction preferred in that language:
               | clases, templates, functions, structures. The key is to
               | reduce the cognitive load for the end programmers, who
               | could be expert in the problem at hand, but not in the
               | underlying language of the embedding.
               | 
               | There is also the possibility of embedding in a non
               | programming language, like XML (E.g. launch language in
               | ROS), or S-exp in the Oracle listener config file. Also
               | you can do ad-hoc like in the .msg files of ROS. But is
               | always about semantics, not syntax. Syntax is the medium
               | only.
        
               | vbezhenar wrote:
               | If you wrote some functions, it's not DSL, it's
               | functions.
               | 
               | If you calling them in a fancy way with overloads and
               | whatnot, it's not DSL, it's fancy functions.
               | 
               | DSL is domain specific language. It includes domain
               | specific syntax, domain specific semantics and domain
               | specific libraries.
        
               | f1shy wrote:
               | Absolutely no. It may have specific syntax, but is not
               | needed. Where do you have such definition? In fact the
               | typical example is in Lisps, where you add no syntax.
               | 
               | Is not about fancy functions. And not about new syntax.
               | Is about adding semantic value. If somebody adds a
               | collection of functions that allow the expression of
               | solutions to a problem in the very language of the
               | problem, that is a DSL, if the syntax chosen, for
               | whatever reason, e.g. simplicity, happens to be the same
               | as some underlying language, that takes nothing to the
               | fact that it is a DSL.
               | 
               | If you look at the examples of SICP, they are "just"
               | fancy functions. But they are DSLs
               | 
               | An extract of the wikipedia article:
               | 
               | As embedded domain-specific language (eDSL)[4] also known
               | as an internal domain-specific language, is a DSL that is
               | implemented as a library in a "host" programming
               | language. The embedded domain-specific language leverages
               | the syntax, semantics and runtime environment
               | (sequencing, conditionals, iteration, functions, etc.)
               | and adds domain-specific primitives that allow
               | programmers to use the "host" programming language to
               | create programs that generate code in the "target"
               | programming language.
        
           | f1shy wrote:
           | Exactly. Good DSL are typically (although not always)
           | embedded in another. When that is the case, they tend to be a
           | perfect abstraction (if decently implemented)
        
           | seanmcdirmid wrote:
           | Eh, you can host DSLs in Kotlin and C# these days, you don't
           | even have to sell your engineering team on Lisp. The biggest
           | challenge is to explain how an embedded DSL differs from
           | being just a library (interop outside of the eDSL to the host
           | language is still hard).
        
           | PaulHoule wrote:
           | with static imports in Java I buld DSLs that are basically
           | Lisp style DSLs like                  var f =
           | f1(f2(a,b,f3(c,quote(f4)))
           | 
           | which have a grammar backed by the full faith and credit of
           | the Java type system. You can code gen the static imports.
        
         | travisjungroth wrote:
         | I've written two DSLs (one with a team) and I'd consider them
         | both successful. They solved the problem and no one cursed them
         | out. I think the most important factor is they were both
         | _small_.
         | 
         | They were very similar. I even reused the code. One was writing
         | rules to validate giant forms, the other was writing rules to
         | for decisions based on form responses.
         | 
         | Ok, just ranting on DSLs. Good DSLs take someone from can't to
         | can. A DSL that's meant to save time is way less likely to be
         | useful because it's very likely to not save you time.
         | 
         | In both of my DSLs, it's that we needed to get complex domain
         | behavior into the program. So you either need to teach a
         | programmer the domain, partner a programmer with a domain
         | expert, or teach a domain expert how to program.
         | 
         | Putting the power in the hands of the domain expert is
         | attractive when there's a lot of work to be done. It frees up
         | programmers to do other things and tightens the feedback loop.
         | If it's a deep domain, it's not like you want to send your
         | programmer to school to learn how to do this. If it's shallow,
         | you can probably have someone cheaper do it.
         | 
         | A DSL comes with a lot of cognitive overhead. If the other
         | option is learning a full programming language, this becomes
         | more reasonable.
         | 
         | A time saving DSL is where someone already knows how to write
         | code, they just want to write less of it. This is generally not
         | as good because the savings are marginal. Then when some
         | programmer wants to change something, they have to
         | learn/remember this whole DSL instead of more straightforward
         | code.
         | 
         | Actually, this makes a simpler rule of thumb. A DSL for
         | programmers is less likely to be a good idea than a DSL for
         | non-programmers.
        
           | f1shy wrote:
           | DSL are just great, have never failed for me. I have done it
           | several times, the last one just the past year, for
           | programming complex ASICs. I've seen uncountable times
           | working like a charm.
           | 
           | I cannot understand why seems it was bad for him...
        
             | travisjungroth wrote:
             | I think there are two big reasons DSLs often underdeliver.
             | The first is not understanding the cognitive overhead
             | budget. If it's something where it's being used
             | infrequently or by a lot of new people, that's a lot of
             | overhead to be spent each time. Sometimes people think of
             | writing DSLs for tests because they have to write a bunch,
             | but it's really easy for this to suck. I have a test turn
             | red and now I have to learn a DSL to deal with it? Ew.
             | 
             | The second is fuzzier. It's putting a DSL over something
             | complex and hoping this will fix things. Writing SQL
             | queries for this system takes a bunch of time and is error
             | prone? Just put a DSL over it! Except all those details and
             | errors are probably going to leak right through your DSL.
             | 
             | You want to master the domain before you put a DSL over it.
        
               | f1shy wrote:
               | >> The first is not understanding the cognitive overhead
               | budget. If it's something where it's being used
               | infrequently or by a lot of new people, that's a lot of
               | overhead to be spent each time. Sometimes people think of
               | writing DSLs for tests because they have to write a
               | bunch, but it's really easy for this to suck. I have a
               | test turn red and now I have to learn a DSL to deal with
               | it?
               | 
               | What is the alternative to the DSL with lower cognitive
               | load? I do not follow. Every single DSL I've seen REDUCES
               | the cognitive load, by allowing to express the concept in
               | the mere language of the problem at hand, for which the
               | SME should be more than familiar with.
               | 
               | About the second point: I see many critics in this thread
               | based on DSls above SQL. Whatever somebody is doing above
               | SQL and selling as a DSL, it is not. Period. I cannot
               | think in any possible way of doing a DSL above a query
               | language. No doubt people hate the idea. Is a BAD one.
        
               | travisjungroth wrote:
               | > What is the alternative to the DSL with lower cognitive
               | load?
               | 
               | In the test example, writing it directly in the
               | programming language. This will usually lead to code that
               | is more verbose and repetitive, but understanding the
               | first example will be faster.
               | 
               | I think of cognitive load like a line. X is the number of
               | cases you're working with, Y is cognitive load [0]. For
               | someone who already knows a programming language, the DSL
               | is going to have a higher Y intercept since you have to
               | learn something new before you understand the first case.
               | Hopefully, it's a shallower slope so as you deal with
               | more cases the upfront cost gets paid back. If you have
               | lots of people dealing with one case or doing it
               | infrequently enough they have to relearn each time, this
               | payoff never happens.
               | 
               | This model extends past DSLs to all abstractions. It's
               | why people often end up happier with test code that's
               | less abstract/DRY. The access pattern supports it.
               | 
               | Looking at it this way also explains why a DSL for a non-
               | programmer is more likely to be useful. The intercept can
               | be lower than an actual programming language, so you're
               | ahead from the start.
               | 
               | [0] It's really more of a curve, but the line model works
               | conceptually.
        
               | epgui wrote:
               | These examples sound like things that aren't really
               | DSLs... Or in other words it sounds like someone is
               | trying to make something "simpler than it actually is".
               | 
               | DSLs are supposed to be for making it easier to perform
               | computation in a specific context. Software tests have
               | about as many degrees of freedom as the programming
               | language they are written in, so I'm not sure they are an
               | ideal use case for a DSL-- not without a lot of
               | discipline at least.
               | 
               | For a DSL to make sense, IMHO, you need to be able to
               | write down a complete and correct specification for it. I
               | doubt that is even possible in the given examples :shrug:
        
             | bjourne wrote:
             | There are hundreds of DSLs for ASIC design but not a single
             | one of them has ever been used for _actual tapeout_. It 's
             | 100% unheard of. Hence, I doubt your DSL saved any time
             | over using an RTL language directly. Sorry for sounding
             | harsh, but if you work in the area you understand my
             | skepticism about ASIC design DSLs.
        
             | layer8 wrote:
             | Are your DSLs used by other people and do they share your
             | opinion? In my experience DSLs are nice to work with for
             | the creator, but it's much more work in documentation,
             | training, intelligible error handling, and so on, to make a
             | DSL that's easy for others to learn and use.
             | 
             | I do like DSLs, but the value proposition is often
             | difficult, IMO.
        
             | throw16180339 wrote:
             | > I cannot understand why seems it was bad for him...
             | 
             | There are many poorly designed libraries, and DSL design is
             | no easier. While I haven't personally encountered any, I'm
             | sure there are numerous half-baked DSLs out there.
        
             | crabbone wrote:
             | There's a difference between "works once for my very
             | specific problem" and "works most of the time for a wide
             | range of problems".
             | 
             | DSLs, in my experience, usually fail in the later
             | definition. It's very hard to make a small language that
             | precisely captures its domain of application, will produce
             | easy to manage programs no matter the size, would be easy
             | to analyze in terms of performance and side-effects.
        
           | Majromax wrote:
           | I think the theory of domain-specific languages is very
           | valuable, even if there's rarely a need for a full
           | implementation of one.
           | 
           | As I see it, a DSL is just the end-state of a programmer
           | creating abstractions and reusable components to ultimately
           | solve the real problem. The nouns and verbs granted by a
           | programming interface constrain how one thinks, so a flexible
           | and intuitive vocabulary and grammar can make the "real
           | program" powerful and easy to maintain. Conversely, a rigid
           | and irregular interface makes the "real program" a brittle
           | maintenance nightmare.
        
             | travisjungroth wrote:
             | I agree. The line between a DSL and regular old programming
             | abstractions is fuzzy. Learning some language design is
             | very helpful because you'll see every abstraction is a
             | little piece of language.
        
           | d0mine wrote:
           | Here's a few DSL (for programmers) examples that might make
           | sense (the context is embedding DSLs into Python literal
           | strings): https://news.ycombinator.com/item?id=41214365
           | 
           | For example, bash, SQL DSLs may be immediately useful by
           | protecting against shell,sql injection: shutil.run(sh"command
           | {arg}") may translate to subprocess.run(["command",
           | os.fspath(arg)])
           | 
           | No shell--no shell injection. The assumption is that it
           | enables sh"a | b > {c}" syntax (otherwise just call
           | subprocess.run directly). Implementing it in pure Python by
           | hand would be more verbose, less readable, more error-prone).
        
             | travisjungroth wrote:
             | Yes, there are definitely counterexamples. It's not black
             | and white at all.
        
           | lifeisstillgood wrote:
           | >>> A DSL for programmers is less likely to be a good idea
           | than a DSL for non-programmers.
           | 
           | Nail on the head time - somewhere else in the thread is jooq
           | which is (yet another) SQL DSL where you end up with
           | from(table).where(student=bob)
           | 
           | This is a perfect example of why the programmer should
           | (just?) learn SQL instead of the DSL - and your comment nails
           | it
        
             | PaulHoule wrote:
             | (1) JooQ shines for query generation. For instance at work
             | we have a search engine that can search maybe 50 fields
             | that are mostly materialized but involve the occasional
             | subquery. Also you can easily write a template method like
             | transitiveQuery(selectClause,relationshipGenerator,where1,.
             | ..)
             | 
             | and have type inference do the right thing in the compiler
             | and IDE and all of that. You get "hygenic macros", you can
             | write an Java class which is parameterized by one or more
             | related tables which can be specialized by adding more
             | parameters, subclassing, etc.
             | 
             | (2) Circa 2005 coding PHP I came to the conclusion that the
             | ORM I needed was                  insert(table,{col1: val1,
             | col2:val2, ...})
             | 
             | because writing simple inserts and updates against SQL is
             | for birds, let freshers do it and they will forget to
             | escape something. Such a "framework" can be so simple that
             | you can bend it to the needs of your application. JooQ
             | gives you something like that but backed by the Java type
             | system.
        
               | lifeisstillgood wrote:
               | I do get it (I think!) - but there is a world of
               | difference between "I have 20 years SQL experience and do
               | not want to spend hours maintaining 200 SQL templates,
               | and believe the overhead of this DSL is worth the trade
               | off" vs "use the DSL and you won't have to teach junior
               | Devs SQL!"
               | 
               | My comment is more aimed at the second part. SQL Is tied
               | to the implementation and demands coders understand it
               | all. A DSL can allow domain experts to express their
               | understanding without having to worry about software
               | trade offs.
               | 
               | The most successful "DSL" I know of like this is fitnesse
               | tests - just a large number of simple tests where domain
               | experts can spreadsheet style throw in the "gotchas".
               | 
               | Something like that but more spool is fixated is a holy
               | grail - Behaviour driven tests like cucumber come close
               | but there is that weird intermediate translation from
               | English phrase to random function - now you have to
               | understand the function to use the phrase and suddenly
               | you are reading real Code to be able to use the fake code
               | and it never feels clean
               | 
               | One day I will be clever enough to be able to write a
               | really good test DSL
               | 
               | It's just whenever I think of "Given used is logged in,
               | visit "textbox" and enter "word" .. it just looks like
               | BDD test not a DSL. Like I said, one day I will be clever
               | enough
        
             | ramchip wrote:
             | I'm not familiar with jooq, but I've used Ecto a ton, and
             | the point was never to avoid learning SQL. It's about
             | making queries composable and mapping to domain objects
             | automatically so eg. there aren't dozens of queries to
             | update when you add a field to a domain object.
        
         | hot_gril wrote:
         | I've yet to see a DSL work great. Every single time, I'm asking
         | "why isn't this just Python (or some other lang)" especially
         | the times when it's some jacked up variant of Python.
        
           | dude187 wrote:
           | Groovy running on jython
        
         | einpoklum wrote:
         | (3.b) "Being clever rather than over-provisioning" is not
         | generally thought of as a good idea. People would be rather
         | apprehensive if you told them "I'm don't something really
         | clever so we can under- or exactly-provision". I mean, sure, it
         | may indeed work, but that's not the same thing.
         | 
         | (5) Hybrid parallelism - also, many people think it's a bad
         | idea because it makes your software system more complex. Again,
         | it may be very useful sometimes, but it's not like many people
         | would go "yes, that's just what I'm missing right now, let's do
         | parallelism with different hardware and different parts of the
         | workflow and everything will run something kind of different
         | and it'll all work great like a symphony of different
         | instruments".
        
         | Spivak wrote:
         | Also k8s is living proof that control loops can and do work.
         | That's like the entire point of its existence.
        
         | crabbone wrote:
         | I've never seen a good DSL beside something like regular
         | expressions, and even there, I hear, a lot of people are upset
         | by the language.
         | 
         | Examples of popular DSLs that I would characterize as bad if
         | not outright failures:
         | 
         | * HCL (Terraform configuration language). It was obvious from
         | the very beginning that very common problems haven't been
         | addressed in the language, like provisioning a variable number
         | of similar appliances. The attempts to add the functionality
         | later were clumsy and didn't solve the problem fully.
         | 
         | * E4X (A JavaScript DSL for working with XML). In simple cases
         | allowed for more concise expression of operations on XML, but
         | very quickly could become an impenetrable wall of punctuation.
         | This is very similar to Microsoft's Linq in that it gave no
         | indication to the authors of how computationally complex the
         | underlying code would be. Eventually, any code using this DSL
         | would rewrite it in a less terse, but more easy to analyze way.
         | 
         | * XUL (Firefox' UI language for extending the browser's
         | chrome). It worked OK if what you wanted to do was Firefox
         | extensions, but Firefox also wanted to sell this as a
         | technology for enterprise to base their in-house applications
         | on Firefox, and it was very lacking in that domain. It would
         | require a lot of trickery and round-about ways of getting
         | simple things done.
         | 
         | * Common Lisp's string formatting language (as well as many
         | others in this domain). Similar to above: works OK for small
         | problems, but doesn't scale. Some formatting problems require
         | some very weird solutions, or don't really have a solution at
         | all (I absolutely hate it when I see code that calls format
         | recursively).
         | 
         | All in all. The most typical problem I see with this approach
         | is that it's temporary and doesn't scale well. I.e. it will
         | very soon run into the problems it doesn't have a good solution
         | for. Large programs in DSL languages are often a nightmare to
         | deal with.
        
       | bee_rider wrote:
       | What do people want to use hybrid parallelism for in systems
       | programming?
       | 
       | It works really well in the HPC world I think (MPI+OpenMP on a
       | node is the de-facto standard). But... I dunno, I guess when I
       | think systems programming I think of the stuff doing all the
       | bookkeeping. The bookkeeper better not steal all my cores!
       | 
       | DSL is a sort of interesting one. What's a DSL for systems
       | programming? I'll naively throw C and Rust as the systems
       | programming DSLs. Of course, the domain of systems programming
       | is, uh, controlling all the hardware. So it isn't that surprising
       | that the DSLs of systems programming quickly become the languages
       | that everybody wants to use for everything, right? The problem
       | isn't the "specific language" part, it is that a good enough
       | systems programming language quickly gets the domain of
       | "everything," haha.
        
         | mrkeen wrote:
         | Here's a DSL example https://copilot-language.github.io/
        
       | shermantanktop wrote:
       | There are many successful examples of all of these. Using
       | "almost" as an escape hatch doesn't work here.
       | 
       | This is just pessimism and weary cynicism. I get it, I've felt
       | that way too, and sometimes it's hard to talk an eager engineer
       | out of a bad idea. But for me, this vibe is toxic.
        
         | yuliyp wrote:
         | A lot of those "successful" examples have teams of battle-
         | scarred engineers dealing with all the failures of those ideas.
         | Control loops running away to infinity or max/min bounds, a
         | cache that can't recover from a distributed failure, corrupted
         | live-migrated state, bursts causing overload at inconvenient
         | times, spurious anomaly-detection alerts informing you of all
         | the world's holidays, etc.
         | 
         | Underneath all of those ideas is a tangle of complexity that
         | _almost_ everyone underestimates.
        
           | NBJack wrote:
           | I think that's fair: the success stories are very rarely
           | "yay, we did it!" but much more often "this single change was
           | the _sole_ focus of a team /multiple teams for X months, and
           | the launch/release/fix was considered a culmination of a
           | significant investment of resources".
        
           | api wrote:
           | I know I'm not alone in this, but after doing this for more
           | than 20 years I can't shake the idea that we are doing it
           | wrong -- meaning programming. Is it really this nit-picky,
           | brittle, and hard?
           | 
           | The brittleness is what gets me. In physical mechanical and
           | even analog electrical systems there are tolerances. Things
           | can almost-work and still work for varying degrees of work.
           | Software on the other hand is unbelievably brittle to the
           | point that after 50 years of software engineering we still
           | really can't re-use or properly modularize code. We are still
           | stuck in "throw it away and do it over" and constantly
           | reinventing wheels because there is no way to make wheels
           | fit. The nature of digital means there are no tolerances. The
           | concept isn't even valid. Things fit 100% or 0%.
           | 
           | We keep inventing languages. They don't help. We keep
           | inventing frameworks. They don't help. We keep trying "design
           | patterns" and "methodologies." They don't help. If anything
           | all the stuff we invent makes the problem worse. Now we have
           | an ecosystem with 20 different languages, 50 different
           | runtimes, and 30 variants of 5 OSes. Complexity goes up,
           | costs go up, reusability never happens, etc.
           | 
           | I remember for a while seeing things like the JVM and CLR
           | (the VM for C# and friends) as a way out-- get away from the
           | brittle C API and fully compiled static native code and into
           | a runtime environment that allowed really solid error
           | handling and introspection. But that paradigm never caught on
           | for whatever reason, probably because it wasn't free enough.
           | WASM is maybe promising.
        
             | wakawaka28 wrote:
             | >The brittleness is what gets me. In physical mechanical
             | and even analog electrical systems there are tolerances.
             | Things can almost-work and still work for varying degrees
             | of work.
             | 
             | I think you are misrepresenting how flexible software is
             | versus hardware. Mechanical and electrical systems have
             | tolerances but if you go outside those tolerances, the
             | whole system can be destroyed. Nothing like that is common
             | in software. Worst-case outcomes might be like "the
             | performance isn't as good as we want" or "this code is
             | difficult to work with." Software components are very
             | flexible compared to anything physical, even in the worst
             | cases.
             | 
             | >We are still stuck in "throw it away and do it over" and
             | constantly reinventing wheels because there is no way to
             | make wheels fit. The nature of digital means there are no
             | tolerances. The concept isn't even valid. Things fit 100%
             | or 0%.
             | 
             | I don't know how one can look at the amazing array of
             | libraries out there and conclude that we have no reuse.
             | Sometimes people build their own solutions because they
             | need something very simple and the libraries are too big to
             | be worth importing and learning in those circumstances.
             | That's not a flaw in the libraries. It's human nature.
             | 
             | >We keep inventing languages. They don't help. We keep
             | inventing frameworks. They don't help. We keep trying
             | "design patterns" and "methodologies." They don't help. If
             | anything all the stuff we invent makes the problem worse.
             | Now we have an ecosystem with 20 different languages, 50
             | different runtimes, and 30 variants of 5 OSes. Complexity
             | goes up, costs go up, reusability never happens, etc.
             | 
             | All of this is too pessimistic. These tools do help.
             | Exactly how many languages do you think we should have? Do
             | you think exactly one group is going to develop for each
             | use case and satisfy everyone?
             | 
             | >WASM has its uses but I can't escape the idea that it's
             | like "let's build a VM and carry all the shortcomings of C
             | into it."
             | 
             | I'm not a web guy but this sounds silly. It's not meant to
             | be written directly. Complaining about shortcomings of WASM
             | is literally like complaining about shortcomings of
             | assembly language. It's not intended for human consumption,
             | in modern times.
        
               | ghaff wrote:
               | There's probably more tendency to be sloppy in software
               | because "we can always fix it in post." But absolutely,
               | with hardware, if you don't get it right--especially with
               | heavy construction or modern electronics, you're going to
               | have to rip a lot out and start over.
        
               | vacuity wrote:
               | I've thought that we should be able to move so fast with
               | software, and sometimes I see that successfully, but
               | usually it seems like we leverage computers' superior
               | performance poorly. To pick on frontend, it's trivial to
               | create the next shiny Javascript framework, but what of
               | it? Or, how come refactoring can be so painful, when the
               | semantic change might be small? I think the flexibility
               | and performance of computers is such that we programmers
               | are usually incapable of effectively using them. It's
               | like a 3D optimization problem visualization, looking for
               | the highest peak around, except the cursor moves a lot
               | faster than it can peruse the landscape. It zooms around
               | aimlessly, easily getting to arbitrary places but without
               | the capacity to make sense of them. When the train is in
               | motion, switching tracks is hard, even if that would be
               | the best move.
        
               | ghaff wrote:
               | It's easy to assume that you can always fix things after
               | the fact. And I was only half joking with the fix it in
               | post comment. Modern film suffers from some of the same
               | problem. No need to get it right on the first pass. We
               | can always apply corrections later.
        
             | intelVISA wrote:
             | No, it's not terribly hard for a suitably compensated and
             | skilled team with the appropriate tools and timeframe. Yes,
             | we are doing it wrong.
             | 
             | Most of the 'inventions' you describe are more aimed toward
             | reducing the barriers of entry: the promise that your team
             | of expensive C wizards would now use Java at greater speed
             | with less defects became "now we can just use cheap CS
             | grads" at slightly worse but still acceptable levels.
             | 
             | Without any real consequences for poor software (see
             | CrowdStrike's YTD despite its multi-billion dollar farce in
             | July) it's only logical that the standard will always be
             | "bare minimum that can be shipped". Developer productivity
             | is a misnomer really - it just means company profits
             | increase thanks to a widening pool to hire from and even
             | more crapware per dollar can now be squeezed from each
             | worker.
        
             | skywhopper wrote:
             | Nah. CLR/JVM/WASM are the same pipe dream of a universal
             | architecture/shared-compute utopia.
             | 
             | But I think you have the wrong take on "reusability". Every
             | non-software engineering project is an exercise in custom
             | solutions as well. The reusable parts are the tools and
             | materials. Likewise in software engineering the languages,
             | OSes, protocols, libraries, design patterns, and frameworks
             | are the reusable bits. Code is how we describe how it all
             | fits together, but a huge amount of what it takes to run a
             | system is being constantly reused, much bigger than the
             | code we write to implement it.
        
               | api wrote:
               | I'm aware that many HN people view this as a pipe dream,
               | but why? It works. The largest compute platform in the
               | world, namely the web, is like this, and many of the
               | largest businesses run on the JVM and the CLR. Loads of
               | nasty problems go away when you are not directly mangling
               | bits in memory and when you have a real runtime.
               | 
               | Of course modern safe languages like Rust give you some
               | of those benefits in compiled code too.
        
         | samatman wrote:
         | Basically agree. This is a list of systems ideas which are
         | harder than one might initially think, and should be approached
         | seriously, never casually.
         | 
         | That slant-rhymes with "sound good but almost never work" but
         | in detail is completely different. When treated as difficult
         | problems, and committed to accordingly, having them work and
         | work well is a normal result, eminently achievable.
         | 
         | As afterthoughts, or when naively thought to be easy, then
         | yeah, they frequently go poorly.
        
         | pizlonator wrote:
         | I think Steven is saying that these things are hard and
         | unusually don't work out, not that they're impossible.
        
           | f1shy wrote:
           | Mmmm danger! Then let's stop doing anything that is
           | difficult?
           | 
           | If that is bot the message, what is that? "These things are
           | hard, often don't work, but GO FOR IT"?
           | 
           | I pretty much read: "try to avoid" which is bad advice in my
           | opinion. Like "documenting SW properly while doing
           | development is hard, and often goea wrong" so what?!
        
             | Spivak wrote:
             | Don't do anything difficult until you've exhausted the easy
             | ways first is generally good advice.
             | 
             | It's "work smarter not harder" for knowledge workers who
             | don't realize that using your brain more is the hard work
             | in the saying.
        
         | tptacek wrote:
         | It's just a high-engagement (if you like, "bait-y") way of
         | saying "these are deceptively tricky things to get right or
         | deploy effectively".
        
         | aprilthird2021 wrote:
         | Pessimism and weary cynicism can be very valuable in many tech
         | environments though. It keeps you stable, working on tried and
         | tested things, and safe. There's a lot of situations where
         | that's super valuable
        
           | shermantanktop wrote:
           | I had to learn that attitude in order to operate at one
           | employer. I had to unlearn it to survive at the next, where
           | engineers were better and routinely pulled off things I had
           | dismissed as unrealistic.
           | 
           | The value of that approach is very situational...though I
           | will acknowledge that the majority of places probably warrant
           | at least some of that.
        
         | phil21 wrote:
         | > This is just pessimism and weary cynicism.
         | 
         | I don't read it that way. I read it as engineers engaging in
         | pre-optimization for no business benefit. It's utterly rampant
         | in the industry because it's fun to design and build a
         | redundant auto-scaling spaceship vs. just over-provisioning
         | your server by 200% for a tenth (or less!) of the cost and
         | having backups ready to deploy in a few hours.
         | 
         | Sometimes these ideas make sense - after you need them. Not
         | designed-in at the early product stage. Very few products go on
         | to need the scale, availability, or complexity most of these
         | implementations try to solve.
        
       | jmsdnns wrote:
       | p2p cache sharing is an interseting one. it seemed to work for
       | spotify for a long time, but they eventually found it easier to
       | use edge caching. afaik they still use it for places with less
       | developed infrastructure
       | 
       | here is a paper written by the people who designed it:
       | https://www.csc.kth.se/~gkreitz/spotify-p2p10/
       | 
       | this part of the abstract gets right to the point: 8.8% of music
       | data played comes from Spotify's servers while the median
       | playback latency is only 265 ms (including cached tracks)
        
         | dividuum wrote:
         | Steam added a P2P caching mechanism recently and allows
         | downloading games for other local machines that already have
         | the game installed. Greatly helps with avoiding duplicate
         | downloads on a steam deck, for example.
         | 
         | I also added a similar mechanism to my own product years ago.
         | Works flawlessly. Using content based addressing already made
         | it quite easy to implement.
        
       | norir wrote:
       | DSLs make the impossible possible. Whether this is good or bad
       | depends on the design and implementation.
        
         | heisenbit wrote:
         | Compared to general purpose programming when using real world
         | DSLs one also can encounter the case where the unusually
         | possible becomes virtually impossible.
        
           | fancyfredbot wrote:
           | Which often is exactly what the DSL authors would have
           | intended. If a DSL can make the common case faster, easier to
           | program, and less buggy then that DSL is a success. The
           | uncommon case should be written in a general purpose language
           | just as it would have been if there was no DSL. Writing a DSL
           | to be general enough to handle every possible use case is
           | probably going to end up being a general purpose language,
           | and it's probably going to be worse than the one you started
           | with.
        
       | ajcp wrote:
       | It feels like problem with these aren't the ideas, rather the
       | "let's just" approach/expectation in front of them.
       | 
       | For instance "Let's just add an API." I think the approach to an
       | API as "just" a feature to your product will be about as
       | successful as saying "let's just add a UI". To implement a
       | successful UI one needs to be thoughtful, thorough, and bring in
       | people who specialize in it. Why should any other interface for
       | your product be any different? It's not that it's a bad, or good,
       | idea, rather one that shouldn't "just" be done.
        
         | vasco wrote:
         | Most professional advice is like relationship advice, people
         | extrapolate something that went wrong for them into general
         | cases but that rarely directly applies to someone who is not
         | you in that very similar situation. And the advice that seems
         | to work consistently gets so generic that is almost useless.
         | Something like "be thoughtful, try to do the right thing and
         | reflect back to see how it went" doesn't write blog posts or
         | sells books!
        
         | gmuslera wrote:
         | Cargo cults around these system ideas are dangerous. The idea
         | may be good, but without understanding what really makes them
         | work or not is dangerous.
        
           | fullstackchris wrote:
           | Dangerous? In the sense a bad API is dangerous? If so then
           | I've been in danger consuming catastrophic APIs for at least
           | a decade now
        
         | justin_oaks wrote:
         | Exactly. There's no "just" when you're adding an API. APIs
         | require a lot of work and complexity to do right:
         | 
         | APIs need to be well-designed or the client may need to make
         | multiple API calls when one should suffice. Or the API could be
         | confusing and people will call it wrong or fail to use it at
         | all.
         | 
         | APIs require authentication and authorization. This means
         | setting up OAuth2 or at least having a secure API token
         | generation, storage, and validation.
         | 
         | APIs need handle data securely or you'll leak data you
         | shouldn't, or allow modification you shouldn't.
         | 
         | APIs need to be performant or you database may be crushed under
         | the load. This may involve caching which then adds the
         | complexity of cache invalidation and other servers/processes to
         | support caching.
         | 
         | APIs need to be rate-limited or sloppy clients will hammer your
         | API.
         | 
         | APIs require thorough documentation or they're useless. You may
         | even have to add SDKs (libraries for different programming
         | languages) to make it easier for people to use the API.
         | 
         | APIs need good error messages or users who are getting started
         | will not know why their calls are failing.
        
           | eddythompson80 wrote:
           | lol, I can already hear the responses of plenty of
           | managers/management I worked with to these points.
           | 
           |  _> APIs need to be well-designed or the client may need to
           | make multiple API calls when one should suffice. Or the API
           | could be confusing and people will call it wrong or fail to
           | use it at all._
           | 
           | That's fine. It's a first iteration we can change it later if
           | people complain. Lets get something out there and iterate.
           | 
           |  _> APIs require authentication and authorization._
           | 
           | Lets not worry about authorization/permissions. It's just one
           | key or whatever account they use to log in with now.
           | 
           |  _> APIs need handle data securely or you 'll leak data you
           | shouldn't, or allow modification you shouldn't._
           | 
           | You're saying you don't know how to do that?
           | 
           |  _> APIs need to be rate-limited or sloppy clients will
           | hammer your API._
           | 
           | Either "Lets worry about that later when it happens" or "Here
           | is the first github link from a google result for 'api rate
           | limit open source free'. Lets use that"
           | 
           |  _> APIs require thorough documentation or they 're useless._
           | 
           | Either "We need to have the API first, docs, clis, etc could
           | come later after we have gauged the usage or had asks for
           | them. We can handhold the customers asking for them for now."
           | or "here is a github project that autogenerates docs, SKDs
           | and clis from an OpenAPI spec"
           | 
           |  _> APIs need good error messages or users who are getting
           | started will not know why their calls are failing._
           | 
           | You're saying you don't know how to do that?
           | 
           | "It doesn't have to be perfect. We have customers who are
           | asking for it and to win them we need to implement something
           | then we can work with them to improve it. Otherwise we will
           | lose them"(tm)
        
             | liontwist wrote:
             | Yep, helps a lot when your management is in the domain.
             | 
             | Phased implementation isn't the worst idea, they just have
             | to be committed to ending the experiment or seeing it
             | through.
        
             | 9dev wrote:
             | Then again, these answers are valid. Your fancy API
             | documentation will only be skimmed, nobody is going to
             | notice the elaborate and elegant URI design or the
             | discussion around whether to use PUT or PATCH, and the
             | super-secure token mechanism you come up with will be
             | written on a sheet of paper in the office.
             | 
             | Most projects are far too fluid in their shape to warrant a
             | proper design up-front anyway.
        
               | lazyasciiart wrote:
               | They're really not. That approach is how you get "why do
               | I need this account ID to look it up in the API but the
               | only way I can get it is by using the UI? Why can I
               | update this object but not see its current state?" and
               | other complaints that users very reasonably have about a
               | thing I worked on.
        
               | 9dev wrote:
               | Don't get me wrong, I love to obsess over that kind of
               | detail, but complaints like these usually aren't a deal
               | breaker, and yet it's reasonable to count those
               | complaining users as successful conversions regardless.
        
               | lucianbr wrote:
               | Recently I was working on a 12-year old project where the
               | product people kept asking for "quick wins" and "leave
               | the refactorings and in-depth analysis and difficult
               | problems for later". When will they get to the "later",
               | if 12 years was not enough? I eventually left.
               | 
               | Of course, YMMV. Every company and manager is different.
        
               | justin_oaks wrote:
               | In every organization I've been in, "later" meant
               | "never".
        
           | nyrikki wrote:
           | I disagree, those are all technical issues that are typically
           | blocked by socio-technical and organizational problems.
           | 
           | Retrofitting an API is a context specific problem, with few
           | things that generalize but the above aren't the hard parts,
           | just the complex ones.
           | 
           | I am going to point you to a Public Policy book here, look at
           | page 32 and their concept of 'wicked problems', which applies
           | to tech but doesn't suffer the problems with the consulting
           | industry co-opting as much as in the tech world.
           | 
           | https://library.oapen.org/bitstream/id/f3358d25-56c0-47f2-90.
           | ..
           | 
           | Obviously most companies won't accept the Amazon style API
           | edicts that require Externalizable interfaces and force a
           | product mindset and an outside in view.
           | 
           | Obviously an API is still a "Cognitively complex problem" as
           | defined by the above.
           | 
           | But what makes most API projects fail is the fact that
           | aspects arise that result in 'wickedness' A.K.A
           | intractability.
           | 
           | API gateways, WAFs, RESTful libs with exponential backoff
           | with jitter....etc... all exist and work well and almost any
           | system that you have a single DB you are concerned about
           | killing is possible to just add a anti-corruption layer if
           | you don't have too much code debt, complicated centralized
           | orchestration etc....
           | 
           | But these general, but difficult tasks like adding an API
           | almost never fail due to tech reasons, but due to politics,
           | poor communication, focusing on tech and not users,
           | unrealistic timelines, and turf battles.
           | 
           | You are correct that "APIs require thorough documentation or
           | they're useless" but more importantly they need enforceable
           | standards on contract ownership, communication.
           | 
           | The direction of control, stability, audience, and a dozen
           | other factors effect the tradeoffs there as to what is
           | appropriate.
           | 
           | Typically that control is dictated purely by politics and not
           | focused on outcomes.
           | 
           | That is often what pushes these complex problems into failed
           | initiatives, and obviously this is far more complicated.
           | Often orgs can't deal with the very real uncertainty in these
           | efforts and waste lots of project time on high effort, low
           | value tasks like producing gantt charts that are so beyond
           | the planning horizon that they can only result in bad
           | outcomes at best.
           | 
           | The point being is that unless you are on the frontier of
           | knowledge and technical capabilities, it is almost _never_
           | the tech that causes these efforts to fail.
        
             | harrall wrote:
             | When I read OP's list of wants for an API, I don't read it
             | so literally.
             | 
             | I read it as an example of the complexity required to
             | implement something like an API.
             | 
             | I agree that it's also a people problem but I disagree that
             | many of these failed initiatives aren't due to bad tech as
             | well. A complex initiative requires picking the right tech
             | that gives you a good benefit/cost ratio. Knowledge of tech
             | plays hand in hand with the political aspects and IMO many
             | orgs lack both completely.
        
               | nyrikki wrote:
               | > A complex initiative requires picking the right tech
               | that gives you a good benefit/cost ratio.
               | 
               | That choice should never be a one way door, you don't
               | control your customers or the future, it is the frame
               | problem.
               | 
               | The idea is to be able to iterate and adapt, not get
               | things perfect from the start.
               | 
               | It is fine and expected that one will use past experience
               | as a starting point, but you always have to reflect and
               | check your assumptions along the way.
               | 
               | It is a complicated topic ruled by nuances and contexts,
               | but if you cannot pivot away from your initial
               | assumptions over time, that lets you know you are leaking
               | implemention details.
               | 
               | While time is limited and abstractions have a real cost,
               | you need to apply them where they make sense.
               | 
               | Some, like just breaking code onto two files in the same
               | directory are fairly low cost, others are much more
               | expensive.
               | 
               | For several years I was jumping in to save failed cloud
               | migrations.
               | 
               | Whenever tech was the 'blocker' it was because of a myth
               | that there was only one way to do things.
               | 
               | But lets say you are writing machine code for Apple
               | silicon? Do you really think those registers are
               | concrete? They aren't. They are a facade hiding 100s
               | behind a legacy interface.
               | 
               | Vendor mitigation important and often neglected and also
               | results in people producing balls of mud.
               | 
               | But there are a lot of distributed monoliths pretending
               | to be micro services out there. And we had decades of
               | people producing fragile enterprise service buses that
               | were built because people wanted to future proof their
               | systems.
               | 
               | The sizing of components and balancing integration and
               | disintegration drivers is incredibly hard, you will never
               | get it right.
               | 
               | You have to leave options open, no matter if that is
               | through abstractions or keeping biz logic centralized to
               | assist in a easy rewrite of context boundaries that arise
               | from scale or changing needs.
               | 
               | Obviously choices have benefits and costs, but for most
               | needs you can keep options open, if not it is probably
               | best to reevaluate the reason for choosing a solution.
               | 
               | Obviously there are predatory vendors like Oracle that
               | base their entire income on captive customers, but that
               | relates to the vendor mitigation above.
               | 
               | I think the US federal government de-risking guide is a
               | good overview of that topic. Microsoft is a strong driver
               | for this BTW.
               | 
               | https://guides.18f.gov/derisking-government-tech/
        
             | btilly wrote:
             | For anyone who wants to read more about what the Amazon
             | experience was like, I highly recommend Steve Yegge's
             | platform rant at
             | https://gist.github.com/chitchcock/1281611.
        
               | com wrote:
               | It's depressing how far Google seems to have fallen since
               | then.
        
           | Groxx wrote:
           | Every single one of these is true of a website, but we have
           | far more websites than APIs.
           | 
           | If there's one thing APIs suffer from more, it's "social cost
           | to make changes". A concurrent vN+1 largely resolves that
           | though, unless your API consuming ecosystem is large enough
           | to be worth investing _vastly_ more resources into.
        
             | saghm wrote:
             | Is "let's just make a website?" a common thing people say?
             | It sounds like the issue isn't with "make an API", it's
             | with "just".
        
           | mdaniel wrote:
           | And there's an implied risk that once you have added an API
           | for customers to exfiltrate data or integrate with their own
           | system (or, worse, IMHO: a _competitor 's tool_) then they'll
           | never return to your site again
           | 
           | I'm cognizant that if it truly is a make-or-break for the
           | deal, there may not be any choice, but along with all the
           | risks you cited is an underlying obsolescence one
        
         | nswanberg wrote:
         | Yegge wrote about the business idea version of this as "Shit's
         | Easy Syndrome":
         | 
         | https://steve-yegge.blogspot.com/2009/04/have-you-ever-legal...
         | 
         | It'd have been delightfully ironic had either of these Steves
         | concluded their essays with a named methodology to "just" apply
         | whenever faced with these "let's just" situations but alas...
        
         | jabroni_salad wrote:
         | I am pretty sure presales as a career exists solely because of
         | the word 'just'. We have to make sure they wont get spooked
         | when we start lifting the rest of the iceberg out of the water.
        
         | aprilthird2021 wrote:
         | Yes, and a very well designed API can be the foundation of and
         | moat for a successful business. Stripe is one of the best
         | examples. Even as hundreds of copycats pop up, their excellent
         | API design and the trust devs have in them to nail the next API
         | design, means they can continue to expand into other financial
         | products with ease.
        
         | btilly wrote:
         | At a previous employer we had a rule. The only person allowed
         | to say, "just", was the developer responsible for actually
         | making it work. Another developer who said it, just volunteered
         | themselves.
         | 
         | It was a rule that worked well for us.
        
           | jrs235 wrote:
           | Needs to also be for "that's easy"...
        
           | sharpy wrote:
           | I am totally going to advocate for this at my workplace. I
           | myself am guilty of it - not saying "just", but questioning
           | people's estimates based on my limited understanding of the
           | research they had to do to come up with the estimates in the
           | first place.
        
             | ffsm8 wrote:
             | It sounds pretty toxic if you're not allowed to even
             | question other people's estimates. IME that's a large part
             | of Grooming/refinements
        
               | dambi0 wrote:
               | Aiming for an environment where uniformed challenges to
               | estimates are discouraged is an entirely different thing
               | to forbidding any challenges to estimates.
        
           | gdsdfe wrote:
           | That's a good one!
        
           | BlarfMcFlarf wrote:
           | Punishing people with more work doesn't make sense in a well
           | run organization. Work is continuous and more work only
           | affects the backlog, not the actual developer's life or
           | experience.
        
             | btilly wrote:
             | It was not a punishment. It was more, "You're sure that you
             | know an efficient way to do this? You get a chance to prove
             | that you're right." It made people careful because being
             | wrong had a consequence, but it usually moved work to the
             | person who thought that they had the best approach.
             | 
             | This happened in Sprint Planning. Obviously by the end of
             | the meeting, everyone would have a full sprint. So
             | volunteering for one thing required giving something else
             | up. This was definitely part of the equation.
        
               | watwut wrote:
               | It is literally punishment for saying something plus
               | passive aggressive explanation. Somewhat manipulative.
               | 
               | "Giving someone a chance" is a positive thing and wont
               | make people careful.
        
               | btilly wrote:
               | How so is it "literally punishment"?
               | 
               | People wind up with as much work as they think they can
               | reasonably do. They would have done that whether or not
               | they "just volunteered" themselves. You only wind up with
               | too much if you didn't estimate your own work well.
               | 
               | As for being passive aggressive, the word "just" usually
               | is used in a passive aggressive way. Making people
               | careful about saying it was an improvement. And we all
               | agreed that it was.
        
           | bryanrasmussen wrote:
           | Ok so if you have a problem and the developer thinks that the
           | solution to it is to add an API, all they have to do to get
           | their way is to say "let's just add an API!", although they
           | would of course be the one that had to make it work.
           | 
           | Yes I like this rule, how many things I could have solved at
           | the worst run place I ever worked at if it had been
           | implemented there. Although I guess the management might not
           | have liked that essentially I could choose what got done and
           | what I got to work on just by running my big mouth.
           | 
           | facetiousness aside, I'm trying to point out that most often
           | when people say "let's just" it is because they are
           | advocating a course of action and that rule won't work unless
           | the course they are advocating gets chosen. If they say
           | "let's just make an api" and you say no we're doing a UI and
           | you will work on it because you used the word "Just" I guess
           | that's a way to lose your tech talent relatively quickly.
        
           | evalstate wrote:
           | Gerry Weinberg wrote beautifully about these "lullaby words".
           | https://www.humansystemsinaction.com/lullaby-language/
           | 
           | > "Precisely. It's what I call a 'Lullaby Word.' Like
           | 'should,' it lulls your mind into a false sense of security.
           | A better translation of 'just' in Jeff's sentence would have
           | been, 'have a lot of trouble to.'"
        
             | natmaka wrote:
             | RFC2119 to the rescue:
             | https://datatracker.ietf.org/doc/html/rfc2119
        
               | tuukkah wrote:
               | You "should" "just" implement the RFC, right? ;-)
        
               | number6 wrote:
               | I do a lot of audition and sometimes start talking in
               | theses phrases - it is really annoying for other people,
               | when my vocabulary differs in meaning to the standard
               | body of language.
        
           | tjalfi wrote:
           | We had a similar rule for application support at my last job.
           | Anyone who introduced an application was assigned as the
           | support lead.
        
           | dataflow wrote:
           | Wouldn't this be a easy way to get out of doing whatever
           | current work you don't want to be doing?
        
             | lukan wrote:
             | Can't you just work harder and smarter?
        
             | btilly wrote:
             | Taking on what others thought was more work, in return for
             | abandoning less current work, is a loss. Unless, of course,
             | you're right about the better approach.
             | 
             | So no, not easy. Not unless you're right, and someone else
             | was wrong.
        
           | cgriswald wrote:
           | There's a fundamental difference between being able to
           | recognize the best option and being able to implement the
           | option.
           | 
           | These sorts of rules are really about people feeling devalued
           | or disliking being volunteered or told what to do (often by
           | people they consider less knowledgable). They aren't really
           | about effectively distributing work.
           | 
           | "Just" gets a bad wrap. There's a sort of hidden assumption
           | here that "you can just" is equivalent to "you can easily".
           | It _sometimes_ means that but more generally it means
           | something more like "...will be easiest" which can be true
           | even when the action itself is hard or a lot of work.
        
           | StefanBatory wrote:
           | Or it lead to people being silent and keeping any ideas to
           | themselves ;P
        
             | btilly wrote:
             | People were free to offer any idea that they want. But you
             | couldn't say "just".
             | 
             | Equivalent weasel words are easy to come by. "I don't see
             | why it wouldn't work to ..." But now you're asking for an
             | explanation, without dismissing the person who will be
             | explaining it.
        
         | karmakaze wrote:
         | Yes exactly. The article would be much better positioned as
         | What it actually takes to make systems ideas "just work". Then
         | it isn't about whether it can or can't but rather how it
         | succeeds or fails. Having details about what it takes then
         | stops people from saying "just" because they don't know what
         | they're talking about.
         | 
         | But I pretty much agree 100% about DSLs, it's an
         | unnecessary/cute complication. The only people who should be
         | allowed to make them should be ones who have made a successful
         | programming language and updated it to deal with all their
         | mistakes or a 2nd version/2nd language, and still got many
         | things wrong.
        
         | fosk wrote:
         | Furthermore, API is a product that needs its own lifecycle to
         | be versioned, decommissioned, continuously improved.
         | 
         | Often I see developers creating new APIs ad-hoc all the time
         | instead of curating and enhancing the one they already have.
        
       | insane_dreamer wrote:
       | "seldom work as expected" would have been more accurate than
       | "almost never work"
       | 
       | I did like this:
       | 
       | > More importantly, an offering an API doesn't mean anyone wants
       | to use it. Almost every new API comes up because the co/product
       | wants features, but it doesn't want to prioritize them enough and
       | the theory is the API will be "evangelized" to some partner in
       | the space. Turns out those people are not sitting around waiting
       | to fill in holes in your product.
        
       | stevebmark wrote:
       | I would add "Domain Driven Design" - locking your business design
       | in place by trying to make your application match your business
       | structure is a recipe for disaster. If you have a small or
       | stagnant business you probably won't notice any issues. If your
       | business is successful and/or grows, you're going to immediately
       | regret trying to build domains with horrific descriptive names
       | tied to your already obsolete business practices. Instead, design
       | around functionality layers (how we've been doing it for decades,
       | tried and true), and as much as possible keep business logic in
       | config, rows in databases, and user workflows, which makes them
       | extremely flexible.
        
         | jkaptur wrote:
         | Along with "make impossible states impossible to represent". If
         | you're designing your types to make a state unrepresentable,
         | you'd better be absolutely sure the state really _is_
         | impossible for the lifetime of the design.
        
           | spencerflem wrote:
           | If its not impossible, you should handle it though.
           | 
           | I think what that quote is against is the common middle
           | ground where states are expected to be 'impossible' and thus
           | not handled and cause bugs when they are found to be not
           | actually.
           | 
           | Either deciding that they are possible or must be impossible
           | is usually better and which one to go with depends on the
           | specifics
        
           | pvillano wrote:
           | The variant of this I've found useful is to have a separate
           | types for raw/dirty and parsed/validated data
        
         | eddythompson80 wrote:
         | You'll regret both options. You mentioned the pitfalls of
         | "domain driven design" (outdated language, little code/systems
         | reuse for new endeavors).
         | 
         | However, a highly abstract design with all the business logic
         | in config, workflows, etc will only makes your system extremely
         | flexible as long every one up and down the organization is
         | fairly aware of the abstractions, the config, and uncountable
         | permutations they can take for your business logic to emerge.
         | 
         | Those permutations quickly explode into a labyrinth of
         | unknown/unexpected behaviors what people will rely on. It also
         | makes the cost of onboarding new developers, changing the
         | development team insurmountable. Your organization will be
         | speaking 2 different languages. Most seemingly straightforward
         | "feature asks" that break your abstraction either become a
         | massive system re-design/re-architect or a "let's just hack
         | this abstraction so it's a safer smaller change for now". The
         | former will always be really hard unless you have excellent
         | engineers who have full understanding of the entire system and
         | its behavior and code base along with and excellent engineering
         | practices and processes, and still will take you months or
         | years to pull off. The latter is the more likely to happen and
         | it's why all those "highly abstract, functionality layers,
         | config driven, business logic emerging) projects start perfect
         | and flexible and end up as a "what the fuck is even this".
         | 
         | After a system is implemented, that emergent business logic
         | becomes the language everyone will speak in. Having your
         | organization speaking 2 or 3 completely irreconcilable
         | languages is very painful and unless you have multiple folks
         | up, down and sideways in the organization that can fluently
         | translate between the 2, you'll be in a world of pain and wish
         | you had some closer representation of your domain
        
       | tinthedev wrote:
       | The whole post, and both the Twitter messages sound like they
       | refer to a very specific style of work.
       | 
       | These things "sound good but almost never work" when they're
       | taken as an afterthought or committed to without due
       | research/process/design.
       | 
       | Any architect worth their salt will avoid implementing hard
       | solutions to these problems. These are mostly here to not solve
       | crucial issues... but the article seems to be addressing a
       | developer profile that adds APIs or asynchronous processing
       | casually.
       | 
       | Sounds like a strawman, or just exceedingly pessimistic look at
       | the industry.
        
       | riwsky wrote:
       | So many people here trying to thread the needle looking for
       | subtle decision functions for exceptions. It's pretty simple,
       | really: these ideas are awesome when I do them, and never work as
       | intended when that idiot before me did them.
        
         | plagiarist wrote:
         | That sounds exactly right. And sometimes the idiot before me
         | was myself from several months ago.
        
       | NBJack wrote:
       | I'm wondering about the first example of device drivers. Whether
       | we are talking about Linux or Windows, anything beyond generic
       | device drivers (which push the problem to the hardware) will
       | frequently be vendor specific. You often can't get the full
       | potential of a device until you do so.
        
       | ww520 wrote:
       | Haha. This is a great list. I have heard variants of these ideas
       | over the years.
        
       | devjab wrote:
       | Around here people abstract because they are taught to abstract
       | in multiple different CS education programs. For whatever reason
       | they are taught Clean Architecture even though it's horrible.
       | Then they spend the next 5-10 years learning not to abstract or
       | write 5 line functions which then lead down a chain of 900
       | function calls. As an external examiner or some or these CS
       | students it always pains me to examine them in a curriculum that
       | I know will lead to dead end job in stagnant small sized
       | companies. It is what it is though. I guess it's hard to compete
       | with the entire industry of pseudo-science and shitty consultancy
       | when nobody is pushing an "indirection manifesto".
       | 
       | Hilariously Uncle Bob will write off any criticism as "they
       | misunderstood the principles". He's correct too, but maybe the
       | principles are simply too vague when we've had them for 20 years
       | and our industry has never been more of a mess.
        
         | bedobi wrote:
         | A lot of Robert C Martins pieces are just variations on his
         | strong belief that ill-defined concepts like "craftsmanship"
         | and "clean code" (which are basically just whatever his
         | opinions are on any given day) is how to reduce defects and
         | increase quality, not built-in safety and better tools, and if
         | you think built-in safety and better tools are desirable,
         | you're not a Real Programmer (tm).
         | 
         | I'm not the only one who is skeptical of this toxic, holier-
         | than-thou and dangerous attitude.
         | 
         | Removing braces from if statements is a great example of
         | another dangerous thing he advocates for no justifiable reason
         | 
         | https://softwareengineering.stackexchange.com/questions/3202...
         | 
         | Which caused the big OSX/iOS SSL bug in 2014, see
         | https://www.imperialviolet.org/2014/02/22/applebug.html
         | 
         | This link and thread on hackernews is good too
         | 
         | https://news.ycombinator.com/item?id=15440848
         | The current state of software safety discussion resembles the
         | state of medical safety discussion 2, 3 decades ago (yeah,
         | software is really really behind time).              Back then,
         | too, the thoughts on medical safety also were divided into 2
         | schools: the professionalism and the process oriented. The
         | former school argues more or less what Uncle Bob argues: blame
         | the damned and * who made the mistakes; be more careful, damn
         | it.              But of course, that stupidity fell out of
         | favor. After all, when mistakes kill, people are serious about
         | it. After a while, serious people realize that blaming and
         | clamoring for care backfires big time. That's when they
         | applied, you know, science and statistic to safety.
         | So, tools are upgraded: better color coded medicine boxes, for
         | example, or checklists in surgery. But it's more. They figured
         | out what trainings and processes provide high impacts and do
         | them rigorously. Nurses are taught (I am not kidding you) how
         | to question doctors when weird things happen; identity
         | verification (ever notice why nurses ask your birthday like a
         | thousand times a day?) got extremely serious; etc.
         | My take: give it a few more years, and software, too, probably
         | will follow the same path. We needs more data, though.
        
           | linhns wrote:
           | I don't know if he considers the expiration date of his
           | ideas. Some of his ideas actually creates worse software and
           | crater projects.
        
             | devjab wrote:
             | I'm not sure any of them do really. It's been 22 years
             | since TDD made its entry into our field and it's still
             | worse than the runtime assertions which helped put people
             | on the moon. I know I was lashing out at uncle Bob before
             | but it's really all of them.
             | 
             | I do agree with these people that nobody has ever regretted
             | writing a test. Well, I mean, someone probably has, but the
             | idea of it is fairly solid. It's just also useless, because
             | it's so vague. You can write a lot of tests and never be
             | safe at runtime.
        
               | hitchstory wrote:
               | There isnt much consensus on the right kind of test to
               | write with TDD, but when you get it right or wrong it
               | makes or breaks TDD.
               | 
               | Recently Ive been writing mostly "end-to-end unit tests"
               | - stateless, faking all external services (database,
               | message queue, etc.) with TDD which works great.
               | 
               | There is a sweet spot on default test types - at a high a
               | level as possible while being hermetic seems to be ideal.
               | 
               | The other un-talked about thing is that to be able to
               | always write this kind of test you need test
               | infrastructure which isnt cheap to build (all those
               | fakes).
        
       | bhouston wrote:
       | I've executed a bunch of these ideas successfully. So it reads a
       | little weird.
        
         | mdavid626 wrote:
         | Either you really know what you are doing, or, you really
         | don't.
        
       | Xcelerate wrote:
       | IDK about anomaly detection being on that list. When I worked at
       | a large tech company, the in-house anomaly detection and root
       | cause analysis capabilities worked like magic compared to other
       | places I've been. When done right, it can be extremely valuable.
        
         | kitd wrote:
         | Sounds like you hit the 1/10. Which is great and I agree, very
         | rewarding.
        
         | jldugger wrote:
         | Half of the magic is knowing _when_ to apply it. If you just
         | dump your prometheus timeseries data into an anomaly detection
         | system and ask it to constantly scan for anomalies, you will
         | always find them.
         | 
         | As an SRE, I don't actually care about anomalies all that much.
         | For initial alerting, I want phase shift detection. One
         | customer sending a few bad API calls on one specific minute is
         | uninteresting and pretty much inactionable. That same error
         | rate over 10 minutes is more interesting and more likely to be
         | a systems problem I can actually resolve. But the raw AD stream
         | is just too damn noisy for all sorts of reasons. This is why
         | our alerting tools have a `duration` field: any signal above
         | threshold must remain so for multiple observation periods
         | before summoning human inspection. And why health checks have
         | grace periods and retries before killing services.
         | 
         | Where anomaly detection works better, IMO, is post-alert
         | analysis. At that point anomalies are welcomed as hypotheses,
         | since the system features complex interactions between
         | components. I've built a couple of dashboards using extremely
         | simple math, like Laplace smoothing and time series correlation
         | that help surface relevant information from the flood of
         | metrics and logs collected. But critically, these tools
         | generally don't use the time domain as their baseline. Usually,
         | I'm comparing a cluster against another one in a different
         | region, or one metric against another, rather than now versus
         | twelve hours ago.
        
       | bcrosby95 wrote:
       | Unless you're a large corporation: fighting Conway's Law.
        
         | hot_gril wrote:
         | Why "unless" and not "especially when"? :)
        
       | slooonz wrote:
       | "Let's just cache this slow call"
        
       | nottorp wrote:
       | > Let's make it cross-platform.
       | 
       | So if making it cross platform "won't work", this is not about
       | what won't work, but what is cheaper?
       | 
       | Since the OP kinda mentions gaming, let the customers fiddle with
       | their wine installs and steam decks and spend the time on adding
       | more loot boxes instead?
        
         | IshKebab wrote:
         | Yeah I don't buy that at all. He says "9 times out of 10" and
         | then gives MS Office as the proof? MS Office is the 10th time
         | out of 10, not the 1-9.
         | 
         | I think they absolutely _could_ have made Office cross platform
         | (it doesn 't really do anything you can't do in Qt), and also
         | the fact that they forked it and made two entirely separate
         | sets of apps has pretty serious consequences. The feature sets
         | are surprisingly different. E.g. you can add PDFs to documents
         | on Mac; not on Windows.
         | 
         | Most software can be made cross platform (as long as you don't
         | explicitly prevent it by using a platform specific GUI toolkit
         | or whatever).
        
       | ghjfrdghibt wrote:
       | The "let's make it cross platform" is what lead me to dart and
       | flutter. These aren't mentioned and for the developers of them is
       | definitely a hard problem. But as far as I'm concerned they're
       | doing a bang up job.
       | 
       | I agree with sync though. Hard problem with no simple solution
       | for an idiot like me to do.
        
       | CyberDildonics wrote:
       | A lot of times what people want from a DSL is really just
       | something that doesn't have to compile and/or something to run in
       | a VM. Sometimes people just want a text format for data that
       | could be done with json.
       | 
       | Making a new language throws away everything and starts over
       | including debugging, tools, syntax checking, auto completion,
       | documentation etc. It is take way too lightly and just becomes a
       | hassle.
       | 
       | You can see this in multiple GUI libraries, where getting the
       | parameters to set up a GUI is really not difficult and is just
       | data through function calls, but it gets made into a separate XML
       | like DSL markup language with it's own quirks and opacity, and
       | that XML is for most people being given using a big string from
       | within the language that they're using.
       | 
       | This stuff persists because it sounds easier on paper and it just
       | creates more problems in practice but it takes experience to
       | realize all the you're losing. That's where designers need to be
       | experienced and do what works instead of what will suck in people
       | that don't know any better.
        
         | adzm wrote:
         | With how easy it is nowadays to embed Lua, and even JavaScript
         | with just a little bit more effort, it's pretty inexcusable to
         | develop your own language for a tiny space. A great example of
         | this is Adobe embracing JavaScript for it's own extension and
         | scripting and expression language, which has allowed it to
         | benefit from an entire ecosystem of JavaScript libraries.
        
       | throwaway63467 wrote:
       | Hm I can think of several instances for each point where the
       | ideas work just great. It's just difficult getting complex stuff
       | to work in general.
        
       | quotemstr wrote:
       | The ideas in the first tweet screenshot: uh, they _do_ sound good
       | and they _do_ work. Your system is full of control loops. (What
       | do you think regulates how many dirty pages the kernel writes
       | back to disk?)
        
       | adzm wrote:
       | > Windows NT is riddled with excess abstractions that were never
       | really used primarily because they were there from the start
       | before there was a real plan to use them.
       | 
       | It is? Like what? I know there are some abstractions for cpu
       | architecture etc but they've come in handy for x64 and now arm
       | and others in the past.
       | 
       | I know the author certainly has some insight into this but I've
       | never really thought of NT as being riddled with excess
       | abstractions.
        
         | senderista wrote:
         | It's possible that he's referring to purely internal
         | interfaces?
        
       | liontwist wrote:
       | > Most of the first 25 years of computer science was figuring out
       | how to make things work asynchronously
       | 
       | This is not true. Computers focused on single threaded designs
       | before getting thrown into parallelism (need parallelism to run
       | into dining philosopher problems).
        
         | mrkeen wrote:
         | I'm not sure what counts as the "first 25 years", but Dining
         | Philosophers is from 1965.
        
       | liontwist wrote:
       | Anyone have any experience with the control loop one?
       | 
       | My uninformed guess is CS people just underestimate the skills
       | and experience to analyze feedback systems and so write it off as
       | a bad technology after a poor implementation.
       | 
       | Maybe the real problem is you want to know when your system is
       | maxing out the range you anticipated?
        
         | avidiax wrote:
         | Control theory is an entire discipline.
         | 
         | One problem that constantly comes up is ratcheting or poisoning
         | your own inputs.
         | 
         | Let's say you want to block "noisy neighbors" from taking large
         | amounts of resources, while allowing all loads to have bursty
         | use of the full system power. Easy, right? Detect the noisy
         | neighbors and throttle them to a small percentage. Unthrottle
         | when they show some substantial idle time. But now, many of
         | those noisy neighbors can't get out of jail because they will,
         | of course, use nearly 100% of their restricted load, even if
         | they would now be well-behaved and merely bursty.
         | 
         | There is also a cascading effect, where you have N related
         | bursty loads. One bursts for too long, gets throttled, and now
         | the load is handled by the remaining N-1 loads. But that makes
         | those loads more likely to get throttled, and so on. Only
         | unthrottling all the loads simultaneously will allow them to
         | return to normal bursty operation.
        
           | bobnamob wrote:
           | The fun thing about control theory in CS is that (in my
           | experience) it's far easier to build accurate simulations or
           | even run scaled down experiments than it would be in a
           | physical setting.
           | 
           | See articles like [1] or any of Marc Brooker's [2] blog for
           | inspiration
           | 
           | [1] https://medium.com/yandex/good-retry-bad-retry-an-
           | incident-s...
           | 
           | [2] https://brooker.co.za/blog/2022/04/11/simulation.html
        
       | k__ wrote:
       | Most of these things work pretty well.
       | 
       | It's just that they don't "just" work.
       | 
       | Just because AWS or Google can pull it off doesn't mean it's
       | something anyone can do.
        
         | sgarland wrote:
         | THIS. And adding to it, most people really do not like being
         | told that they aren't as clever as AWS or Google.
        
         | jcims wrote:
         | Yep, I think the title should be "things that are way harder
         | than they seem".
        
       | asveikau wrote:
       | Just remember that this is the genius who brought Windows 8 and
       | the Windows RT tablet that wouldn't run recompiled Win32 apps.
        
         | plagiarist wrote:
         | Maybe some experience making large mistakes is how they came up
         | with the list?
        
           | asveikau wrote:
           | Judging by some of the list I don't think he's learned the
           | right lessons.
        
       | rogerthis wrote:
       | Event sourcing.
       | 
       | PS: on mobile, can't expand (issues around versioning, data
       | migration)
        
         | klysm wrote:
         | Incredibly hard to make work in practice because of the
         | semantic muddiness that results from having events about events
         | etc. You have to think about the domain and its first
         | derivative
        
       | mikewarot wrote:
       | >Let's just add access controls later.
       | 
       | This (Computer Security) is a _solved_ problem, kids. One of the
       | lessons learned in the Viet Nam conflict was the need for a
       | computer system that could safely handle multiple levels of
       | classified data. The solution was Multilevel security, included
       | in that was the Bell-LaPadula model. Several actually secure OSs
       | emerged over the decades since, including KeyKOS, CapROS and
       | Eros.
       | 
       | I'm hoping that someday I can make Genode, the latest capability
       | based operating system, my daily driver, so I never have to worry
       | about virus scanners again.
        
         | nostrademons wrote:
         | Computer security is a solved problem _for systems with a
         | fully-specified set of capabilities, usage patterns, and legal
         | operations_. You follow the Principle of Least Privilege, build
         | in defense in depth, use capabilities, and never expose
         | anything that the user doesn 't have a reason to access.
         | 
         | The problem is that the constraints this imposes on the system
         | usually do not line up with the constraints that the market
         | will pay for. It's very common for customers to change their
         | mind; decide they need to hack around access protections; add
         | new users with new roles that are some hybrid of current
         | access; ask for new features; not think through who should have
         | access to new features; want to enable serendipity where
         | untrusted users discover new use-cases and new markets for
         | their product; and so on. It's also very common for them to
         | ignore security as a differentiator when making their purchase
         | decision, figuring that if there's a breach, somebody else will
         | pay for it, or they'll be long gone from the company and unable
         | to be blamed for it. So the market ends up bypassing the secure
         | solutions that exist and choosing to buy insecure systems that
         | can offer the features they want right now.
         | 
         | When security is absolutely critical, like in military or
         | certain financial applications, it's pretty easy to achieve.
         | There are companies like Galois that specialize in "high
         | assurance systems". But they are expensive for their feature
         | set, and so the general public would rather buy from cheaper
         | and more insecure options.
        
           | mikewarot wrote:
           | Eventually we're going to have to collectively decide that
           | the operating system is the correct place to enforce
           | capabilities, as is done in mainframe OSs.
           | 
           | Memory holing any mention of this _solution_ isn 't
           | productive in the long run.
        
             | nostrademons wrote:
             | Yes, the operating system is the correct place to enforce
             | capabilities.
             | 
             | The problem with this is that no mainstream OS does this
             | correctly, which means that correctly doing security
             | requires writing a new OS and getting all the userspace
             | programs ported over to it (which is a non-trivial port,
             | because the programming model for capabilities is pretty
             | significantly different from mainstream OSes). It's very
             | hard to convince users to ditch their entire computing
             | ecosystem for a new one unless all of their devices get
             | pwned and they can't access their computing ecosystem
             | anyway.
        
               | mikewarot wrote:
               | I'm convinced the way this will be done is to take a
               | capabilities based OS, and tack on an emulation layer to
               | allow Windows or Linux binaries to run, and only let them
               | see the things that the user has decided they need to
               | see, by emulating the dialog boxes to the app, and then
               | transparently enforcing those choices. Thus a copy of a
               | windows Text editor could run, and ONLY get access to the
               | file the user chooses, without having to re-write
               | anything.
               | 
               | The crux of the issue is command line programs... I'm not
               | sure how to deal with those, but I suspect it'll be an
               | outer job control language.
        
         | wpollock wrote:
         | You're right about Bell-LaPadula, but note that model isn't
         | used as much as the Biba model, the basis of Windows UAP. Even
         | "solved" problems tend to evolve or get superceded over time;
         | nothing seems to stay solved for long!
        
           | mikewarot wrote:
           | Windows UAP is a horrible thing. It would be far better to
           | just replace the system dialog boxes like file open, save,
           | etc. with power boxes that then give file capabilities to
           | applications.
           | 
           | It would be minimal work to refactor applications, and
           | provide almost perfect security with no UX change.
        
         | mdavid626 wrote:
         | Hmm? As the joke goes, the only secure computer is the one
         | which is turned off. But I guess, that's not even true nowadays
         | (taking security keys from cooled memory modules).
        
       | exabrial wrote:
       | I thought live migrating process state did work...? we have
       | containers that can freeze/resume.
       | 
       | Granted, you have to freeze the whole damn OS image, but it
       | _does_ seem to work.
       | 
       | All of the others I agree, I've seen them tried so many times in
       | my lifetime, from mainframes to serverless, and nobody gets them
       | right.
        
         | thraxil wrote:
         | Yeah, and I was live migrating entire VMs between hosts on a
         | Xen cluster 10-15 years ago. It was easy for me to get working,
         | reliable, and I was far from an expert.
        
       | xyzsparetimexyz wrote:
       | I've been writing glsl recently, and I think it's a good example
       | of a DSL that _doesn't__ work. It should have just been C without
       | the standard library, or ideally C++. Not being able to use
       | pointers is especially painful. Hlsl is better and has some nice
       | quality of life features. Rust-gpu is the best approach but
       | basically nobody uses it
        
       | nostrademons wrote:
       | There is a pattern to all of these problems, notably that they
       | are all _orthogonal concerns that add constraints_ to the
       | sequential data-munging programming model that programmers are
       | familiar with. Whenever you add constraints, you add things that
       | future programmers need to think about, for _all future
       | development they do on the system_. It is very easy to get into a
       | situation where the system is overconstrained and it 's
       | impossible to make forward progress without relaxing some of the
       | constraints. Even if it's not impossible, it's going to be _slow_
       | , as developers need to consider how their new feature interacts
       | with the API/security/synchronization/latency/other-
       | platforms/native-code that the system has already committed to
       | supporting.
       | 
       | That's also why it's _possible_ to support all of these
       | attributes. If you make say transparent data synchronization a
       | core value prop of the platform, then all future development
       | supports that _first_ , and you evolve your feature set based on
       | what's possible with that constraint. That feature set might not
       | be exactly what your users want, but it's what you support. Your
       | product appeals to the customers for whom that is their #1
       | purchase decision.
        
       | qianli_cs wrote:
       | Or let's just reinvent a new database
        
         | Aloha wrote:
         | Or worse, lets reinvent the database, but not call it or think
         | of it as a database.
         | 
         | Or even worse yet, lets reinvent the filesystem then host it on
         | top of a filesystem.
        
           | hot_gril wrote:
           | I've heard "it's not a database" multiple times, in reference
           | to whatever custom DB our team inherited. Terrifying.
        
       | ibejoeb wrote:
       | "Let's just sync the data" is the reason why rough days exist, as
       | far as I'm concerned. I've run into so many systems that were
       | designed for "internet scale" or whatever that add queues, event
       | processing, etc., when the natural range is so far below that
       | threshold. These teams are either naive or, at worst, taking
       | advantage of non-engineering management and funneling money
       | toward playing with these problems for the fun of it.
        
         | mrkeen wrote:
         | I read "just sync the data" as naively doing reads over here
         | and writes over there, and hoping that the two sources of truth
         | don't diverge.
         | 
         | Queues and event processing are a necessity to do it right.
        
           | ibejoeb wrote:
           | That's sort of the point. The operative word is "just." In
           | reality, it adds a huge subsystem that needs more resources
           | than the overall system does. Rather than solve the
           | underlying problem directly, they introduce (more
           | interesting) problems to solve transitively via these
           | complexities.
        
             | mrkeen wrote:
             | I don't know of any way to replicate data ("solve the
             | underlying problem") aside from a replicated log.
             | 
             | It's like git solving the problem without commits, or
             | banking solving the problem without transactions.
        
               | ibejoeb wrote:
               | Ah, ok, I'm failing to communicate my point. The
               | underlying problem in this scenario is something basic
               | which doesn't require replication/sync at all. I'll use
               | an actual example that fell on me a while back. There was
               | an app that displayed a news feed. The theoretical upper
               | limit of the volume, due to the size of audience, was
               | hundreds of posts per day, while the practical number was
               | dozens. The architecture was:
               | 
               | * postgresql as the system of record
               | 
               | * firestore as the upstream source for clients
               | 
               | * ES for full-text search
               | 
               | * client-side store for actual client-side reads
               | 
               | * http api for mutations
               | 
               | That required three sync systems: pg->firestore, pg->ES,
               | and firestore->local store. Then it needed messages for
               | the async mutations to propagate back to the clients. And
               | then these things require more things to make them work,
               | like data transformers to support the three different
               | formats for each stage.
               | 
               | This certainly did not require some giant CQRS system and
               | could have been built entirely on postgres. It was a
               | fractal of code that didn't go toward the actual
               | objective.
        
       | Brian_K_White wrote:
       | I don't understand the load-responsive control loop one. That's a
       | basic and fundamental component in countless systems. The
       | centrifugal governor on a 1800's steam engine or 1900's victrola
       | record player is a load-responsive control loop. All of
       | electronics is a mesh of load-responsive control loops. The
       | automatic transmission in your car...
        
         | panic wrote:
         | It's not totally clear, but it could be talking about CPU load
         | in particular, which has some problems as described in
         | https://arxiv.org/abs/2312.10172.
        
           | Spivak wrote:
           | I've always used connection backlog as the metric for load
           | and it's worked pretty well. Most web servers have it as a
           | number you can expose as a metric. It's not perfect but it's
           | at least a true measure of when servers are behind.
        
         | lclarkmichalek wrote:
         | The usual issue is the addition of control loops without much
         | understanding of the signals (CPU utilization is a fun one),
         | and the addition of control loops without the consideration of
         | other control loops. For example, you might find that your
         | cross region load balancer gets into a fight with your in-
         | process load shedding, because the load balancer's signals do
         | not account for load shedding (or the way they account for the
         | load shedding is inaccurate). Other issues might be the
         | addition of control loops to optimize service local outcomes,
         | to the detriment of global outcomes.
         | 
         | My general take is that you want relatively few control loops,
         | in positions of high leverage.
        
       | LAC-Tech wrote:
       | There are equations/properties for data that syncs nicely and
       | allows mutli-master writes. Ie, if your merges follow a few
       | simple properties, this stuff will work.
       | 
       | These are basically what CRDTs are. And I know what you're
       | thinking; "but I'm not writing a mutli user text editor!" or
       | "automerge won't scale!". But CRDTs aren't a library, they're a
       | set of properties. if your whole data system obeys the properties
       | - you have a CRDT.
        
         | klysm wrote:
         | I'm not really sure what you're saying. Do you mean you achieve
         | the gaurantees of CRDTs without using CRDTs? What technology
         | are you implicitly referring to?
         | 
         | Whenever I hear someone say "sync" data, I instantly get
         | scared. Consensus is fraught with peril and very very very
         | difficult to implement correctly.
        
           | LAC-Tech wrote:
           | It's more shifting the view point. CRDTs are not just
           | something you 'use'. They are laws a system must be obey to
           | achieve _strong eventual consistency_.
           | 
           | Eventual Consistency: If Node A and Node B have received the
           | same set of events (ie in any order), they will _eventually_
           | have the same state.
           | 
           | Strong eventual consistency: Exactly the same, but replace
           | "eventually" with "immediately".
           | 
           | Meaning as long as all nodes get the same events, they'll
           | have the same state, straight away. That's sync that works.
           | 
           | As long as your merge algorithm is commutative, associative,
           | and atomic, your sync will work. That's what the CRDT people
           | uncovered.
           | 
           | A data structure can obey these laws (the aforementioned CRDT
           | libraries)
           | 
           | A database can obey these laws (the original Amazon Dynamo
           | did this, it was a CRDT).
           | 
           | Any arbitrary system can obey these laws (there's 1982 paper
           | described this for a distributed file system)
        
             | klysm wrote:
             | Okay I'm with you, but if I'm not mistaken I believe you
             | have slightly mis-characterized what strong eventual
             | consistency is.
             | 
             | > Strong eventual consistency: Exactly the same, but
             | replace "eventually" with "immediately".
             | 
             | I believe this isn't quite correct. I was under the
             | impression that the delivery doesn't have to be immediate,
             | but rather that any two nodes with the same set of events,
             | regardless of received order, must be in the same state.
        
               | LAC-Tech wrote:
               | https://inria.hal.science/hal-00932836/file/CRDTs_SSS-201
               | 1.p...
               | 
               | I'm taking my definitions from section 2.2 here. I feel
               | like I've summarised it fairly accurately, but if I've
               | made a mistake would be happy to be corrected.
        
       | poooka wrote:
       | Disagree with plugin talk. I use plugin system that loads from
       | submodules so I can have same architecture in multiple airgaped
       | networks. Can have implementations load (for that environment)
       | outside of main code base. Nothing wrong w that given
       | complexities of maintaining complex ass configs if everything was
       | shoved into the same artifact.
        
       | fancyfredbot wrote:
       | I've worked on several DSLs, a P2P cache, and a project employing
       | hybrid parallelism. All of them worked. All great fun to create.
       | With one exception these projects were good investments (The P2P
       | cache wasn't necessary so never really paid off). My point is
       | that it's definitely wrong to say these things almost never work.
       | They are complicated but that complexity brings functionality
       | which is hard to achieve in another way. The lesson from the P2P
       | cache example was to be sure you actually need that
       | functionality.
        
       | mlhpdx wrote:
       | Everything on the list I've seen done (been hands-on with) very
       | successfully, most of them repeatedly. I've also seen many, many
       | more failed (and often pointless) attempts. I don't have any
       | great answer to why other than the importance (practical value to
       | customers) of it being clear, well understood by leaders and
       | doers, and high.
       | 
       | If having an API is core to the value customers will realize then
       | it's likely a good API will emerge. Etc.
        
       | mjaseem wrote:
       | I feel "Let's just" is doing much of the work here. You can add
       | the phrase in front of any idea and make it sound awful:
       | 
       | "Let's just make planes fly themselves"
       | 
       | "Let's just put a giant battery in a car instead of an engine"
       | 
       | "Let's just make electricity from wind"
       | 
       | "Let's just be friends"
       | 
       | I would love to see a list like this without such a big asterisk.
        
         | earnestinger wrote:
         | Let's just make a list without asterisk.
        
       | levzettelin wrote:
       | Lost me at "DSLs".
        
       | retrocryptid wrote:
       | I think each of the ideas listed: dsls, control loops, being
       | clever, etc. They all have their place, and I wouldn't say they
       | never work. I've had each of these work well in specific
       | situations.
       | 
       | But the OP does have a point, they each _can_ introduce more
       | trouble then they 're worth. Were I to write this post, I would
       | have titled it something more like "Systems Ideas Yo Really
       | Should Think About Long And Hard Before Doing."
       | 
       | But yeah, that might not be enough warning.
        
       | Sparkyte wrote:
       | All rely heavily on one aspect, consistency. If you system
       | engineering can consistently operate without change, innovation
       | or issue, then it is possible that these ideas could exist in a
       | closed environment. However they just don't work when dealing
       | with external factors.
        
       | debarshri wrote:
       | I would like to add building a PaaS/ internal tools that abstract
       | cloud for productivity sake. Bad idea.
        
       | banq wrote:
       | Why do people keep talking about it? Is it because they've fallen
       | into a dialectical trap? No, it's because they lack the awareness
       | that "Context is King."
        
       | dools wrote:
       | An amazing example of a DSL was Butler for Trello before it was
       | acquired and dumbed down to a "no code" tool. It was magical to
       | use. I had started consulting using it but they said I wasn't
       | allowed to use it anymore so I replicated the functionality as a
       | library to run inside a Google sheet using Google apps script.
        
       | ChrisMarshallNY wrote:
       | Happy 2025, folks!
       | 
       | My (recent) experience:
       | 
       |  _> Let 's make that asynchronous_
       | 
       | It can be done, but here, there be dragonnes.
       | 
       | I just had to make a formerly synchronous load in a recently
       | released app, into an async one, because users with large
       | connection lists (think "friends," in Facebook, but not as
       | "friendly"), were having extremely slow loads.
       | 
       | This was a _big_ change.
       | 
       | First, I had to swap out an entire SDK that accessed the most
       | important server in the app, because the old one didn't play
       | well, with threads (my bad). That actually went fairly smoothly,
       | because of the abstraction _(boo hiss, I guess?)_ that I had used
       | for the SDK. Took about a day, to have the operation running
       | smoothly.
       | 
       | Testing...Testing...Testing...
       | 
       | Next, I had to _test like crazy, for weeks_ , on the user-level
       | code, because the new threading brought the beast out in that
       | code. I found _all kinds_ of places, where I had written thread-
       | unstudly code. None of the issues were serious, and many folks
       | would have said  "Fuck it. Let's ship," but I'm a bit anal about
       | certain things, and I'm not being paid, anyway...
       | 
       | In the end, the conversion was a success (not one complaint --
       | _fingers crossed_ ), and we got the results we needed, but it
       | took a _hell_ of a lot of testing (especially monkey testing),
       | and Release Day was a nervous one. The UI basically didn 't
       | change at all (except for some loading throbbers on the profile
       | avatars), but under the hood, a lot had changed.
       | 
       | Sometimes, it needs to be done, but it's tempting to make it seem
       | easy (which many folks will). I am a rather scarred veteran of
       | "That should be easy," so I went in, with eyes open, and (as
       | noted) had already prepared, with a certain level of abstraction.
       | I figured that the SDK swap would be [relatively] easy, but
       | didn't bargain for all the little bugs, in the code I thought was
       | already sorted.
        
       | cesaref wrote:
       | Let's consider the original list. It starts with DSLs. Are we
       | saying that using SQL 'almost never works'? Shader languages?
       | BPF? If I squint a bit i'd say that all the inference engines are
       | either DSLs or APIs which wrap an underlying DSL, so TensorFlow
       | etc.
       | 
       | I could probably come up with similar examples from the rest of
       | the list. If the message is 'system ideas which seem simple but
       | are hard, so you should only use existing off the shelf examples
       | of' then i'd certainly have more sympathy for the statement.
        
       | ChrisMarshallNY wrote:
       | The Programmer's Credo:
       | 
       |  _We do what we do. Not because it is easy; but because we
       | thought it would be easy._
        
       | Spiwux wrote:
       | I've implemented several of these ideas, and they worked just
       | fine.
        
       | Guthur wrote:
       | I don't necessarily think the ideas themselves are wrong but
       | rather the devil is in the details.
       | 
       | A good example is the DSL one. In reality every significantly
       | complex software system is essentially a DSL, it will have it's
       | own collection of nouns and verbs that constitute the application
       | domain, this is a language and rarely is it understandable to
       | anyone without the domain knowledge.
       | 
       | The problems often arise when custom denotational semantics are
       | added that hinder composition. This is overcome by 'not using a
       | DSL' and instead just expose the semantics of your implementation
       | language, but essentially an API is little different from a DSL
       | just with all the baggage and coupling to the underlying
       | implementation details.
        
       ___________________________________________________________________
       (page generated 2025-01-01 23:01 UTC)