[HN Gopher] Systems ideas that sound good but almost never work
___________________________________________________________________
Systems ideas that sound good but almost never work
Author : kaycebasques
Score : 224 points
Date : 2024-12-31 16:47 UTC (6 hours ago)
(HTM) web link (hardcoresoftware.learningbyshipping.com)
(TXT) w3m dump (hardcoresoftware.learningbyshipping.com)
| PaulHoule wrote:
| (1) DSLs work great sometimes. See https://www.jooq.org/
|
| (2) Elastic Load Balancer is a control loop responsive to
| workloads, that kind of thing is a commodity
|
| (3) Under-provisioning is rampant in most industries; see
| https://erikbern.com/2018/03/27/waiting-time-load-factor-and...
| and https://www.amazon.com/Goal-Process-Ongoing-
| Improvement/dp/0...
|
| (4) Anomaly detection is not inherently a problem of distributed
| systems like the others, but someone facing the problems they've
| been burned with might think they need it. Intellectually it's
| tough. The first algorithm I saw that felt halfway smart was
| https://scikit-learn.org/1.5/modules/outlier_detection.html#...
| which is sometimes a miracle and I had good luck using it on text
| with the CNN-based embeddings we had in 2018 but none at all w/
| SBERT.
| billyp-rva wrote:
| > (1) DSLs work great sometimes.
|
| I'll take a stab at fleshing this out: DSLs work great when
| they have an IDE with autocomplete and a quick (or instant)
| feedback loop.
| diggan wrote:
| I was gonna say something like "DSLs work great when they're
| small, purposeful and easy to test", I guess yours kind of
| helps when they're not what I'd suggest :)
| PaulHoule wrote:
| Maybe his problem is with those yucky distributed systems
| as in Kube, Antilles, etc. I write plain ordinary Java
| programs that work with _a_ cloud API and compile bash
| scripts that colonize machines.
| verdverm wrote:
| I think of Kube as more of abstractions whereas Wasp &
| Dark lang would be DSLs for the same concepts
| tinthedev wrote:
| This whole thread said exactly what I wanted to write.
| Feels bad to be so pre-empted.
|
| DSLs that solve a specific problem with a page or two of
| documentation overhead are great.
|
| Trying to reinvent paradigms or scope creep is where the
| pain comes in. Seems like the post author has been burned
| by that type of DSLs.
| f1shy wrote:
| I would say that is the line between DSL and just "L"
| another language...
| earnestinger wrote:
| > DSLs that solve a specific problem with a page or two
| of documentation overhead are great.
|
| Do you have any example? I've heard lots of good things
| of dsl, but never had the luck to witness it's full
| glory.
|
| (except for regex, which I love, but it has more than two
| pages of docs)
| f1shy wrote:
| So any DSL as an embedded language inside another.
| pizlonator wrote:
| I think you're responding to the tweet from Martin that Steven
| includes at the top, not to Steven's list.
| kelnos wrote:
| That's fine. The points made in the tweet are interesting to
| discuss too.
| echelon wrote:
| The tweet has some claims that don't fly.
|
| You don't get the luxury of offline migrations or single-
| master writes in the high-volume payments space. You simply
| don't have the option. The money must continually flow.
| stickfigure wrote:
| jOOQ is a disaster and I would not recommend it to anyone.
|
| You write some SQL queries, test them in datagrip or whatnot,
| then spend the next several hours figuring out how to convert
| them to the DSL. This problem is compounded when you use
| "exotic" SQL features like json expressions. Debugging is
| "print the generated sql, copy it into datagrip/whatnot, tune
| the query, then figure out how to retrofit that back into the
| DSL".
|
| It's a huge waste of time.
|
| The primary selling point of jOOQ is "type safe queries". That
| became irrelevant when IntelliJ started validating SQL in
| strings in your code against the real data. The workflow of
| editing SQL and testing it directly against the database is
| just better.
|
| jOOQ reinforces the OP's point about DSLs.
| geophile wrote:
| This is a very specific and popular subset of the DSL point:
| Let's just invent a language L that is better than horrible
| standard language X but translates to X. Imagine the vast
| cubicle farms of X programmers who will throw off their
| chains and flock to our better language L!
|
| In many scenarios (including JOOQ and all ORMs), X is SQL. I
| should know, I spent years working on a Java-based ORM. So
| believe me when I say: ORMs are terrible. To use SQL
| effectively, you have to understand how databases work at the
| physical level -- what's a B-tree lookup, what's a scan, how
| these combine, etc. etc. You can often rely on the optimizer
| to do a good job, but must also be able to figure out the
| physical picture when the optimizer (or DBA) got things
| wrong. You're using an ORM? To lift a phrase from another
| don't-do-this context: congratulations, you now have two
| problems. You now have to get the ORM to generate the SQL to
| do what really needs to be done.
|
| And then there are the generalizations of point made above:
| There are lots of tools that work with SQL. Lots of
| programmers who know SQL. Lots of careers that depend on SQL.
| Nobody gives a shit about your ORM just because it saves you
| the trouble of the easiest part of the data management
| problem.
| sroussey wrote:
| > To use SQL effectively, you have to understand how
| databases work at the physical level -- what's a B-tree
| lookup, what's a scan, how these combine, etc.
|
| This is a good reason to use an ORM. But also, as a ORM
| designer, don't let the ORM be flexible to do any SQL. Only
| let it do performant data access.
| stickfigure wrote:
| This is an odd take. Your programming language works with
| objects, the data is in relational tables, you need
| software to map the relations to objects. Thus the Object
| Relational Mapper. There's no reason you can't write SQL
| and let an ORM handle the result set mapping.
| sgarland wrote:
| If that's all you were doing, then _maybe_ , but it never
| is. ORMs enable people who have no idea how RDBMS works
| to use them, which rarely ends well.
|
| I'm not suggesting that to use RDBMS you should know how
| to administrate and tune it (though it helps), but
| knowing their language, and understanding a single data
| structure (B+ trees) isn't too much to ask, I think.
| gedy wrote:
| > ORMs enable people who have no idea how RDBMS works to
| use them, which rarely ends well.
|
| In some cases, but the more frequent issue I saw back in
| the day was the DBA making some really complex schema
| tuned for what they wanted, then an application trying to
| use the data in a pretty reasonable OOP manner (1 to many
| relationships, etc) and the DBA pissed they were using an
| ORM instead of their perfect SQL queries and procedures.
| stickfigure wrote:
| Of course. And after you understand SQL and databases,
| ORMs can save you a lot of typing. I've never understood
| the either/or attitude.
| hot_gril wrote:
| Yep, abstracting away SQL is a common and very costly
| mistake. The article is about more general system design,
| otherwise I would have expected to see that in the list.
| corinroyal wrote:
| I'm always baffled by hate for DSLs until I realize that what
| people are criticizing aren't DSLs, but DSLs you have to write
| from scratch. If you host your DSL on Lisp, then all you have
| to write is your domain logic, not the base language. Most of
| the work is already done, and your language is useful from day
| one. I don't understand why people insist on creating new
| languages from scratch just to watch them die on the vine, when
| these langs could have been hosted DSLs on Lisp and actually
| get used.
| TimTheTinker wrote:
| Not just Lisp, but any language that has strong support for
| either literal in-language data expressions like JSON or
| YAML, or meta-language support like Ruby, Elixir, JSX/TSX (or
| both!).
|
| Every time you write a React JSX expression, terraform file,
| config.yaml, etc., you're using a DSL.
|
| I once wrote a JSON DSL in Ruby that I used for a template-
| based C# code generator. This enabled a .NET reporting web
| app to create arbitrarily shaped reports from arbitrary rdmbs
| tables, saving our team thousands of hours. Another team
| would upload report data to a SQL Server instance, write a
| JSON file in the DSL, check it against a tiny schema
| validator website, submit it, and their reports would soon be
| live. One of the most productive decisions I ever made.
| CyberDildonics wrote:
| This is generally a terrible way to work. Making a bunch of
| custom syntax even in the same language is just adding more
| stuff to memorize for no gain.
|
| Even in C using the "goes to operator" of while(i --> 0) or
| using special operator overloading like the C++ STL >> and <<
| operators for concatenation is just making people memorize
| nonsense so someone writing can be clever.
|
| People don't give presentations with riddles and limericks
| either. It can be clever as a puzzle but when things need to
| get done, it is just indulging someone showing off their
| cleverness at the expense of everyone who has to deal with
| it.
| f1shy wrote:
| I think you misunderstood what a DSL is, or at least the
| point of the OP?
|
| We are advocating exactly to keep the syntax the same as
| the base language, and add semantic value through the
| abstractions of the language.
| CyberDildonics wrote:
| I didn't misunderstand anything.
|
| If you're not changing any syntax and are just using
| normal function calls, that's an API and that's direct.
|
| If you're not just using normal function calls and are
| making your own "semantic value through abstractions of
| the language" you aren't making something that is direct
| and are creating something that needs to be memorized.
|
| The cleverness and indirection of the new stuff that
| hides what is really going on is 99% of the time not
| worth what it gives you, because you have to memorize
| this new clever thing someone came up with, then you have
| to learn what it is actually doing underneath that is
| being hidden.
| f1shy wrote:
| > If you're not changing any syntax and are just using
| normal function calls, that's an API and that's direct.
|
| No. Sorry. Wrong. Look SICP where they explain the
| concept of embedded DSl. Hint: you may be conflating
| syntax and language.
| CyberDildonics wrote:
| Everyone understands the concept. Understanding why you
| shouldn't do it is what takes experience.
|
| If you look at the source code for doom it is very
| straight forward. No fancy stuff, not cleverness, no
| pageantry of someone else's idea of what "good
| programming" is, just what needs to happen to make the
| program.
|
| I'll even give you an example of an exception. Most for
| loops in C and successors are more complicated than they
| need to be. Many loops are looping from 0 to a final
| index and they need a variable to keep track which index
| they are on. Instead of a verbose for loop, you can make
| a macro to always loop from 0 and always give you an
| index variable, so you just give it the length and what
| symbol to use. Then you have something simplified and
| that's useful. It's shorter, it's clear, it will save
| bugs and be easier to read when you need nested loops
| through arrays with multiple dimensions.
|
| I already gave examples before where clever extra syntax
| creates an exceptional situation but gains nothing.
|
| The fundamental point here is that these opportunities
| are rare. Thinking that making up new syntax is a goal of
| programming is doing a disservice to everyone who has to
| deal with it in the future.
| f1shy wrote:
| You are 100% right in all, except you are talking about
| syntax extensions (the for example) and not DSL. A DSL
| does not need a new syntax, is a collection of
| abstractions that allow to express problems in the
| language of the domain problem. It is not an API, because
| is not an interface for a functionality. Is not about
| exposing functionality, but to add semantic value to the
| upper layers. May (not necessarily, but may) be formed by
| a collection of functions, in that case similar to an
| API, in that sense. Sometimes may include indeed
| extensions to a language, but in that case by the
| standard means of abstraction preferred in that language:
| clases, templates, functions, structures. The key is to
| reduce the cognitive load for the end programmers, who
| could be expert in the problem at hand, but not in the
| underlying language of the embedding.
|
| There is also the possibility of embedding in a non
| programming language, like XML (E.g. launch language in
| ROS), or S-exp in the Oracle listener config file. Also
| you can do ad-hoc like in the .msg files of ROS. But is
| always about semantics, not syntax. Syntax is the medium
| only.
| vbezhenar wrote:
| If you wrote some functions, it's not DSL, it's
| functions.
|
| If you calling them in a fancy way with overloads and
| whatnot, it's not DSL, it's fancy functions.
|
| DSL is domain specific language. It includes domain
| specific syntax, domain specific semantics and domain
| specific libraries.
| f1shy wrote:
| Absolutely no. It may have specific syntax, but is not
| needed. Where do you have such definition? In fact the
| typical example is in Lisps, where you add no syntax.
|
| Is not about fancy functions. And not about new syntax.
| Is about adding semantic value. If somebody adds a
| collection of functions that allow the expression of
| solutions to a problem in the very language of the
| problem, that is a DSL, if the syntax chosen, for
| whatever reason, e.g. simplicity, happens to be the same
| as some underlying language, that takes nothing to the
| fact that it is a DSL.
|
| If you look at the examples of SICP, they are "just"
| fancy functions. But they are DSLs
| f1shy wrote:
| Exactly. Good DSL are typically (although not always)
| embedded in another. When that is the case, they tend to be a
| perfect abstraction (if decently implemented)
| travisjungroth wrote:
| I've written two DSLs (one with a team) and I'd consider them
| both successful. They solved the problem and no one cursed them
| out. I think the most important factor is they were both
| _small_.
|
| They were very similar. I even reused the code. One was writing
| rules to validate giant forms, the other was writing rules to
| for decisions based on form responses.
|
| Ok, just ranting on DSLs. Good DSLs take someone from can't to
| can. A DSL that's meant to save time is way less likely to be
| useful because it's very likely to not save you time.
|
| In both of my DSLs, it's that we needed to get complex domain
| behavior into the program. So you either need to teach a
| programmer the domain, partner a programmer with a domain
| expert, or teach a domain expert how to program.
|
| Putting the power in the hands of the domain expert is
| attractive when there's a lot of work to be done. It frees up
| programmers to do other things and tightens the feedback loop.
| If it's a deep domain, it's not like you want to send your
| programmer to school to learn how to do this. If it's shallow,
| you can probably have someone cheaper do it.
|
| A DSL comes with a lot of cognitive overhead. If the other
| option is learning a full programming language, this becomes
| more reasonable.
|
| A time saving DSL is where someone already knows how to write
| code, they just want to write less of it. This is generally not
| as good because the savings are marginal. Then when some
| programmer wants to change something, they have to
| learn/remember this whole DSL instead of more straightforward
| code.
|
| Actually, this makes a simpler rule of thumb. A DSL for
| programmers is less likely to be a good idea than a DSL for
| non-programmers.
| f1shy wrote:
| DSL are just great, have never failed for me. I have done it
| several times, the last one just the past year, for
| programming complex ASICs. I've seen uncountable times
| working like a charm.
|
| I cannot understand why seems it was bad for him...
| travisjungroth wrote:
| I think there are two big reasons DSLs often underdeliver.
| The first is not understanding the cognitive overhead
| budget. If it's something where it's being used
| infrequently or by a lot of new people, that's a lot of
| overhead to be spent each time. Sometimes people think of
| writing DSLs for tests because they have to write a bunch,
| but it's really easy for this to suck. I have a test turn
| red and now I have to learn a DSL to deal with it? Ew.
|
| The second is fuzzier. It's putting a DSL over something
| complex and hoping this will fix things. Writing SQL
| queries for this system takes a bunch of time and is error
| prone? Just put a DSL over it! Except all those details and
| errors are probably going to leak right through your DSL.
|
| You want to master the domain before you put a DSL over it.
| f1shy wrote:
| >> The first is not understanding the cognitive overhead
| budget. If it's something where it's being used
| infrequently or by a lot of new people, that's a lot of
| overhead to be spent each time. Sometimes people think of
| writing DSLs for tests because they have to write a
| bunch, but it's really easy for this to suck. I have a
| test turn red and now I have to learn a DSL to deal with
| it?
|
| What is the alternative to the DSL with lower cognitive
| load? I do not follow. Every single DSL I've seen REDUCES
| the cognitive load, by allowing to express the concept in
| the mere language of the problem at hand, for which the
| SME should be more than familiar with.
|
| About the second point: I see many critics in this thread
| based on DSls above SQL. Whatever somebody is doing above
| SQL and selling as a DSL, it is not. Period. I cannot
| think in any possible way of doing a DSL above a query
| language. No doubt people hate the idea. Is a BAD one.
| epgui wrote:
| These examples sound like things that aren't really
| DSLs... Or in other words it sounds like someone is
| trying to make something "simpler than it actually is".
|
| DSLs are supposed to be for making it easier to perform
| computation in a specific context. Software tests have
| about as many degrees of freedom as the programming
| language they are written in, so I'm not sure they are an
| ideal use case for a DSL-- not without a lot of
| discipline at least.
|
| For a DSL to make sense, IMHO, you need to be able to
| write down a complete and correct specification for it. I
| doubt that is even possible in the given examples :shrug:
| Majromax wrote:
| I think the theory of domain-specific languages is very
| valuable, even if there's rarely a need for a full
| implementation of one.
|
| As I see it, a DSL is just the end-state of a programmer
| creating abstractions and reusable components to ultimately
| solve the real problem. The nouns and verbs granted by a
| programming interface constrain how one thinks, so a flexible
| and intuitive vocabulary and grammar can make the "real
| program" powerful and easy to maintain. Conversely, a rigid
| and irregular interface makes the "real program" a brittle
| maintenance nightmare.
| hot_gril wrote:
| I've yet to see a DSL work great. Every single time, I'm asking
| "why isn't this just Python (or some other lang)" especially
| the times when it's some jacked up variant of Python.
| einpoklum wrote:
| (3.b) "Being clever rather than over-provisioning" is not
| generally thought of as a good idea. People would be rather
| apprehensive if you told them "I'm don't something really
| clever so we can under- or exactly-provision". I mean, sure, it
| may indeed work, but that's not the same thing.
|
| (5) Hybrid parallelism - also, many people think it's a bad
| idea because it makes your software system more complex. Again,
| it may be very useful sometimes, but it's not like many people
| would go "yes, that's just what I'm missing right now, let's do
| parallelism with different hardware and different parts of the
| workflow and everything will run something kind of different
| and it'll all work great like a symphony of different
| instruments".
| bee_rider wrote:
| What do people want to use hybrid parallelism for in systems
| programming?
|
| It works really well in the HPC world I think (MPI+OpenMP on a
| node is the de-facto standard). But... I dunno, I guess when I
| think systems programming I think of the stuff doing all the
| bookkeeping. The bookkeeper better not steal all my cores!
|
| DSL is a sort of interesting one. What's a DSL for systems
| programming? I'll naively throw C and Rust as the systems
| programming DSLs. Of course, the domain of systems programming
| is, uh, controlling all the hardware. So it isn't that surprising
| that the DSLs of systems programming quickly become the languages
| that everybody wants to use for everything, right? The problem
| isn't the "specific language" part, it is that a good enough
| systems programming language quickly gets the domain of
| "everything," haha.
| mrkeen wrote:
| Here's a DSL example https://copilot-language.github.io/
| shermantanktop wrote:
| There are many successful examples of all of these. Using
| "almost" as an escape hatch doesn't work here.
|
| This is just pessimism and weary cynicism. I get it, I've felt
| that way too, and sometimes it's hard to talk an eager engineer
| out of a bad idea. But for me, this vibe is toxic.
| yuliyp wrote:
| A lot of those "successful" examples have teams of battle-
| scarred engineers dealing with all the failures of those ideas.
| Control loops running away to infinity or max/min bounds, a
| cache that can't recover from a distributed failure, corrupted
| live-migrated state, bursts causing overload at inconvenient
| times, spurious anomaly-detection alerts informing you of all
| the world's holidays, etc.
|
| Underneath all of those ideas is a tangle of complexity that
| _almost_ everyone underestimates.
| NBJack wrote:
| I think that's fair: the success stories are very rarely
| "yay, we did it!" but much more often "this single change was
| the _sole_ focus of a team /multiple teams for X months, and
| the launch/release/fix was considered a culmination of a
| significant investment of resources".
| api wrote:
| I know I'm not alone in this, but after doing this for more
| than 20 years I can't shake the idea that we are doing it
| wrong -- meaning programming. Is it really this nit-picky,
| brittle, and hard?
|
| The brittleness is what gets me. In physical mechanical and
| even analog electrical systems there are tolerances. Things
| can almost-work and still work for varying degrees of work.
| Software on the other hand is unbelievably brittle to the
| point that after 50 years of software engineering we still
| really can't re-use or properly modularize code. We are still
| stuck in "throw it away and do it over" and constantly
| reinventing wheels because there is no way to make wheels
| fit. The nature of digital means there are no tolerances. The
| concept isn't even valid. Things fit 100% or 0%.
|
| We keep inventing languages. They don't help. We keep
| inventing frameworks. They don't help. We keep trying "design
| patterns" and "methodologies." They don't help. If anything
| all the stuff we invent makes the problem worse. Now we have
| an ecosystem with 20 different languages, 50 different
| runtimes, and 30 variants of 5 OSes. Complexity goes up,
| costs go up, reusability never happens, etc.
|
| I remember for a while seeing things like the JVM and CLR
| (the VM for C# and friends) as a way out-- get away from the
| brittle C API and fully compiled static native code and into
| a runtime environment that allowed really solid error
| handling and introspection. But that paradigm never caught on
| for whatever reason, probably because it wasn't free enough.
| WASM is maybe promising.
| wakawaka28 wrote:
| >The brittleness is what gets me. In physical mechanical
| and even analog electrical systems there are tolerances.
| Things can almost-work and still work for varying degrees
| of work.
|
| I think you are misrepresenting how flexible software is
| versus hardware. Mechanical and electrical systems have
| tolerances but if you go outside those tolerances, the
| whole system can be destroyed. Nothing like that is common
| in software. Worst-case outcomes might be like "the
| performance isn't as good as we want" or "this code is
| difficult to work with." Software components are very
| flexible compared to anything physical, even in the worst
| cases.
|
| >We are still stuck in "throw it away and do it over" and
| constantly reinventing wheels because there is no way to
| make wheels fit. The nature of digital means there are no
| tolerances. The concept isn't even valid. Things fit 100%
| or 0%.
|
| I don't know how one can look at the amazing array of
| libraries out there and conclude that we have no reuse.
| Sometimes people build their own solutions because they
| need something very simple and the libraries are too big to
| be worth importing and learning in those circumstances.
| That's not a flaw in the libraries. It's human nature.
|
| >We keep inventing languages. They don't help. We keep
| inventing frameworks. They don't help. We keep trying
| "design patterns" and "methodologies." They don't help. If
| anything all the stuff we invent makes the problem worse.
| Now we have an ecosystem with 20 different languages, 50
| different runtimes, and 30 variants of 5 OSes. Complexity
| goes up, costs go up, reusability never happens, etc.
|
| All of this is too pessimistic. These tools do help.
| Exactly how many languages do you think we should have? Do
| you think exactly one group is going to develop for each
| use case and satisfy everyone?
|
| >WASM has its uses but I can't escape the idea that it's
| like "let's build a VM and carry all the shortcomings of C
| into it."
|
| I'm not a web guy but this sounds silly. It's not meant to
| be written directly. Complaining about shortcomings of WASM
| is literally like complaining about shortcomings of
| assembly language. It's not intended for human consumption,
| in modern times.
| ghaff wrote:
| There's probably more tendency to be sloppy in software
| because "we can always fix it in post." But absolutely,
| with hardware, if you don't get it right--especially with
| heavy construction or modern electronics, you're going to
| have to rip a lot out and start over.
| vacuity wrote:
| I've thought that we should be able to move so fast with
| software, and sometimes I see that successfully, but
| usually it seems like we leverage computers' superior
| performance poorly. To pick on frontend, it's trivial to
| create the next shiny Javascript framework, but what of
| it? Or, how come refactoring can be so painful, when the
| semantic change might be small? I think the flexibility
| and performance of computers is such that we programmers
| are usually incapable of effectively using them. It's
| like a 3D optimization problem visualization, looking for
| the highest peak around, except the cursor moves a lot
| faster than it can peruse the landscape. It zooms around
| aimlessly, easily getting to arbitrary places but without
| the capacity to make sense of them. When the train is in
| motion, switching tracks is hard, even if that would be
| the best move.
| ghaff wrote:
| It's easy to assume that you can always fix things after
| the fact. And I was only half joking with the fix it in
| post comment. Modern film suffers from some of the same
| problem. No need to get it right on the first pass. We
| can always apply corrections later.
| intelVISA wrote:
| No, it's not terribly hard for a suitably compensated and
| skilled team with the appropriate tools and timeframe. Yes,
| we are doing it wrong.
|
| Most of the 'inventions' you describe are more aimed toward
| reducing the barriers of entry: the promise that your team
| of expensive C wizards would now use Java at greater speed
| with less defects became "now we can just use cheap CS
| grads" at slightly worse but still acceptable levels.
|
| Without any real consequences for poor software (see
| CrowdStrike's YTD despite its multi-billion dollar farce in
| July) it's only logical that the standard will always be
| "bare minimum that can be shipped". Developer productivity
| is a misnomer really - it just means company profits
| increase thanks to a widening pool to hire from and even
| more crapware per dollar can now be squeezed from each
| worker.
| skywhopper wrote:
| Nah. CLR/JVM/WASM are the same pipe dream of a universal
| architecture/shared-compute utopia.
|
| But I think you have the wrong take on "reusability". Every
| non-software engineering project is an exercise in custom
| solutions as well. The reusable parts are the tools and
| materials. Likewise in software engineering the languages,
| OSes, protocols, libraries, design patterns, and frameworks
| are the reusable bits. Code is how we describe how it all
| fits together, but a huge amount of what it takes to run a
| system is being constantly reused, much bigger than the
| code we write to implement it.
| samatman wrote:
| Basically agree. This is a list of systems ideas which are
| harder than one might initially think, and should be approached
| seriously, never casually.
|
| That slant-rhymes with "sound good but almost never work" but
| in detail is completely different. When treated as difficult
| problems, and committed to accordingly, having them work and
| work well is a normal result, eminently achievable.
|
| As afterthoughts, or when naively thought to be easy, then
| yeah, they frequently go poorly.
| pizlonator wrote:
| I think Steven is saying that these things are hard and
| unusually don't work out, not that they're impossible.
| f1shy wrote:
| Mmmm danger! Then let's stop doing anything that is
| difficult?
|
| If that is bot the message, what is that? "These things are
| hard, often don't work, but GO FOR IT"?
|
| I pretty much read: "try to avoid" which is bad advice in my
| opinion. Like "documenting SW properly while doing
| development is hard, and often goea wrong" so what?!
| tptacek wrote:
| It's just a high-engagement (if you like, "bait-y") way of
| saying "these are deceptively tricky things to get right or
| deploy effectively".
| aprilthird2021 wrote:
| Pessimism and weary cynicism can be very valuable in many tech
| environments though. It keeps you stable, working on tried and
| tested things, and safe. There's a lot of situations where
| that's super valuable
| shermantanktop wrote:
| I had to learn that attitude in order to operate at one
| employer. I had to unlearn it to survive at the next, where
| engineers were better and routinely pulled off things I had
| dismissed as unrealistic.
|
| The value of that approach is very situational...though I
| will acknowledge that the majority of places probably warrant
| at least some of that.
| phil21 wrote:
| > This is just pessimism and weary cynicism.
|
| I don't read it that way. I read it as engineers engaging in
| pre-optimization for no business benefit. It's utterly rampant
| in the industry because it's fun to design and build a
| redundant auto-scaling spaceship vs. just over-provisioning
| your server by 200% for a tenth (or less!) of the cost and
| having backups ready to deploy in a few hours.
|
| Sometimes these ideas make sense - after you need them. Not
| designed-in at the early product stage. Very few products go on
| to need the scale, availability, or complexity most of these
| implementations try to solve.
| jmsdnns wrote:
| p2p cache sharing is an interseting one. it seemed to work for
| spotify for a long time, but they eventually found it easier to
| use edge caching. afaik they still use it for places with less
| developed infrastructure
|
| here is a paper written by the people who designed it:
| https://www.csc.kth.se/~gkreitz/spotify-p2p10/
|
| this part of the abstract gets right to the point: 8.8% of music
| data played comes from Spotify's servers while the median
| playback latency is only 265 ms (including cached tracks)
| dividuum wrote:
| Steam added a P2P caching mechanism recently and allows
| downloading games for other local machines that already have
| the game installed. Greatly helps with avoiding duplicate
| downloads on a steam deck, for example.
|
| I also added a similar mechanism to my own product years ago.
| Works flawlessly. Using content based addressing already made
| it quite easy to implement.
| norir wrote:
| DSLs make the impossible possible. Whether this is good or bad
| depends on the design and implementation.
| heisenbit wrote:
| Compared to general purpose programming when using real world
| DSLs one also can encounter the case where the unusually
| possible becomes virtually impossible.
| ajcp wrote:
| It feels like problem with these aren't the ideas, rather the
| "let's just" approach/expectation in front of them.
|
| For instance "Let's just add an API." I think the approach to an
| API as "just" a feature to your product will be about as
| successful as saying "let's just add a UI". To implement a
| successful UI one needs to be thoughtful, thorough, and bring in
| people who specialize in it. Why should any other interface for
| your product be any different? It's not that it's a bad, or good,
| idea, rather one that shouldn't "just" be done.
| vasco wrote:
| Most professional advice is like relationship advice, people
| extrapolate something that went wrong for them into general
| cases but that rarely directly applies to someone who is not
| you in that very similar situation. And the advice that seems
| to work consistently gets so generic that is almost useless.
| Something like "be thoughtful, try to do the right thing and
| reflect back to see how it went" doesn't write blog posts or
| sells books!
| gmuslera wrote:
| Cargo cults around these system ideas are dangerous. The idea
| may be good, but without understanding what really makes them
| work or not is dangerous.
| fullstackchris wrote:
| Dangerous? In the sense a bad API is dangerous? If so then
| I've been in danger consuming catastrophic APIs for at least
| a decade now
| justin_oaks wrote:
| Exactly. There's no "just" when you're adding an API. APIs
| require a lot of work and complexity to do right:
|
| APIs need to be well-designed or the client may need to make
| multiple API calls when one should suffice. Or the API could be
| confusing and people will call it wrong or fail to use it at
| all.
|
| APIs require authentication and authorization. This means
| setting up OAuth2 or at least having a secure API token
| generation, storage, and validation.
|
| APIs need handle data securely or you'll leak data you
| shouldn't, or allow modification you shouldn't.
|
| APIs need to be performant or you database may be crushed under
| the load. This may involve caching which then adds the
| complexity of cache invalidation and other servers/processes to
| support caching.
|
| APIs need to be rate-limited or sloppy clients will hammer your
| API.
|
| APIs require thorough documentation or they're useless. You may
| even have to add SDKs (libraries for different programming
| languages) to make it easier for people to use the API.
|
| APIs need good error messages or users who are getting started
| will not know why their calls are failing.
| eddythompson80 wrote:
| lol, I can already hear the responses of plenty of
| managers/management I worked with to these points.
|
| _> APIs need to be well-designed or the client may need to
| make multiple API calls when one should suffice. Or the API
| could be confusing and people will call it wrong or fail to
| use it at all._
|
| That's fine. It's a first iteration we can change it later if
| people complain. Lets get something out there and iterate.
|
| _> APIs require authentication and authorization._
|
| Lets not worry about authorization/permissions. It's just one
| key or whatever account they use to log in with now.
|
| _> APIs need handle data securely or you 'll leak data you
| shouldn't, or allow modification you shouldn't._
|
| You're saying you don't know how to do that?
|
| _> APIs need to be rate-limited or sloppy clients will
| hammer your API._
|
| Either "Lets worry about that later when it happens" or "Here
| is the first github link from a google result for 'api rate
| limit open source free'. Lets use that"
|
| _> APIs require thorough documentation or they 're useless._
|
| Either "We need to have the API first, docs, clis, etc could
| come later after we have gauged the usage or had asks for
| them. We can handhold the customers asking for them for now."
| or "here is a github project that autogenerates docs, SKDs
| and clis from an OpenAPI spec"
|
| _> APIs need good error messages or users who are getting
| started will not know why their calls are failing._
|
| You're saying you don't know how to do that?
|
| "It doesn't have to be perfect. We have customers who are
| asking for it and to win them we need to implement something
| then we can work with them to improve it. Otherwise we will
| lose them"(tm)
| liontwist wrote:
| Yep, helps a lot when your management is in the domain.
|
| Phased implementation isn't the worst idea, they just have
| to be committed to ending the experiment or seeing it
| through.
| 9dev wrote:
| Then again, these answers are valid. Your fancy API
| documentation will only be skimmed, nobody is going to
| notice the elaborate and elegant URI design or the
| discussion around whether to use PUT or PATCH, and the
| super-secure token mechanism you come up with will be
| written on a sheet of paper in the office.
|
| Most projects are far too fluid in their shape to warrant a
| proper design up-front anyway.
| lazyasciiart wrote:
| They're really not. That approach is how you get "why do
| I need this account ID to look it up in the API but the
| only way I can get it is by using the UI? Why can I
| update this object but not see its current state?" and
| other complaints that users very reasonably have about a
| thing I worked on.
| 9dev wrote:
| Don't get me wrong, I love to obsess over that kind of
| detail, but complaints like these usually aren't a deal
| breaker, and yet it's reasonable to count those
| complaining users as successful conversions regardless.
| lucianbr wrote:
| Recently I was working on a 12-year old project where the
| product people kept asking for "quick wins" and "leave
| the refactorings and in-depth analysis and difficult
| problems for later". When will they get to the "later",
| if 12 years was not enough? I eventually left.
|
| Of course, YMMV. Every company and manager is different.
| nyrikki wrote:
| I disagree, those are all technical issues that are typically
| blocked by socio-technical and organizational problems.
|
| Retrofitting an API is a context specific problem, with few
| things that generalize but the above aren't the hard parts,
| just the complex ones.
|
| I am going to point you to a Public Policy book here, look at
| page 32 and their concept of 'wicked problems', which applies
| to tech but doesn't suffer the problems with the consulting
| industry co-opting as much as in the tech world.
|
| https://library.oapen.org/bitstream/id/f3358d25-56c0-47f2-90.
| ..
|
| Obviously most companies won't accept the Amazon style API
| edicts that require Externalizable interfaces and force a
| product mindset and an outside in view.
|
| Obviously an API is still a "Cognitively complex problem" as
| defined by the above.
|
| But what makes most API projects fail is the fact that
| aspects arise that result in 'wickedness' A.K.A
| intractability.
|
| API gateways, WAFs, RESTful libs with exponential backoff
| with jitter....etc... all exist and work well and almost any
| system that you have a single DB you are concerned about
| killing is possible to just add a anti-corruption layer if
| you don't have too much code debt, complicated centralized
| orchestration etc....
|
| But these general, but difficult tasks like adding an API
| almost never fail due to tech reasons, but due to politics,
| poor communication, focusing on tech and not users,
| unrealistic timelines, and turf battles.
|
| You are correct that "APIs require thorough documentation or
| they're useless" but more importantly they need enforceable
| standards on contract ownership, communication.
|
| The direction of control, stability, audience, and a dozen
| other factors effect the tradeoffs there as to what is
| appropriate.
|
| Typically that control is dictated purely by politics and not
| focused on outcomes.
|
| That is often what pushes these complex problems into failed
| initiatives, and obviously this is far more complicated.
| Often orgs can't deal with the very real uncertainty in these
| efforts and waste lots of project time on high effort, low
| value tasks like producing gantt charts that are so beyond
| the planning horizon that they can only result in bad
| outcomes at best.
|
| The point being is that unless you are on the frontier of
| knowledge and technical capabilities, it is almost _never_
| the tech that causes these efforts to fail.
| harrall wrote:
| When I read OP's list of wants for an API, I don't read it
| so literally.
|
| I read it as an example of the complexity required to
| implement something like an API.
|
| I agree that it's also a people problem but I disagree that
| many of these failed initiatives aren't due to bad tech as
| well. A complex initiative requires picking the right tech
| that gives you a good benefit/cost ratio. Knowledge of tech
| plays hand in hand with the political aspects and IMO many
| orgs lack both completely.
| nyrikki wrote:
| > A complex initiative requires picking the right tech
| that gives you a good benefit/cost ratio.
|
| That choice should never be a one way door, you don't
| control your customers or the future, it is the frame
| problem.
|
| The idea is to be able to iterate and adapt, not get
| things perfect from the start.
|
| It is fine and expected that one will use past experience
| as a starting point, but you always have to reflect and
| check your assumptions along the way.
|
| It is a complicated topic ruled by nuances and contexts,
| but if you cannot pivot away from your initial
| assumptions over time, that lets you know you are leaking
| implemention details.
|
| While time is limited and abstractions have a real cost,
| you need to apply them where they make sense.
|
| Some, like just breaking code onto two files in the same
| directory are fairly low cost, others are much more
| expensive.
|
| For several years I was jumping in to save failed cloud
| migrations.
|
| Whenever tech was the 'blocker' it was because of a myth
| that there was only one way to do things.
|
| But lets say you are writing machine code for Apple
| silicon? Do you really think those registers are
| concrete? They aren't. They are a facade hiding 100s
| behind a legacy interface.
|
| Vendor mitigation important and often neglected and also
| results in people producing balls of mud.
|
| But there are a lot of distributed monoliths pretending
| to be micro services out there. And we had decades of
| people producing fragile enterprise service buses that
| were built because people wanted to future proof their
| systems.
|
| The sizing of components and balancing integration and
| disintegration drivers is incredibly hard, you will never
| get it right.
|
| You have to leave options open, no matter if that is
| through abstractions or keeping biz logic centralized to
| assist in a easy rewrite of context boundaries that arise
| from scale or changing needs.
|
| Obviously choices have benefits and costs, but for most
| needs you can keep options open, if not it is probably
| best to reevaluate the reason for choosing a solution.
|
| Obviously there are predatory vendors like Oracle that
| base their entire income on captive customers, but that
| relates to the vendor mitigation above.
|
| I think the US federal government de-risking guide is a
| good overview of that topic. Microsoft is a strong driver
| for this BTW.
|
| https://guides.18f.gov/derisking-government-tech/
| btilly wrote:
| For anyone who wants to read more about what the Amazon
| experience was like, I highly recommend Steve Yegge's
| platform rant at
| https://gist.github.com/chitchcock/1281611.
| Groxx wrote:
| Every single one of these is true of a website, but we have
| far more websites than APIs.
|
| If there's one thing APIs suffer from more, it's "social cost
| to make changes". A concurrent vN+1 largely resolves that
| though, unless your API consuming ecosystem is large enough
| to be worth investing _vastly_ more resources into.
| nswanberg wrote:
| Yegge wrote about the business idea version of this as "Shit's
| Easy Syndrome":
|
| https://steve-yegge.blogspot.com/2009/04/have-you-ever-legal...
|
| It'd have been delightfully ironic had either of these Steves
| concluded their essays with a named methodology to "just" apply
| whenever faced with these "let's just" situations but alas...
| jabroni_salad wrote:
| I am pretty sure presales as a career exists solely because of
| the word 'just'. We have to make sure they wont get spooked
| when we start lifting the rest of the iceberg out of the water.
| aprilthird2021 wrote:
| Yes, and a very well designed API can be the foundation of and
| moat for a successful business. Stripe is one of the best
| examples. Even as hundreds of copycats pop up, their excellent
| API design and the trust devs have in them to nail the next API
| design, means they can continue to expand into other financial
| products with ease.
| btilly wrote:
| At a previous employer we had a rule. The only person allowed
| to say, "just", was the developer responsible for actually
| making it work. Another developer who said it, just volunteered
| themselves.
|
| It was a rule that worked well for us.
| jrs235 wrote:
| Needs to also be for "that's easy"...
| sharpy wrote:
| I am totally going to advocate for this at my workplace. I
| myself am guilty of it - not saying "just", but questioning
| people's estimates based on my limited understanding of the
| research they had to do to come up with the estimates in the
| first place.
| ffsm8 wrote:
| It sounds pretty toxic if you're not allowed to even
| question other people's estimates. IME that's a large part
| of Grooming/refinements
| gdsdfe wrote:
| That's a good one!
| BlarfMcFlarf wrote:
| Punishing people with more work doesn't make sense in a well
| run organization. Work is continuous and more work only
| affects the backlog, not the actual developer's life or
| experience.
| btilly wrote:
| It was not a punishment. It was more, "You're sure that you
| know an efficient way to do this? You get a chance to prove
| that you're right." It made people careful because being
| wrong had a consequence, but it usually moved work to the
| person who thought that they had the best approach.
|
| This happened in Sprint Planning. Obviously by the end of
| the meeting, everyone would have a full sprint. So
| volunteering for one thing required giving something else
| up. This was definitely part of the equation.
| bryanrasmussen wrote:
| Ok so if you have a problem and the developer thinks that the
| solution to it is to add an API, all they have to do to get
| their way is to say "let's just add an API!", although they
| would of course be the one that had to make it work.
|
| Yes I like this rule, how many things I could have solved at
| the worst run place I ever worked at if it had been
| implemented there. Although I guess the management might not
| have liked that essentially I could choose what got done and
| what I got to work on just by running my big mouth.
|
| facetiousness aside, I'm trying to point out that most often
| when people say "let's just" it is because they are
| advocating a course of action and that rule won't work unless
| the course they are advocating gets chosen. If they say
| "let's just make an api" and you say no we're doing a UI and
| you will work on it because you used the word "Just" I guess
| that's a way to lose your tech talent relatively quickly.
| evalstate wrote:
| Gerry Weinberg wrote beautifully about these "lullaby words".
| https://www.humansystemsinaction.com/lullaby-language/
|
| > "Precisely. It's what I call a 'Lullaby Word.' Like
| 'should,' it lulls your mind into a false sense of security.
| A better translation of 'just' in Jeff's sentence would have
| been, 'have a lot of trouble to.'"
| karmakaze wrote:
| Yes exactly. The article would be much better positioned as
| What it actually takes to make systems ideas "just work". Then
| it isn't about whether it can or can't but rather how it
| succeeds or fails. Having details about what it takes then
| stops people from saying "just" because they don't know what
| they're talking about.
|
| But I pretty much agree 100% about DSLs, it's an
| unnecessary/cute complication. The only people who should be
| allowed to make them should be ones who have made a successful
| programming language and updated it to deal with all their
| mistakes or a 2nd version/2nd language, and still got many
| things wrong.
| insane_dreamer wrote:
| "seldom work as expected" would have been more accurate than
| "almost never work"
|
| I did like this:
|
| > More importantly, an offering an API doesn't mean anyone wants
| to use it. Almost every new API comes up because the co/product
| wants features, but it doesn't want to prioritize them enough and
| the theory is the API will be "evangelized" to some partner in
| the space. Turns out those people are not sitting around waiting
| to fill in holes in your product.
| stevebmark wrote:
| I would add "Domain Driven Design" - locking your business design
| in place by trying to make your application match your business
| structure is a recipe for disaster. If you have a small or
| stagnant business you probably won't notice any issues. If your
| business is successful and/or grows, you're going to immediately
| regret trying to build domains with horrific descriptive names
| tied to your already obsolete business practices. Instead, design
| around functionality layers (how we've been doing it for decades,
| tried and true), and as much as possible keep business logic in
| config, rows in databases, and user workflows, which makes them
| extremely flexible.
| jkaptur wrote:
| Along with "make impossible states impossible to represent". If
| you're designing your types to make a state unrepresentable,
| you'd better be absolutely sure the state really _is_
| impossible for the lifetime of the design.
| spencerflem wrote:
| If its not impossible, you should handle it though.
|
| I think what that quote is against is the common middle
| ground where states are expected to be 'impossible' and thus
| not handled and cause bugs when they are found to be not
| actually.
|
| Either deciding that they are possible or must be impossible
| is usually better and which one to go with depends on the
| specifics
| eddythompson80 wrote:
| You'll regret both options. You mentioned the pitfalls of
| "domain driven design" (outdated language, little code/systems
| reuse for new endeavors).
|
| However, a highly abstract design with all the business logic
| in config, workflows, etc will only makes your system extremely
| flexible as long every one up and down the organization is
| fairly aware of the abstractions, the config, and uncountable
| permutations they can take for your business logic to emerge.
|
| Those permutations quickly explode into a labyrinth of
| unknown/unexpected behaviors what people will rely on. It also
| makes the cost of onboarding new developers, changing the
| development team insurmountable. Your organization will be
| speaking 2 different languages. Most seemingly straightforward
| "feature asks" that break your abstraction either become a
| massive system re-design/re-architect or a "let's just hack
| this abstraction so it's a safer smaller change for now". The
| former will always be really hard unless you have excellent
| engineers who have full understanding of the entire system and
| its behavior and code base along with and excellent engineering
| practices and processes, and still will take you months or
| years to pull off. The latter is the more likely to happen and
| it's why all those "highly abstract, functionality layers,
| config driven, business logic emerging) projects start perfect
| and flexible and end up as a "what the fuck is even this".
|
| After a system is implemented, that emergent business logic
| becomes the language everyone will speak in. Having your
| organization speaking 2 or 3 completely irreconcilable
| languages is very painful and unless you have multiple folks
| up, down and sideways in the organization that can fluently
| translate between the 2, you'll be in a world of pain and wish
| you had some closer representation of your domain
| tinthedev wrote:
| The whole post, and both the Twitter messages sound like they
| refer to a very specific style of work.
|
| These things "sound good but almost never work" when they're
| taken as an afterthought or committed to without due
| research/process/design.
|
| Any architect worth their salt will avoid implementing hard
| solutions to these problems. These are mostly here to not solve
| crucial issues... but the article seems to be addressing a
| developer profile that adds APIs or asynchronous processing
| casually.
|
| Sounds like a strawman, or just exceedingly pessimistic look at
| the industry.
| riwsky wrote:
| So many people here trying to thread the needle looking for
| subtle decision functions for exceptions. It's pretty simple,
| really: these ideas are awesome when I do them, and never work as
| intended when that idiot before me did them.
| plagiarist wrote:
| That sounds exactly right. And sometimes the idiot before me
| was myself from several months ago.
| NBJack wrote:
| I'm wondering about the first example of device drivers. Whether
| we are talking about Linux or Windows, anything beyond generic
| device drivers (which push the problem to the hardware) will
| frequently be vendor specific. You often can't get the full
| potential of a device until you do so.
| ww520 wrote:
| Haha. This is a great list. I have heard variants of these ideas
| over the years.
| devjab wrote:
| Around here people abstract because they are taught to abstract
| in multiple different CS education programs. For whatever reason
| they are taught Clean Architecture even though it's horrible.
| Then they spend the next 5-10 years learning not to abstract or
| write 5 line functions which then lead down a chain of 900
| function calls. As an external examiner or some or these CS
| students it always pains me to examine them in a curriculum that
| I know will lead to dead end job in stagnant small sized
| companies. It is what it is though. I guess it's hard to compete
| with the entire industry of pseudo-science and shitty consultancy
| when nobody is pushing an "indirection manifesto".
|
| Hilariously Uncle Bob will write off any criticism as "they
| misunderstood the principles". He's correct too, but maybe the
| principles are simply too vague when we've had them for 20 years
| and our industry has never been more of a mess.
| bedobi wrote:
| A lot of Robert C Martins pieces are just variations on his
| strong belief that ill-defined concepts like "craftsmanship"
| and "clean code" (which are basically just whatever his
| opinions are on any given day) is how to reduce defects and
| increase quality, not built-in safety and better tools, and if
| you think built-in safety and better tools are desirable,
| you're not a Real Programmer (tm).
|
| I'm not the only one who is skeptical of this toxic, holier-
| than-thou and dangerous attitude.
|
| Removing braces from if statements is a great example of
| another dangerous thing he advocates for no justifiable reason
|
| https://softwareengineering.stackexchange.com/questions/3202...
|
| Which caused the big OSX/iOS SSL bug in 2014, see
| https://www.imperialviolet.org/2014/02/22/applebug.html
|
| This link and thread on hackernews is good too
|
| https://news.ycombinator.com/item?id=15440848
| The current state of software safety discussion resembles the
| state of medical safety discussion 2, 3 decades ago (yeah,
| software is really really behind time). Back then,
| too, the thoughts on medical safety also were divided into 2
| schools: the professionalism and the process oriented. The
| former school argues more or less what Uncle Bob argues: blame
| the damned and * who made the mistakes; be more careful, damn
| it. But of course, that stupidity fell out of
| favor. After all, when mistakes kill, people are serious about
| it. After a while, serious people realize that blaming and
| clamoring for care backfires big time. That's when they
| applied, you know, science and statistic to safety.
| So, tools are upgraded: better color coded medicine boxes, for
| example, or checklists in surgery. But it's more. They figured
| out what trainings and processes provide high impacts and do
| them rigorously. Nurses are taught (I am not kidding you) how
| to question doctors when weird things happen; identity
| verification (ever notice why nurses ask your birthday like a
| thousand times a day?) got extremely serious; etc.
| My take: give it a few more years, and software, too, probably
| will follow the same path. We needs more data, though.
| linhns wrote:
| I don't know if he considers the expiration date of his
| ideas. Some of his ideas actually creates worse software and
| crater projects.
| bhouston wrote:
| I've executed a bunch of these ideas successfully. So it reads a
| little weird.
| Xcelerate wrote:
| IDK about anomaly detection being on that list. When I worked at
| a large tech company, the in-house anomaly detection and root
| cause analysis capabilities worked like magic compared to other
| places I've been. When done right, it can be extremely valuable.
| kitd wrote:
| Sounds like you hit the 1/10. Which is great and I agree, very
| rewarding.
| jldugger wrote:
| Half of the magic is knowing _when_ to apply it. If you just
| dump your prometheus timeseries data into an anomaly detection
| system and ask it to constantly scan for anomalies, you will
| always find them.
|
| As an SRE, I don't actually care about anomalies all that much.
| For initial alerting, I want phase shift detection. One
| customer sending a few bad API calls on one specific minute is
| uninteresting and pretty much inactionable. That same error
| rate over 10 minutes is more interesting and more likely to be
| a systems problem I can actually resolve. But the raw AD stream
| is just too damn noisy for all sorts of reasons. This is why
| our alerting tools have a `duration` field: any signal above
| threshold must remain so for multiple observation periods
| before summoning human inspection. And why health checks have
| grace periods and retries before killing services.
|
| Where anomaly detection works better, IMO, is post-alert
| analysis. At that point anomalies are welcomed as hypotheses,
| since the system features complex interactions between
| components. I've built a couple of dashboards using extremely
| simple math, like Laplace smoothing and time series correlation
| that help surface relevant information from the flood of
| metrics and logs collected. But critically, these tools
| generally don't use the time domain as their baseline. Usually,
| I'm comparing a cluster against another one in a different
| region, or one metric against another, rather than now versus
| twelve hours ago.
| bcrosby95 wrote:
| Unless you're a large corporation: fighting Conway's Law.
| hot_gril wrote:
| Why "unless" and not "especially when"? :)
| slooonz wrote:
| "Let's just cache this slow call"
| nottorp wrote:
| > Let's make it cross-platform.
|
| So if making it cross platform "won't work", this is not about
| what won't work, but what is cheaper?
|
| Since the OP kinda mentions gaming, let the customers fiddle with
| their wine installs and steam decks and spend the time on adding
| more loot boxes instead?
| IshKebab wrote:
| Yeah I don't buy that at all. He says "9 times out of 10" and
| then gives MS Office as the proof? MS Office is the 10th time
| out of 10, not the 1-9.
|
| I think they absolutely _could_ have made Office cross platform
| (it doesn 't really do anything you can't do in Qt), and also
| the fact that they forked it and made two entirely separate
| sets of apps has pretty serious consequences. The feature sets
| are surprisingly different. E.g. you can add PDFs to documents
| on Mac; not on Windows.
|
| Most software can be made cross platform (as long as you don't
| explicitly prevent it by using a platform specific GUI toolkit
| or whatever).
| ghjfrdghibt wrote:
| The "let's make it cross platform" is what lead me to dart and
| flutter. These aren't mentioned and for the developers of them is
| definitely a hard problem. But as far as I'm concerned they're
| doing a bang up job.
|
| I agree with sync though. Hard problem with no simple solution
| for an idiot like me to do.
| CyberDildonics wrote:
| A lot of times what people want from a DSL is really just
| something that doesn't have to compile and/or something to run in
| a VM. Sometimes people just want a text format for data that
| could be done with json.
|
| Making a new language throws away everything and starts over
| including debugging, tools, syntax checking, auto completion,
| documentation etc. It is take way too lightly and just becomes a
| hassle.
|
| You can see this in multiple GUI libraries, where getting the
| parameters to set up a GUI is really not difficult and is just
| data through function calls, but it gets made into a separate XML
| like DSL markup language with it's own quirks and opacity, and
| that XML is for most people being given using a big string from
| within the language that they're using.
|
| This stuff persists because it sounds easier on paper and it just
| creates more problems in practice but it takes experience to
| realize all the you're losing. That's where designers need to be
| experienced and do what works instead of what will suck in people
| that don't know any better.
| adzm wrote:
| With how easy it is nowadays to embed Lua, and even JavaScript
| with just a little bit more effort, it's pretty inexcusable to
| develop your own language for a tiny space. A great example of
| this is Adobe embracing JavaScript for it's own extension and
| scripting and expression language, which has allowed it to
| benefit from an entire ecosystem of JavaScript libraries.
| throwaway63467 wrote:
| Hm I can think of several instances for each point where the
| ideas work just great. It's just difficult getting complex stuff
| to work in general.
| quotemstr wrote:
| The ideas in the first tweet screenshot: uh, they _do_ sound good
| and they _do_ work. Your system is full of control loops. (What
| do you think regulates how many dirty pages the kernel writes
| back to disk?)
| adzm wrote:
| > Windows NT is riddled with excess abstractions that were never
| really used primarily because they were there from the start
| before there was a real plan to use them.
|
| It is? Like what? I know there are some abstractions for cpu
| architecture etc but they've come in handy for x64 and now arm
| and others in the past.
|
| I know the author certainly has some insight into this but I've
| never really thought of NT as being riddled with excess
| abstractions.
| senderista wrote:
| It's possible that he's referring to purely internal
| interfaces?
| liontwist wrote:
| > Most of the first 25 years of computer science was figuring out
| how to make things work asynchronously
|
| This is not true. Computers focused on single threaded designs
| before getting thrown into parallelism (need parallelism to run
| into dining philosopher problems).
| mrkeen wrote:
| I'm not sure what counts as the "first 25 years", but Dining
| Philosophers is from 1965.
| liontwist wrote:
| Anyone have any experience with the control loop one?
|
| My uninformed guess is CS people just underestimate the skills
| and experience to analyze feedback systems and so write it off as
| a bad technology after a poor implementation.
|
| Maybe the real problem is you want to know when your system is
| maxing out the range you anticipated?
| k__ wrote:
| Most of these things work pretty well.
|
| It's just that they don't "just" work.
|
| Just because AWS or Google can pull it off doesn't mean it's
| something anyone can do.
| sgarland wrote:
| THIS. And adding to it, most people really do not like being
| told that they aren't as clever as AWS or Google.
| jcims wrote:
| Yep, I think the title should be "things that are way harder
| than they seem".
| asveikau wrote:
| Just remember that this is the genius who brought Windows 8 and
| the Windows RT tablet that wouldn't run recompiled Win32 apps.
| plagiarist wrote:
| Maybe some experience making large mistakes is how they came up
| with the list?
| rogerthis wrote:
| Event sourcing.
|
| PS: on mobile, can't expand (issues around versioning, data
| migration)
| mikewarot wrote:
| >Let's just add access controls later.
|
| This (Computer Security) is a _solved_ problem, kids. One of the
| lessons learned in the Viet Nam conflict was the need for a
| computer system that could safely handle multiple levels of
| classified data. The solution was Multilevel security, included
| in that was the Bell-LaPadula model. Several actually secure OSs
| emerged over the decades since, including KeyKOS, CapROS and
| Eros.
|
| I'm hoping that someday I can make Genode, the latest capability
| based operating system, my daily driver, so I never have to worry
| about virus scanners again.
| nostrademons wrote:
| Computer security is a solved problem _for systems with a
| fully-specified set of capabilities, usage patterns, and legal
| operations_. You follow the Principle of Least Privilege, build
| in defense in depth, use capabilities, and never expose
| anything that the user doesn 't have a reason to access.
|
| The problem is that the constraints this imposes on the system
| usually do not line up with the constraints that the market
| will pay for. It's very common for customers to change their
| mind; decide they need to hack around access protections; add
| new users with new roles that are some hybrid of current
| access; ask for new features; not think through who should have
| access to new features; want to enable serendipity where
| untrusted users discover new use-cases and new markets for
| their product; and so on. It's also very common for them to
| ignore security as a differentiator when making their purchase
| decision, figuring that if there's a breach, somebody else will
| pay for it, or they'll be long gone from the company and unable
| to be blamed for it. So the market ends up bypassing the secure
| solutions that exist and choosing to buy insecure systems that
| can offer the features they want right now.
|
| When security is absolutely critical, like in military or
| certain financial applications, it's pretty easy to achieve.
| There are companies like Galois that specialize in "high
| assurance systems". But they are expensive for their feature
| set, and so the general public would rather buy from cheaper
| and more insecure options.
| mikewarot wrote:
| Eventually we're going to have to collectively decide that
| the operating system is the correct place to enforce
| capabilities, as is done in mainframe OSs.
|
| Memory holing any mention of this _solution_ isn 't
| productive in the long run.
| wpollock wrote:
| You're right about Bell-LaPadula, but note that model isn't
| used as much as the Biba model, the basis of Windows UAP. Even
| "solved" problems tend to evolve or get superceded over time;
| nothing seems to stay solved for long!
| mikewarot wrote:
| Windows UAP is a horrible thing. It would be far better to
| just replace the system dialog boxes like file open, save,
| etc. with power boxes that then give file capabilities to
| applications.
|
| It would be minimal work to refactor applications, and
| provide almost perfect security with no UX change.
| exabrial wrote:
| I thought live migrating process state did work...? we have
| containers that can freeze/resume.
|
| Granted, you have to freeze the whole damn OS image, but it
| _does_ seem to work.
|
| All of the others I agree, I've seen them tried so many times in
| my lifetime, from mainframes to serverless, and nobody gets them
| right.
| thraxil wrote:
| Yeah, and I was live migrating entire VMs between hosts on a
| Xen cluster 10-15 years ago. It was easy for me to get working,
| reliable, and I was far from an expert.
| xyzsparetimexyz wrote:
| I've been writing glsl recently, and I think it's a good example
| of a DSL that _doesn't__ work. It should have just been C without
| the standard library, or ideally C++. Not being able to use
| pointers is especially painful. Hlsl is better and has some nice
| quality of life features. Rust-gpu is the best approach but
| basically nobody uses it
| nostrademons wrote:
| There is a pattern to all of these problems, notably that they
| are all _orthogonal concerns that add constraints_ to the
| sequential data-munging programming model that programmers are
| familiar with. Whenever you add constraints, you add things that
| future programmers need to think about, for _all future
| development they do on the system_. It is very easy to get into a
| situation where the system is overconstrained and it 's
| impossible to make forward progress without relaxing some of the
| constraints. Even if it's not impossible, it's going to be _slow_
| , as developers need to consider how their new feature interacts
| with the API/security/synchronization/latency/other-
| platforms/native-code that the system has already committed to
| supporting.
|
| That's also why it's _possible_ to support all of these
| attributes. If you make say transparent data synchronization a
| core value prop of the platform, then all future development
| supports that _first_ , and you evolve your feature set based on
| what's possible with that constraint. That feature set might not
| be exactly what your users want, but it's what you support. Your
| product appeals to the customers for whom that is their #1
| purchase decision.
| qianli_cs wrote:
| Or let's just reinvent a new database
| Aloha wrote:
| Or worse, lets reinvent the database, but not call it or think
| of it as a database.
|
| Or even worse yet, lets reinvent the filesystem then host it on
| top of a filesystem.
| hot_gril wrote:
| I've heard "it's not a database" multiple times, in reference
| to whatever custom DB our team inherited. Terrifying.
| ibejoeb wrote:
| "Let's just sync the data" is the reason why rough days exist, as
| far as I'm concerned. I've run into so many systems that were
| designed for "internet scale" or whatever that add queues, event
| processing, etc., when the natural range is so far below that
| threshold. These teams are either naive or, at worst, taking
| advantage of non-engineering management and funneling money
| toward playing with these problems for the fun of it.
| mrkeen wrote:
| I read "just sync the data" as naively doing reads over here
| and writes over there, and hoping that the two sources of truth
| don't diverge.
|
| Queues and event processing are a necessity to do it right.
| ibejoeb wrote:
| That's sort of the point. The operative word is "just." In
| reality, it adds a huge subsystem that needs more resources
| than the overall system does. Rather than solve the
| underlying problem directly, they introduce (more
| interesting) problems to solve transitively via these
| complexities.
| Brian_K_White wrote:
| I don't understand the load-responsive control loop one. That's a
| basic and fundamental component in countless systems. The
| centrifugal governor on a 1800's steam engine or 1900's victrola
| record player is a load-responsive control loop. All of
| electronics is a mesh of load-responsive control loops. The
| automatic transmission in your car...
| panic wrote:
| It's not totally clear, but it could be talking about CPU load
| in particular, which has some problems as described in
| https://arxiv.org/abs/2312.10172.
| LAC-Tech wrote:
| There are equations/properties for data that syncs nicely and
| allows mutli-master writes. Ie, if your merges follow a few
| simple properties, this stuff will work.
|
| These are basically what CRDTs are. And I know what you're
| thinking; "but I'm not writing a mutli user text editor!" or
| "automerge won't scale!". But CRDTs aren't a library, they're a
| set of properties. if your whole data system obeys the properties
| - you have a CRDT.
| poooka wrote:
| Disagree with plugin talk. I use plugin system that loads from
| submodules so I can have same architecture in multiple airgaped
| networks. Can have implementations load (for that environment)
| outside of main code base. Nothing wrong w that given
| complexities of maintaining complex ass configs if everything was
| shoved into the same artifact.
| fancyfredbot wrote:
| I've worked on several DSLs, a P2P cache, and a project employing
| hybrid parallelism. All of them worked. All great fun to create.
| With one exception these projects were good investments (The P2P
| cache wasn't necessary so never really paid off). My point is
| that it's definitely wrong to say these things almost never work.
| They are complicated but that complexity brings functionality
| which is hard to achieve in another way. The lesson from the P2P
| cache example was to be sure you actually need that
| functionality.
| mlhpdx wrote:
| Everything on the list I've seen done (been hands-on with) very
| successfully, most of them repeatedly. I've also seen many, many
| more failed (and often pointless) attempts. I don't have any
| great answer to why other than the importance (practical value to
| customers) of it being clear, well understood by leaders and
| doers, and high.
|
| If having an API is core to the value customers will realize then
| it's likely a good API will emerge. Etc.
| mjaseem wrote:
| I feel "Let's just" is doing much of the work here. You can add
| the phrase in front of any idea and make it sound awful:
|
| "Let's just make planes fly themselves"
|
| "Let's just put a giant battery in a car instead of an engine"
|
| "Let's just make electricity from wind"
|
| "Let's just be friends"
|
| I would love to see a list like this without such a big asterisk.
| levzettelin wrote:
| Lost me at "DSLs".
| retrocryptid wrote:
| I think each of the ideas listed: dsls, control loops, being
| clever, etc. They all have their place, and I wouldn't say they
| never work. I've had each of these work well in specific
| situations.
|
| But the OP does have a point, they each _can_ introduce more
| trouble then they 're worth. Were I to write this post, I would
| have titled it something more like "Systems Ideas Yo Really
| Should Think About Long And Hard Before Doing."
|
| But yeah, that might not be enough warning.
| Sparkyte wrote:
| All rely heavily on one aspect, consistency. If you system
| engineering can consistently operate without change, innovation
| or issue, then it is possible that these ideas could exist in a
| closed environment. However they just don't work when dealing
| with external factors.
___________________________________________________________________
(page generated 2024-12-31 23:00 UTC)