[HN Gopher] I am afraid to inform you that you have built a comp...
___________________________________________________________________
I am afraid to inform you that you have built a compiler (2022)
Author : mutant_glofish
Score : 148 points
Date : 2023-08-17 15:10 UTC (7 hours ago)
(HTM) web link (rachit.pl)
(TXT) w3m dump (rachit.pl)
| RandyRanderson wrote:
| Creating a configuration file? I am afraid to inform you that you
| have started writing a compiler. What's the only way to avoid
| this? Your software not being successful.
| dessimus wrote:
| Dammit! Every time I try putting together an Ikea bookshelf, this
| happens!
| rozap wrote:
| A funny thing is that this can go N layers deep. At a previous
| job we had a DSL which was very well specified, pretty straight
| forward, and had been written by an exceptionally brilliant
| engineer (who was even still at the company!). We had two backend
| services that would operate on that AST in a query context and in
| a streaming transformation context, and they shared the parser,
| compiler, and typechecker. The problem was that the engineering
| team was so fragmented and lacking in stable technical leadership
| that a large part of the engineering team had no idea about how
| these things worked, so on every project they were constantly
| going "Ooops, I created a feature which has its own intermediate
| representation with slightly different semantics so reuse is
| impossible". It was super hard to deal with and stuff was
| constantly being invented and then thrown out because it couldn't
| be extended.
|
| The only thing worse than creating a compiler is being unaware
| that one already exists and creating a new one on top of the
| existing one.
| renox wrote:
| What's strange is that if you had started with a Lisp, it would
| have been much simpler!
|
| And yet few people think about using a Lisp for their DSL.
| mbork_pl wrote:
| Without even reading TFA, it is obvious that choosing a Lisp
| for a DSL is probably at least a good enough choice.
| amelius wrote:
| A compiler typically transforms programs that run in O(N) time
| into programs that run in O(N) time, for whatever suitable
| definition of N, so it's not really something to be super
| thrilled about.
| ashton314 wrote:
| It's funny how projects grow like this organically.
|
| Perhaps burned by experience, one time I implemented a mini-
| language for specifying some business logic that I knew--just
| _knew_ --that our client would change his mind on a dozen times
| and not understand the half of the ramifications of his requests
| and would only arrive at the solution he _really_ wanted by
| trial-and-error. Was the little language I made as complex as a
| "real" compiler? Goodness no. Was I happy to have a flexible
| language-based solution? True to my predictions, I _did_ get
| many, many logic change request and handled them with ease. Yes,
| I was very happy after that.
| mikepurvis wrote:
| What were the requirements that led to implementing your own
| language vs embedding lua?
| toyg wrote:
| I cut that particular gordian node, once, by simply embedding a
| JS engine, and explaining to them (and documenting) the bare
| minimum of the language that they needed to achieve their
| objectives. After a week of getting to grip with it, they loved
| it so much that they kept it into production for years, happily
| tweaking scripts as they needed.
|
| I should have charged them more than I did.
| mjr00 wrote:
| Depends on the project, but if you're handling these client
| requests anyway, this is not the best solution IMO, at least
| not for cloud/SaaS.
|
| The best solution is to have a good deployment process in place
| that makes it easy for you to make business logic changes using
| your language of choice, rather than building and maintaining a
| secondary language that is understood only by you.
|
| No need to create a DSL or compiler or worry about maintenance
| when you're just using the same tooling as everything else you
| use.
|
| Seen way too many times in my career developers who build the
| fun, interesting solution (DSL/compiler/parser) instead of the
| boring, but far more practical solution of fixing their
| deployment processes.
| yesbutum wrote:
| [dead]
| alanfranz wrote:
| Now the question is: did you make your client pay for each
| request _just as if you didn't build such dsl_?
|
| In my experience, most developers-as-consultants aren't good at
| that. So, when we're (rightfully) late, we probably reduce our
| margins. When we do a great job, we don't make the client pay
| for his own mistakes.
| tshaddox wrote:
| > did you make your client pay for each request _just as if
| you didn't build such dsl_?
|
| I'm not exactly sure what you're asking. Should you be
| charging clients for time that you _didn 't_ have to spend
| because you made some good decisions upfront? That strikes me
| as odd. If you agreed upfront on the cost for a certain end
| result, then sure, charge that full agreed-upon amount even
| if it took you less time than expected. But it seems odd to
| say "this change only took me an hour, but it would have
| taken 10 hours if I had made worse choices in designing this
| system, therefore I'm going to bill you 10 hours." After all,
| there's always a hypothetical situation where any given
| change could have taken arbitrarily more time if you had made
| worse decisions in the past.
| xelxebar wrote:
| > Should you be charging clients for time that you didn't
| have to spend
|
| Yes! Completing a job or task in 1 week is more valuable
| than completing it in 5. The lion's share of value we
| provide as developers and consultants is not output but
| battle-tested intuitions that let us navigate the giant
| solution space, so bad decisions we _don 't_ make is
| definitely hard-earned value servicing the client.
|
| A man once interrupted Picasso at his evening meal. Pulling
| a napkin from his pocket, the man said,
|
| "Could you sketch something for me? I'll pay you for it.
| Name your price."
|
| Picasso took a charcoal pencil from his pocket made a rapid
| sketch of a goat. It took only a few strokes, yet was
| unmistakably a Picasso. The man reached out for the napkin,
| but Picasso did not hand it over. "You owe me $100,000," he
| said.
|
| The man was outraged. "$100,000? Why? That took you no more
| than 30 seconds to draw!"
|
| Picasso crumpled up the napkin and stuffed it into his
| jacket pocket. "You are wrong," he said, dismissing the
| man. "It took me 40 years."
| jimvdv wrote:
| You are right most developers are not very good at that. I
| think it's important to stress that this goes both ways. You
| should not cut your margins if you take longer, but I also
| think it is a little unethical to bill more hours than one
| spends no? At least if you have a clear agreement to bill by
| the hour.
| awkward wrote:
| The way to ethically handle this is to have a clear and
| contractual minimum number of billed hours in a day. That
| will get you paid well for built in efficiencies like OP
| mentioned, and keep you paid fairly on more substantial
| change requests.
| thfuran wrote:
| It's outright fraud. It's clearly unethical. But your
| billing shouldn't be strictly hourly if you're building
| tooling to speed up your work.
| [deleted]
| derefr wrote:
| What you're saying is that it's inherently unethical (to
| yourself!) to accept hourly billing for work the whole
| point of which is exponential automation.
| guideamigo_com wrote:
| At a billion-dollar fintech, some engineer thought gRPC is too
| complicated. So, he built someything from scratch.
|
| A year later, there was a team of 5 engineers maintaining a half-
| baked implementation of gRPC. Good for him. Bad for the company.
| hsaliak wrote:
| Did that "some engineer" get promoted though? That's the real
| question.
| amoss wrote:
| Don't hate the player, hate the game.
| paulddraper wrote:
| Who makes the game?
| [deleted]
| vishnugupta wrote:
| There was a joke at Uber about beginning with a configuration
| management system and ending up with a version control system.
|
| And somewhere else about accidentally building a real time chat
| service (or was it email? Don't recall).
| david_allison wrote:
| > And somewhere else about accidentally building a real time
| chat service (or was it email? Don't recall).
|
| Maybe Zawinski's Law?[0]
|
| > Every program attempts to expand until it can read mail.
| Those programs which cannot so expand are replaced by ones
| which can.
|
| [0] https://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski's_Law
| random3 wrote:
| Given how easy it is to build a Turing machine, I'd argue that
| building something you don't envision is the 99% of the time rule
| and it's already captured by
| https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule "Any
| sufficiently complicated C or Fortran program contains an ad hoc,
| informally-specified, bug-ridden, slow implementation of half of
| Common Lisp."
|
| In fact I'd argue that it's much harder to not accidentally build
| something that you don't envision.
| bstpierre wrote:
| I accidentally built a kind-of compiler last year.
|
| It started as a few sed commands to merge TeX+code -> TeX for a
| book project. I ran these sed commands from a makefile. Life was
| easy.
|
| But then there were complications, and I needed to make slightly
| more sophisticated substitutions. So the sed commands moved into
| an awk script, run by the makefile. This was better than
| maintaining a handful of little commands that were growing on a
| weekly basis. Life was good.
|
| The transformations I needed kept growing a bunch of little
| variations, and the awk script became hard to maintain, so I
| rewrote it in go, with proper parsing and output. (And even unit
| tests, after the 2nd time I broke some output.) Designing it as
| almost-a-proper-compiler was 10x better than maintaining an ad
| hoc script. Life was great, even with the overhead of maintaining
| a separate processing tool.
| eternityforest wrote:
| ... how does this even happen? What bizarre use case doesn't
| allow for just using an off the shelf scripting language?
|
| I've only done anything like this once and not regretted it, and
| it's purely visual scripting, if this then that style. Anything
| that can't be handled by an event that triggers a list of
| actions, then stops when one returns False, I will hardcore a
| hack just for that feature.
|
| Most actions and triggers are responses to specific use cases.
| jcarrano wrote:
| The key is to know, acknowledge and accept where you are going,
| and to go boldly and deliberately- or not go at all. Say "this
| problem looks like making a small program so I'm making a mini-
| language". However, if you find yourself saying ""this looks like
| a database..." then stop there and please do not build a db.
| alexchantavy wrote:
| I work on a Python project where I need to take class definitions
| and generate database query statements because all ORMs that
| currently exist don't work for my needs. I'm currently doing this
| with string templates that I've defined by hand. Is there a
| smarter way?
|
| I've looked into some compiler-like tools (can't remember the
| specific ones, sorry), and from what I can tell their code
| generation phase looks very similar to mine in that they use
| string templates.
| aidos wrote:
| Why don't ORMs work for you? Have you looked at sqlalchemy?
| alexchantavy wrote:
| My project uses the Neo4j graph database and the ORMs
| available here aren't great: they don't handle batched writes
| so they are super slow, or they do weird hacks like requiring
| that you run a webserver to let them work, or they don't use
| managed transaction functions so that write operations aren't
| automatically retried for you.
| 89vision wrote:
| https://github.com/zio/zio-quill
|
| This library does exactly what you prescribe. Pretty sure under
| the hood it's using macros with string templates
| alexchantavy wrote:
| Very cool, thank you. Anything in Python?
| noelwelsh wrote:
| Scala macros and quasiquote templates do have some notable
| differences to pure strings. The two main ones are:
|
| - the value that's constructed has to be valid code - macro
| "hygiene" is maintained
| makach wrote:
| How embarrassing. I've done this. Programming is easy, making
| things easy is difficult.
| wrs wrote:
| Other needed topics in this series: "You have built a database",
| "You have built an orchestrator", "You have built an RPC layer",
| "You have built a build system"...
| dang wrote:
| And Greenspun's 10th rule I suppose.
| daniel-s wrote:
| For those that needed to Google like me:
|
| "Any sufficiently complicated C or Fortran program contains
| an ad hoc, informally-specified, bug-ridden, slow
| implementation of half of Common Lisp."
| hinkley wrote:
| In the age of horizontal scalability, what you've built is
| likely more like a nightmare version of Erlang, rather than
| Common Lisp.
| Detrytus wrote:
| You can see that with so called "Big Data" tools. Those
| which originated as databases (Mongo), ended up adding
| "Map-reduce" feature. Those which started as map-reduce
| tools evolved to support SQL (Hadoop->Spark). Those which
| started as SQL engines (Spark) added support for
| streaming, while those who started as steaming platforms
| (Kafka) added SQL support (KSQLDB). Traditional DB
| engines evolve to allow document data (Postgres with JSON
| column type). One more decade until one-tool-to-rule-
| them-all emerges :)
| namaria wrote:
| It's the Cassandra curse of computer scientists. Business
| people think they don't have time or resources for
| correctness, and end up forcing incorrect systems to be
| held by hand. Which, despite the high cost of this labor,
| is a decent trade-off because thanks to the internet
| business models have very low marginal cost per costumer.
| I think that the fact that Netflix has a very high
| marginal cost per costumer and actually hires very highly
| qualified programmers and pioneers systems designs
| illustrates my point.
| glonq wrote:
| And of course the classic "you have re-invented TCP"
|
| ...which often happens when somebody creates a UDP-based
| protocol and then adds their own reliability and robustness on
| top of it ;)
| kmeisthax wrote:
| To be fair to people reinventing TCP, the two transport
| protocols that will reliably traverse the Internet are often
| not great fits for most applications. Most people either want
| to process multiple streams or datagrams independently[0]
| (which TCP can't do) while having some amount of reliability
| guarantees[1] (which UDP can't provide).
|
| And if you don't have that problem then you're probably also
| reinventing HTTP as well as TCP.
|
| [0] without head-of-line blocking
|
| [1] ideally with controls over how reliable we want our
| messages to be. Real-time usecases like videoconferencing or
| multiplayer games tend to fail horribly over TCP.
| hckr1292 wrote:
| This is why QUIC/http3 is happening right?
| kmeisthax wrote:
| Yes.
| jraph wrote:
| The last "You have built a browser" lead to Ladybird and is now
| sponsored. (I've heard or read its creator say that its web
| engine was initially meant to display help pages in Serenity,
| or something like this)
| hinkley wrote:
| Would you consider orchestrator and scheduler as two separate
| things or variations on a common theme?
| swyx wrote:
| here! https://dx.tips/oops-database
| throwway120385 wrote:
| lol I have accidentally written databases for embedded
| systems many times. Usually the team lead or architect of the
| project doesn't see the need for it until we already have too
| much data and we're close to releasing the product, at which
| point it's impossible to retrofit.
| dang wrote:
| Related:
|
| _Dear sir, you have built a compiler_ -
| https://news.ycombinator.com/item?id=29891428 - Jan 2022 (175
| comments)
| swyx wrote:
| I also wrote a post inspired by this post: Oops, You Wrote a
| Database!
|
| https://dx.tips/oops-database
| jacquesm wrote:
| And usually that's a crappy database and a crappy compiler.
| Because they didn't start out as such.
| Detrytus wrote:
| Totally offtopic but I can't help but wonder why this guy has a
| site with .pl TLD. He seems to be based in the US, not Poland.
| Does he think "pl" stands for "programming languages"? :)
| hermitdev wrote:
| > Does he think "pl" stands for "programming languages"?
|
| More likely pl -> perl, I think.
| tmtvl wrote:
| Well, vanity URLs have been a thing for a while now. E.g.
| lobste.rs.
| paulddraper wrote:
| FWIW I see lots of non-Montenegrin people with .me addresses.
| [deleted]
___________________________________________________________________
(page generated 2023-08-17 23:00 UTC)