[HN Gopher] Selecting a programming language can be a form of pr...
___________________________________________________________________
Selecting a programming language can be a form of premature
optimization
Author : genericlemon24
Score : 83 points
Date : 2021-11-15 08:44 UTC (14 hours ago)
(HTM) web link (snarky.ca)
(TXT) w3m dump (snarky.ca)
| resonious wrote:
| It'd be great if we could start measuring how much $ it costs to
| run our services vs how much developer salary $ it costs to
| maintain and build them.
|
| If Rust or something really saves money because it means we don't
| need 50 extra web workers for the same TPS then I don't think
| it's "premature optimization" - unless the dev salary is too
| much!
|
| As it stands we're all just saying stuff with no data or evidence
| to back it up.
| mynameisash wrote:
| One of my anecdata:
|
| My team does a ton of large-scale processing (most of which is
| ETL), and we have TONS of servers at our disposal. We
| effectively throw hardware at our compute, and even then, jobs
| can take 5-10 hours to run. No big deal - just run the job
| before you call it a day and it's done when you start tomorrow
| morning. Or when you have daily jobs, you schedule it and it's
| just done whenever it's done.
|
| I can't tell you the number of times I've looked at a job
| someone wrote, made a 5 LOC change in 2 minutes, and cut
| processing from 8 hours to 5 hours. I literally had one that
| went from 5 hours to 5 minutes.
|
| These extremely simple investments save a lot of money, but
| it's a problem that's spread across hundreds of different jobs,
| and it takes so much effort to convince people that we should
| invest in it. A daily job that costs $1500/mo to run (based on
| amortized hardware costs) needs only be optimized once, and
| those savings are yours forever. But I've had the conversation
| that is effectively: my time is more costly than a measly $1500
| job, so it's okay if the job is expensive.
| Karrot_Kream wrote:
| It's not going to change any time soon, IMHO. Programmers have
| too big of a culture of individuality, blogging, and
| intellectual daydreaming to actually try to come to consensus
| on terms of art. On top of which businesses are loathe to share
| data because it's tied very closely to their competitive
| advantage. Programming culture is more about memetic and shared
| ideas and uses tradition to select winning strategies. Until
| this changes, it'll continue to be discordant voices talking
| over each other. Ideology [1] is a great talk about this.
|
| [1]: https://www.destroyallsoftware.com/talks/ideology
| Jensson wrote:
| > Programming culture is more about memetic and shared ideas
| and uses tradition to select winning strategies.
|
| Sounds exactly like business management culture. Makes sense
| though, developer productivity is first and foremost a
| business management issue so discussions about it ought to
| follow the same path as business management. And business
| management love their anecdotes, great person citations and
| jargon.
| wongarsu wrote:
| With silicon valley salaries maybe the trade-off is more
| between Developer time vs. Devops time. Those 50 extra web
| workers might not matter much on their own, but someone has to
| set them up, orchestrate them and deal with all weird
| interactions that arise.
|
| And of course the further you get from silicon valley the more
| the hardware/service costs matter. A developer in Sofia easily
| earns an order of magnitude less than one in San Francisco,
| which shifts the whole optimization tradeoff a lot.
| Jensson wrote:
| Silicon valley companies tend to have more users and
| therefore more servers, so saying they mostly don't need to
| care about server costs should be wrong. I worked at a small
| team at Google that had many thousands of servers, those
| servers cost way more to run than the engineers working on
| them. Compare that to a small company I worked at where one
| server with an extra as fallback was more than enough, there
| performance didn't matter at all even though salaries were
| much smaller.
| eptcyka wrote:
| I really don't find writing in Python to be _that_ much more
| productive than using many other modern language.
| letwhile wrote:
| While I see a point in this article, I personally run into
| performance problems with python at a very early stage nearly
| every time. The overhead of starting to use some ffi is huge.
| Suddenly you have two projects, with two different build systems
| etc. Way faster to just use a performant language. Besides this,
| dynamic typing gets frustrating quickly, as the loc rise, and
| distribution is not very fun. I love the language, but won't use
| it anymore for any "serious" project.
| lewisjoe wrote:
| Very true. People discriminate against dynamic languages with
| usually two measurements: runtime errors and performance.
|
| But we often overlook the fact that it's possible to scale those
| two, incrementally. With today's convenience in interop, we can
| always start with a prototype friendly technology and later
| switch tech for real bottlenecks.
| revskill wrote:
| Agree with title. In 99% of business use cases,
| typescript/nodejs/reactjs is enough.
| woah wrote:
| I thought this was going to be satire, but no, it seems that he
| is completely serious.
| saila wrote:
| Is the premise unreasonable? Of course, it depends on what
| you're doing--there are certainly use cases where you _know_
| Python or a similar language wouldn 't be appropriate (for
| performance or other reasons)--but it seems to me that there
| are plenty of scenarios where using a "faster" language up
| front isn't warranted, and you may never need to switch.
| number6 wrote:
| Writing Code at all can be a Form of premature optimization
| mro_name wrote:
| ...or even doing it with computers.
|
| Or even doing it at all.
| abrichr wrote:
| When it comes to starting a business, this is very good and
| often overlooked advice.
| eesmith wrote:
| Yeah, I remember working on one of the Project Euler problems,
| getting the right answer, then seeing someone else had solved
| it with a pocket calculator in less time than I took to write
| the program.
| JohnWhigham wrote:
| Ain't it funny how these programming exercise websites are
| essentially the adult versions of busywork?
| deckard1 wrote:
| It was a rather embarrassing realization for me that the
| Fibonacci sequence has a closed-form expression and can be
| calculated without an algorithm. I had always seen it done in
| pedagogical fashion as an exercise in explaining recursion
| and memoization and just assumed it had to be done that way.
| The downside of using toy/straw problems...
| raxxorrax wrote:
| Selecting a language is like selecting a tool. It would be worse
| to select a language that on first sight fits the problem if you
| don't have any experience with it.
|
| To be honest, today there are many viable solutions with
| different languages. I would not recommend to start with a plan
| where you have to reimplement the system in another language at
| some point. Experience tells me that almost none of such projects
| survive.
| olivierduval wrote:
| Actually, writing a POC or a MVP is quite different to writing a
| "long run" production product. In the first case, coding speed is
| surely a must - and any untyped (or loosely typed) language might
| be good enough - but in the last case, when dev teams have to
| maintain some code in the long run, during multiple versions,
| with turnover, typed (or pedantic) language might be easier to
| work with because the language already include some kind of
| documentation (types) and automatic checks (type verification).
|
| Moreover, the availability of "average" (and "cheap") programmers
| matters a lot in the long run: if only genius can maintain your
| system, then you'll have problem in the long run because you'll
| either need to keep them at all price or need a lot of time to
| replace them. So, in the long run, you should better use a wide
| audience language with a lot of available programmer (even if
| they are "average") than a specific language requiring good
| programmers. However, for an MVP, you can recruit a genius
| programmer using the fastest tool for the job.
|
| Obviously, some domain are more oriented toward some language...
| and for ML for example, python is quite a good choice because of
| the libs (as Java could be for - lets say web servers)
|
| So it matters a lot what your system will be used for and how
| much time you will require it to run before needing to rewrite it
| from scratch
| brabel wrote:
| This blog post assumes Python is more productive than other
| languages. People like to make this claim about not only Python,
| but many other dynamic programming languages, specially Ruby, PHP
| and Lisp... but there's very little evidence to support that...
| programmers tend to be most productive in whatever language they
| know best. If they know both Python and Java equally well, I
| would bet they would be almost exactly as productive in either.
| moksly wrote:
| I spent 7 years working with C#, I like C# and I think what
| they've done in the past few years is amazing.
|
| I'm much more productive in Python. Not as in "I feel" more
| productive, but as in measurably more productive.
|
| Where Python falls short of something like .Net and C# is that
| I probably wouldn't have been more productive in Python if I
| didn't have 7 years experience with a rather strict
| environment, and as far as using Python on major projects, well
| let's just say that there is a reason people use TypeScript
| instead of JavaScript and Python doesn't fall into that
| category.
|
| But for most programming and for most minor systems or
| services, Python is just wildly good.
| okwubodu wrote:
| Same here. I do game development in C# and Python is my goto
| for sketching POCs for complex mechanics, despite never
| formally learning the language. It could easily be the
| difference between spending the whole day vs two hours on the
| same issue.
|
| > there is a reason people use TypeScript instead of
| JavaScript and Python doesn't fall into that category.
|
| I thought the dynamic vs static typing debate was pointless
| until I joined a large Python project. Now I pretty much
| consider anyone that can keep up with one superhuman.
| xapata wrote:
| I think part of the problem is that many people who are
| used to static type analysis get lazy with naming and
| interface design. Toss one of them into your team and
| they'll "prove" the need for static type analysis.
| iimblack wrote:
| Possibly. IME it's quick Python POCs that didn't bother
| with good practices that turn into critical production
| services without anybody going back and applying good
| coding standards.
| harpiaharpyja wrote:
| I also spent many years working with C# before working with
| Python professionally. I've also spent a lot of time with
| Lua, though not for paid work.
|
| The second point matches my experience with dynamic languages
| as well. To use them well really takes a certain level of
| discipline, but it pays off. It's why I'm a bit skeptical of
| Python as a beginners language, which it sometimes has been
| touted as.
|
| About large projects, presently I am working with Python on a
| somewhat large project (currently 53k sloc, I guess large is
| relative). Its been great, though we are pretty strict about
| almost absolutely everything being type-hinted.
| goohle wrote:
| 53k SLOC is tiny project. 5 senior developers at 1k SLOC
| per day will generate 53k SLOC in two weeks, just one
| sprint.
| mynameisash wrote:
| How many people are writing 1k SLOC/day? And doing so
| consistently? If you're banging out that much code, I'd
| be severely concerned for the quality of the code itself
| as well as overall design.
| goohle wrote:
| Senior developers (really seniors, not 3 year seniors)
| can trivially produce such amount of code, because they
| wrote almost same code dozen times already.
| mynameisash wrote:
| My point isn't about who _can_ do it but who _does_. If
| anyone on my team was writing anywhere near 1k LOC/day,
| I'd think it was a serious problem. If the whole team was
| pumping out 1k LOC/day/person, I have to imagine I'd be
| getting as far away from the entire team/org as possible.
|
| Maybe not the best example, but I used to work with a guy
| who had pretty large scripts that he was writing for one
| of our projects. It turns out it was mostly copy-paste,
| so it sort of got the job done -- except he fixed bugs in
| one place but not in the other. The whole thing was a
| huge mess to understand and maintain. And he was a senior
| engineer at the time (now principal, to my great
| amazement).
| smt88 wrote:
| > _programmers tend to be most productive in whatever language
| they know best_
|
| This is true, but most software is built by multiple people
| (for corporations) and will eventually be maintained by
| multiple _other_ people. Whether an individual is "most
| productive" in a language is rarely important.
| flohofwoe wrote:
| Sometimes you just want to throw entirely differently shaped
| data into the same array or dictionary and be done with it
| instead of creating a "proper" type-safe design which most
| likely means writing lots of pointless boilerplate code. IMHO
| that's the whole point of a scripting language like Python, to
| write quick'n'dirty scripts, not "real programs".
| npsimons wrote:
| > but there's very little evidence to support that
|
| Not if you don't go looking for it: http://www.norvig.com/java-
| lisp.html
|
| ETA: And don't get me started on Design Patterns:
| https://norvig.com/design-patterns/design-patterns.pdf
| taylodl wrote:
| I've been programming for nearly 40 years now and I've used all
| the popular languages in that time (popular for microcomputers)
| and I've mastered a few. I just started Python for the first
| time last year. Am I a Python expert? No. But I can tell you
| right now I can get a project done faster in Python than using
| any language in which I'm an expert. It's that productive.
|
| Does that mean Python has all the performance you would ever
| need? No - we know it doesn't, but we also know that you rarely
| need that kind of performance and when you do it's usually
| localized to a very narrow portion of your code. Take that
| portion of code and implement in C and optimize to your heart's
| content. Even with that you'll still get your project done much
| faster.
|
| Python is the tool of choice for those developers who just want
| to get stuff done.
| lvass wrote:
| I've used Python for 15 years and I'm far more productive
| with Clojure. Don't generalize.
| wallscratch wrote:
| What features of clojure do you feel make you more
| productive?
|
| I'm primarily a python user, but I spent time writing code
| in Ocaml to understand the potential benefits, but... (I
| would love to have my mind changed)... it feels like many
| of the features people touted about Ocaml have already made
| their way into python...?
|
| 1. Immutability / referential transparency seem more like
| nice-to-haves for a codebase, rather than the real reason
| people tout FP...?
|
| 2. Sum/product types and pattern matching are being added
| soon to python
|
| 3. Mypy is starting to gradually enable python typing,
| although I assume it's still a work in progress
|
| 4. Python allows us to use map/reduce, and to pass
| functions around as arguments...?
|
| I want to understand the potential benefits of FP more, but
| my experience with Ocaml hasn't shown me great improvements
| yet. Open to having my mind changed.
| ubercore wrote:
| Presenting a specific experience refuting an earlier
| generalization is reasonable here, I think.
| shadowfox wrote:
| I mean it is not _just_ mentioning a specific experience
| to refute a generalization when there is also this:
| "Python is the tool of choice for those developers who
| just want to get stuff done."
| taylodl wrote:
| Admittedly that last bit was a bit inflammatory, but I've
| gotten tired of the constant arguing over languages at
| where I work. I'm on a bit of a rogue team that's just
| adopted Python and are getting stuff done. Other people
| are just now starting to pay attention because we're
| getting so much done and now work is headed our way as a
| result. Wait until they find out we're not using any of
| the languages they're bickering about and are using
| _lowly Python_! Still, as awesome as my team is I think
| they 'd draw the line at Clojure.
| taylodl wrote:
| I agree with you on Clojure, but I'm not going to be able
| to get my team to adopt Clojure - unless you're going to
| tell me how you convinced your team to adopt Clojure! :)
| rambojazz wrote:
| Granted, people come from different backgrounds, and "the right
| tool for the right job" always applies. But it's undeniable
| that generally speaking, ie. not considering any specific cases
| or requirements, some languages are definitely more productive
| than others.
| josephg wrote:
| I doubt it, though if anyone has data I'd love to see it. At
| the moment I'm quite comfortable in both javascript and rust.
| But I find javascript (/ typescript) noticeably more productive
| for small to medium projects. I can get more done with less
| work. For a variety of reasons it seems to take more work to
| write good rust code than good javascript code.
|
| This isn't a knock on rust - I just think rust trades off
| programmer productivity for correctness and performance. And it
| shows, on both sides.
| smt88 wrote:
| > _But I find javascript ( / typescript) noticeably more
| productive for small to medium projects._
|
| This seems like a pretty apples-to-oranges comparison. Rust
| is intended to do things you couldn't/wouldn't use JavaScript
| for.
|
| If you're talking about a small-to-medium project where you
| don't care much about mishandling memory, of course
| JavaScript is going to be more productive. Rust is forcing
| you to tell the compiler a lot of things that JavaScript
| assumes you don't care about (and is, most of the time,
| correct).
| KronisLV wrote:
| > I doubt it, though if anyone has data I'd love to see it.
|
| I recall a study being quoted in my university courses, which
| described the amount of code needed to get certain things
| done between different languages, which showed that Python,
| Ruby and others are on the less verbose side, the reasoning
| being that on average they'd also be more productive.
|
| This seems to coincide with my personal experience, where
| JavaScript with React was much easier to work with in smaller
| projects, whereas using TypeScript with React lead to much
| slower development because of all the typing that needed to
| be handled, especially in cases of union types. Now, one can
| say that it's worth the effort to be more confident in your
| code doing what you want it to both now and after X months,
| much like you could sometimes prefer the type systems of Java
| or .NET over Python, Ruby or PHP, but in my eyes those
| tradeoffs always come with slower development velocity.
|
| The sad thing, however, is that DuckDuckGo (and possibly
| other search engines) failed to return the original study or
| anything like it, only resulting in low quality blog content:
| - https://duckduckgo.com/?t=ffab&q=programming+language+compa
| rison+amount+of+code - https://duckduckgo.com/?q=progra
| mming+language+comparison+verbosity - https://duckduckg
| o.com/?q=programming+language+lines+of+code+java+ruby -
| http://libgen.is/scimag/?q=programming+language+verbosity
| -
| http://libgen.is/scimag/?q=programming+language+lines+of+code
|
| Anyone have any better search queries for this? Any idea why
| the search result quality is generally so low? Any ideas
| which study it might have been referencing?
| brabel wrote:
| The research on this topic has been discussed on HN
| multiple times. It is indeed hard to look up previous
| discussions, but the conclusion is always inconclusive...
| one thing everyone seems to agree nowadays is that
| measuring LOC is a bad proxy for programmer productivity,
| specially when you take code maintenance into consideration
| (which very few research papers do), and that types do have
| a small, but measurable effect on improving quality (though
| whether that's at the cost of productivity is still unclear
| as far as I know - yes, programmers need to spend a bit
| more time to declare their types, but without them they
| have to spend more time every time they need to call
| anything).
| eesmith wrote:
| Google Scholar is a better method to find published papers.
|
| The first result for "program language comparison", at
| least when I view https://scholar.google.com/scholar?q=prog
| ramming+language+co... is: Prechelt, Lutz. "An empirical
| comparison of seven programming languages." Computer 33.10
| (2000): 23-29.
|
| There's all sorts of work which cite that paper, like "An
| empirical investigation of the effects of type systems and
| code completion on api usability using typescript and
| javascript in ms visual studio".
|
| For fun, here are some of those citation which themselves
| use the phrase "An empirical study" or similar in the
| title:
|
| "An empirical study on the impact of static typing on
| software maintainability"
|
| "An empirical study of the influence of static type systems
| on the usability of undocumented software"
|
| "Do developers benefit from generic types? An empirical
| comparison of generic and raw types in Java"
|
| "An empirical study on C++ concurrency constructs"
|
| "An empirical study on the factors affecting software
| development productivity"
|
| "An empirical study to revisit productivity across
| different programming languages"
| blooperdoop wrote:
| You'd bet wrong. Say I want to add two numbers and ship it to
| prod. In Python, that's one line and an scp. In Java, that's at
| least three, a compile step, and an scp. By definition I'm more
| productive in Python due to the language tradeoffs. That's
| reducing to absurdity but the principle holds to a solution of
| any complexity, particularly in SRE/ops.
|
| This really rears its head when Go and Rust shops say "oh let's
| do all our operational scripting in the same language, too." It
| seems nice in theory but is a complete mess in operations. As
| an SRE manager, I have data to back up the assertion that a
| compile step gets directly in the way, particularly in an
| outage situation (we need to recompile the script with a flag
| to change it - and we are usually pushed toward CI/CD for that
| outcome). If you are asking this of your operations teams, they
| absolutely hate you regardless of their experience with the
| language.
|
| I have 21 years writing Java and three writing Python and I'll
| reach for Python for "move files from directory to other
| directory based on heuristics" loooooong before I touch Java
| for the same purpose, and that's a high bar to even get there
| beyond doing it in a shell. Reducing this discussion to
| familiarity with a language is a complete red herring and
| dismisses every argument the other side has with "if you were
| just better at Java this wouldn't be an issue," which is flat
| out wrong.
|
| Right tool, right job. I see I'm -2 before I'm finished editing
| typos which I think speaks to the article's point about
| language ideology more than anything.
| Dobbs wrote:
| > Say I want to add two numbers and ship it to prod. In
| Python, that's one line and an scp.
|
| This example ignores what is actually involved in production
| code. Yes there is a compile step involved in things like
| Java. In python you have to manage your dependencies on the
| deployment box.
|
| In Java (minus the runtime), Go, Rust, etc they all support
| managing dependencies on the client side. In my experience
| the messiness of that in python and ruby far outstrips the
| complexities of a compile step in the makefile or CI
| pipeline.
| blooperdoop wrote:
| Correct. I reduced the comparison to absurdity and
| intentionally overlooked several concerns, and pointed that
| out, to note that binary yes/no is more nuanced here and
| that the alternative opinion can be built from first
| principles without invoking familiarity with a language. I
| apologize that I led you to respond to an absurd argument.
| I'll try harder to note it as an absurd argument in the
| future - should I use a better word than absurdity?
| dtech wrote:
| Yes, you can write your 2 line program faster, no one is
| disputing that.
|
| But that's not the real world, in real world you have to
| change and debug large amounts of existing code, and
| collaborate with colleagues. There's little conclusive
| research on it, but so far everything hints at typed
| languages being superior for that. Heck, untyped "script-
| like" languages tend to evolve into the direction of added
| types and build systems (e.g. Typescript, Python type
| annotations, Babel, Webpack etc.)
| blooperdoop wrote:
| Actually, the person I responded to did dispute exactly
| that by claiming the limiting effect of language choice is
| gated by experience. Read what they said again. That's also
| reducing the argument to absurdity, which I pointed out,
| and which apparently did nothing to dissuade you from
| taking it as my actual position.
|
| You're also explaining a single sector of software usage to
| me (i.e., the revenue SaaS of the company) as "the real
| world," which itself dismisses every other aspect of
| software development in a shop. Again, including SRE and
| ops scripts, which are almost always inappropriate in a
| mainline, industrial, compiled language. Please try to
| avoid explaining your point with the beginning assertion
| that those who disagree with you do not live in the real
| world.
| brabel wrote:
| I am OP you're talking about... you misinterpreted my
| point completely.
|
| I was saying that Python and other dynamic languages are
| not demonstrably more productive for writing general
| purpose software... as I did not mention scripting at
| all, I thought it would be obvious that this is what I
| meant.
|
| > Read what they said again. That's also reducing the
| argument to absurdity, which I pointed out, and which
| apparently did nothing to dissuade you from taking it as
| my actual position.
|
| This is absurd. What "they" said was _programmers tend to
| be most productive in whatever language they know best_.
| How the hell is this disputing "you can write your 2 line
| program faster (in Python), no one is disputing that."??
|
| With all the due respect, you need to work on your text
| interpretation skills. Do you understand that
| "programmers tend to be more productive" is not the same
| as "programs are always, without exception, more
| productive"?
| blooperdoop wrote:
| It's pretty ironic that I was referring to me reducing
| the argument to absurdity and you're going after me "with
| all due respect" for my text interpretation skills. You
| didn't think it weird I was talking about pointing
| something out and making an argument _in your point_ , by
| your interpretation, before you launched off on this
| crusade to defend your honor?
|
| With all due respect, read better and save the lecture.
| rich_sasha wrote:
| I used to work at a C++ for prod, Python for prototype place.
| Time and time again, a 1-2kloc Python prototype, the bulk of
| which was written in a week, would take months to translate to
| C++, at times longer than the whole process of
| research+prototype.
|
| I think it was 2 things: one, it is genuinely harder to write a
| valid C++ program. The language imposes non-trivial constraints
| on the programmer, for better and worse, and you have to
| navigate them for the thing to even compile. But two, C++
| invites pedantry. I heard endless discussions about "ooh,
| template meta-programming", "struct or class", "but is this
| idempotent", "you should be using move semantics", "why on
| earth is this a unique_ptr".
|
| I'm not saying C++ is worse, or that its trade-offs are not
| worth it, but for sure, from my experience, translating an
| algorithm into code is done far more quickly in Python than in
| C++. YMMV
|
| Ah and also, this was definitely not a question of deficient
| C++ coders. The hiring standards for C++ were so high we were
| always short of devs. Meanwhile, Python prototypes were usually
| written by part-time ex-academia Python dabblers.
| munificent wrote:
| _> Time and time again, a 1-2kloc Python prototype, the bulk
| of which was written in a week, would take months to
| translate to C++, at times longer than the whole process of
| research+prototype._
|
| I've heard reports like this for many years.
|
| My pet theory is that the discrepancy here is ~10% static
| types and ~90% memory management. Having a GC (or refcounting
| or whatever) fully baked into the language so that you don't
| have to think _at all_ about how values are passed around and
| stored is a monumental productivity boost. Possibly the
| biggest win in software engineering productivity in the
| history of the field.
| ptx wrote:
| Absolutely. I recently translated Microsoft's example code
| for using the Windows crypto API from C++ to Python, and
| huge chunks of the code melted away as it was mainly
| concerned with freeing buffers. Exceptions also helped.
| professoretc wrote:
| > But two, C++ invites pedantry. I heard endless discussions
| about "ooh, template meta-programming", "struct or class",
| "but is this idempotent", "you should be using move
| semantics", "why on earth is this a unique_ptr".
|
| Interestingly, I've consistently had the opposite experience.
| In C++, when I ask, "how do I do X" there will be a few
| different options, but pretty much any of them will do. As
| long as you avoid things that are definitely UB, you're fine.
| In Python, I'll find endless religious arguments about which
| is the most "Pythonic" method. Whenever I try to just hack
| out Python code to just get things done, I always feel like
| I'm being judged, because usually the quick-and-dirty method
| is far from the most Pythonic.
| twobitshifter wrote:
| I think it all depends on who does your code reviews. I
| agree with GP that c++ has religious adherents to the one-
| true-undocumented-in-their-head standard who will make
| writing it a nightmare. In python the philosophy is
| supposed to be - there is an obvious way to do things and
| that's the right way. However this does fail from time to
| time.
|
| Ruby on the other hand embraces many different syntaxes to
| do the same thing, but they're all accepted as correct in
| the traditional view.
| xscott wrote:
| I wonder how your C++ would've gone if you had a C++ library
| of Python-like data types. Something like a 'variant' type
| that could hold arbitrary integers, strings, and dicts or
| lists of those (or anything else you commonly used).
|
| You would initially trade some of the C++ performance, but
| the code shouldn't be more complicated or difficult to write
| or compile than Python. Then, you'd only need to change the
| performance sensitive parts to use _real_ C++ arrays or
| structs.
| xapata wrote:
| Isn't that what Python is? Except C and not C++.
| xscott wrote:
| No, because you wouldn't have to pay the virtual machine
| overhead (among others). Another possibility would be
| using Cython - write the prototype in pure Python, then
| Cythonize the parts which need to run more quickly.
| Kye wrote:
| I thought the C++ tendency was to optimize performance
| sensitive parts with C or assembly.
| taylodl wrote:
| C++ _is_ C. We used to optimize with assembly but
| nowadays with a processor 's superpipelined architecture
| and the compiler's global register optimizations and so
| forth your inlined assembly is more likely to throw a
| monkey wrench into the whole works and make everything
| run _slower._ It 's a better use of your time to look for
| a better algorithm.
| khoobid_shoma wrote:
| Not really. C++ is not C, nor faster than C. e.g. TCO in
| C++ is awful due to ctor/dtor (could be mitigated by some
| compiler hints if available).
| typon wrote:
| This is something I do except the slow parts are in Python
| and the fast parts are in C++ with pybdind11 connecting the
| two worlds
| gpderetta wrote:
| I think a large part of the difference is the prototype vs
| prod. I have written prototypes in C++ in a few weeks that
| still took months to go to production. It is just that
| production-ready code require significant more ceremony, more
| config files, more complex error handling, code reviews and
| of course feature creep.
|
| I still agree that Python can be much more efficient to write
| than C++. I just wish it wasn't so unbearably slow.
| rich_sasha wrote:
| I would mostly agree, though not entirely. Continuing with
| the anecdata, at times I had enough waiting and put my
| prototype into prod, with significant hardening. I can't
| say I always got it right, but usually 3-5 days of very
| focused poking made the thing at least reliably run at the
| desired cadence, with rare but loud and graceful failures.
| For sure though, there is a tax for doing it all
| "properly".
|
| Even then though, I have the impression that Python just
| makes it easier to write it all more quickly. It just
| doesn't invite you (as much) down rabbit holes of doing
| things even more properly.
| andai wrote:
| _There should be one-- and preferably only one --obvious way
| to do it._
|
| -PEP 20 -- The Zen of Python
| michaelcampbell wrote:
| Hold my beer.
|
| -- Django
| bierjunge wrote:
| Amateurs.
|
| -- Perl
| jorgeleo wrote:
| Slow kids
|
| -- assembler
| uranusjr wrote:
| _Although that way may not be obvious at first unless you
| 're Dutch._
|
| --The next line
|
| This ZoP line is by far the most misused cliche of all
| things Python. How is it determine that "prototyping in C++
| and rewriting in C++" is not the obvious "one" way? Because
| you don't like it? A Zen is intentionally self
| contradicting to allow introspection, not judge others :)
| kitd wrote:
| _But two, C++ invites pedantry. I heard endless discussions
| about "ooh, template meta-programming", "struct or class",
| "but is this idempotent", "you should be using move
| semantics", "why on earth is this a unique_ptr"._
|
| I wonder what the result would have been in eg. Go or Rust,
| where that level of conversation doesn't exist. True, others
| would have taken place instead, but at a higher level nearer
| the design, where they should have been happening anyway.
| tcbasche wrote:
| > Python prototypes were usually written by part-time ex-
| academia Python dabblers
|
| This seems like a monumental level of operational waste. I'd
| love to hear more about this particular setup
| rich_sasha wrote:
| Well, part-time programmers. Full-time employees, doing
| mostly semi-academic stuff.
| CJefferson wrote:
| I think different languages do have sweet-spots, and it often
| has to do with library availablity (for me at least).
|
| I consider myself as someone who knows "far too much about
| C++", but recently when I had to spider some webpages and read
| some values out of them to populate a database, I did that in
| Python because it's a handful of lines due to high quality
| libraries that all work nicely together. I honestly wouldn't
| know how to do that in a short amount of time in C++.
| brabel wrote:
| I admit that Python is more productive than C++, but the
| reason is mostly to do with memory management... a language
| with a GC, like Java or JavaScript, would use a very similar
| level of abstraction as Python and therefore, should "cost"
| about the same to write.
|
| Python and other dynamic languages don't use static types and
| that may give them a (very) small advantage in productivity
| as well, but only for very small programs... as program size
| increases, in my experience, statically typed languages take
| the lead in productivity... as OP is about writing general
| software, not just small scripts, I really have a hard time
| agreeing that Python is either the best tool for the job, or
| the most productive tool at all.
| WithinReason wrote:
| It's more than that. For example, converting a string to
| uppercase in C++: #include <string>
| #include <cctype> #include <algorithm>
| #include <iostream> int main() {
| std::string s("hello");
| std::transform(s.begin(), s.end(), s.begin(),
| [](unsigned char c) { return std::toupper(c); });
| std::cout << s; }
|
| Same thing in Python:
| print('hello'.upper())
| capitalsigma wrote:
| C++ is forcing a style on you that encourages you to
| avoid a copy. You could, of course, have a library like:
| std::string ToUpper(const std::string& str) {
| std::string as_lower(str);
| absl::c_transform(str, as_lower, std::toupper);
| return as_lower; }
|
| to enable: int main() {
| std::cout << ToUpper("hello"); return 0;
| }
|
| but this would be no good! You have chosen an API that
| forces your users to be gratuitously inefficient. Maybe
| that is OK for your personal project, but an API like
| that has no business in the standard library.
| WithinReason wrote:
| You just explained why C++ will never be as productive as
| Python.
| capitalsigma wrote:
| No one is stopping you from writing your C++
| inefficiently if you choose to. I am skeptical that
| spending a bit of time thinking about ownership/lifetimes
| is really such a big hit to productivity.
| Hizonner wrote:
| That takes advantage of the fact that Python puts a bunch
| of relatively obscure functionality in scope by
| default... thus increasing the chance that programs will
| use the wrong method or rely on stuff they don't really
| intend to. Case manipulation isn't something most modern
| programs should even be doing at all, since it's almost
| impossible to internationalize it.
|
| In fact it's really _mostly_ name space management.
| Python doesn 't require you to pull in "upper" because it
| starts with a relatively big name space. It doesn't
| require you declare the main program because you're
| relying on the fact that Python executes a file on
| loading it (and that reliance is arguably bad Python
| style). All of the "std::" stuff is a name space
| management choice.
|
| ... and the rest has nothing to do with static typing
| either. The way the code is indented on multiple lines is
| a stylistic convention. The transform primitive requiring
| you to specify bounds is a library choice, related to the
| language's apparent lack of a universal way to map over a
| container. I'm not sure _why_ you need the lambda. The
| rewrite in place is memory model stuff inherited from C.
|
| In Haskell, which is fully compiled and is about the most
| rigid static typed language you could ever imagine:
| import Data.Char(toUpper) main = putStrLn (toUpper
| <$> "hello")
|
| If you'd asked for something that didn't require oddball
| functionality, then that could probably also have been a
| one-liner.
| yetihehe wrote:
| I've done big desktop program in python. For some parts, we
| just used modules written in C, because code was easily 20
| times faster than python, I was even surprised at how fast
| simple blits could be on modern hardware without any
| graphics card acceleration. But python version was several
| times faster to write.
| CodeGlitch wrote:
| > Python and other dynamic languages don't use static types
| and that may give them a (very) small advantage in
| productivity as well, but only for very small programs...
| as program size increases, in my experience, statically
| typed languages take the lead in productivity... as OP is
| about writing general software, not just small scripts, I
| really have a hard time agreeing that Python is either the
| best tool for the job, or the most productive tool at all.
|
| As a counter point to this - Python has a large number of
| C-based libraries with excellent bindings. To take Numpy
| has an example, you can the static-types and speed of Numpy
| to do all your maths, whilst Python acts as the manager.
| Using Numpy "feels" like normal Python, you don't really
| feel like you're working at native-level, but you get all
| the advantages of native speeds and memory usage.
|
| And to your other point, I've worked on large C++-based
| projects which were a complete mess, especially when you
| had to make any changes to existing code. I'm not sure the
| language has as much an affect here as just using the
| correct design principles.
| deltaonefour wrote:
| I'm a C++ programmer and a python programmer. I assure you your
| conclusion is incorrect. Python with type checking is by far
| more productive. Many people who know python and another
| language can testify to this general truth.
| simiones wrote:
| C++ is probably the least productive non-joke language ever
| created: it has deterministic destruction which requires
| (manual) ownership tracking, it has an extremely slow
| compiler with horrible error messages, it has massive
| performance differences between debug and optimized builds,
| it's focus on performance brings lots and lots of extraneous
| concepts to its libraries when just seeking to get something
| working (std::allocator, std::string_traits and many other
| similar examples).
|
| Comparing Python to Java, C#, Haskell, Erlang, Go, maybe even
| C or Rust, would show a much smaller performance difference.
| shepardrtc wrote:
| > requires (manual) ownership tracking
|
| Have you tried RAII?
|
| Would you like to know more?
| https://stackoverflow.com/questions/2321511/what-is-meant-
| by...
| simiones wrote:
| RAII is exactly what I mean by "manual ownership tracking
| and deterministic destruction". RAII is better than
| completely manual memory management (C style
| malloc/free), but it still requires you to design your
| program such that every piece of memory is owned by some
| pointer with the correct properties. You have to choose
| between copying, bare pointers/references, unique_ptr and
| shared_ptr whenever two pieces of data are related to
| each other, or whenever you pass a piece of data to
| another function.
|
| Rust is a little better since at least the ownership
| concept is known to the compiler and automatically
| enforced.
|
| (Tracing/Copying/Compacting) GC is much easier since this
| entire concept goes away. You always pass references to
| data, and the physical storage is "owned" by the GC
| itself. It's also much faster for certain workflow
| patterns, though it always consumes more memory than
| deterministic destruction schemes.
| isubasinghe wrote:
| I do a lot of Rust programming, I personally find Rust more
| productive right now.
| eesmith wrote:
| The classic, now quiet dated citation, is "An empirical
| comparison of C, C++, Java, Perl, Python, Rexx, and Tcl" by
| Lutz Prechelt.
|
| Recent-ish commentary (July 2021) about it at
| https://renato.athaydes.com/posts/revisiting-prechelt-paper-...
| , HN commentary at
| https://news.ycombinator.com/item?id=28108806 .
|
| Google Scholar gives about 476 paper which cite Prechelt's
| work. I have not followed other work in that field.
| npsimons wrote:
| Norvig had a response to Prechelt as well:
| http://www.norvig.com/java-lisp.html
| eesmith wrote:
| Mentioned and used as the Lisp baseline in the link I gave.
| ;)
| npsimons wrote:
| > Mentioned and used as the Lisp baseline in the link I
| gave. ;)
|
| Just saw that, nice. I like Norvig because he really cuts
| to the chase.
| tinco wrote:
| There's a great deal of evidence that supports that, it's just
| not scientifically organized. If you just look at the amount of
| high quality high polished projects out there for the dynamic
| programming languages it absolutely dwarfs those of static
| languages. And then add to that the fact that most of those are
| by solo devs in their free time.
|
| It's not just that they are more popular, Java reigned supreme
| for years when Ruby and PHP and later Node.js outpaced it's
| communities by miles. And you can't say Java developers are not
| open source oriented, because they 100% are.
|
| And the argument that programmers are simply most productive in
| the language they know best is trivially untrue. When I first
| learned of Ruby, I could dream in C#, I was absolutely fluent,
| knew the standard library by heart. I switched to Ruby and only
| looked back whenever I needed tight performance. And this is
| not just an anecdote, the whole Ruby on Rails movement was
| basically Java developers fleeing to greener pastures.
|
| I don't know a single highly experienced multi-lingual
| developer who does not reach for a dynamic language when they
| need to deliver something quick and easy.
|
| The only exception I know off is using Go for small networked
| services, which is quite comfortable and intuitive for a static
| language. Also outside that niche Go quickly loses its
| productivity edge.
| smt88 wrote:
| > _I don 't know a single highly experienced multi-lingual
| developer who does not reach for a dynamic language when they
| need to deliver something quick and easy._
|
| If you're picking from a popular language, the language
| itself rarely matters for this at all. Every popular language
| does roughly the same things. Writing speed (length of
| keywords, for example) is a non-issue. The ecosystem is by
| far the most important thing.
|
| It doesn't matter how efficient I am in Go if I'm missing a
| crucial library that I'll now have to write myself.
|
| JavaScript is a classic example where, even if you are pretty
| fast at writing your code, you will be dramatically slowed
| down by: 1) needing to add a new package every 5 min to do
| something trivial, 2) looking through five half-dead
| libraries to find one that seems maintained and usable, and
| 3) finding out that some of the libraries you chose are
| buggy.
|
| So for a quick prototype, I'd weight a language's qualities
| as follows:
|
| - ecosystem: 90%
|
| - static analysis/tooling: 5%
|
| - stdlib: 3%
|
| - syntax: 2%
|
| And for a long-term, complex project, I'd weight them closer
| to:
|
| - ecosystem: 50%
|
| - static analysis/tooling: 46%
|
| - stdlib: 3%
|
| - syntax: 1%
| pyrale wrote:
| > I don't know a single highly experienced multi-lingual
| developer who does not reach for a dynamic language when they
| need to deliver something quick and easy.
|
| Maybe this shines light on your social circles more than
| actual language impact on proficiency?
|
| The "our language is so much more productive" myth exists in
| many languages, including the one I currently use. But when
| you dig deeper, there are almost always other social or
| technical factors that explain the productivity gap. That
| kind of gap exists even between people using the same tool.
|
| The reality of it is that assessing what makes a developer
| productive is incredibly hard, and people doing so to claim
| their language of choice is better rely on anecdata and
| couldn't explain what "methodology" means to save their life.
| brabel wrote:
| > I don't know a single highly experienced multi-lingual
| developer who does not reach for a dynamic language when they
| need to deliver something quick and easy.
|
| This argument is not addressing the claim for general purpose
| software, only for "quick and easy" software.
|
| To people in this thread: please stop to think before
| responding to the "wrong point". I think we can all agree
| that dynamic, scripting languages are more adequate for
| "quick and easy" software, but this is not what the point of
| the discussion is. The point of the discussion is, or should
| be in my opinion, whether it's true that for general purpose
| software, written by teams, that is not trivial to write by
| oneself in 5 minutes, that scripting languages are more
| productive than statically typed, compiled languages.
| tinco wrote:
| Where do you draw the line? Is GitHub quick and easy
| software? Is Shopify? Were LinkedIn, Twitter? Are things
| built on Tensorflow?
|
| I think you misunderstand the scripting language revolution
| of the mid 2000's. It's not that we suddenly realized
| scripting languages were the best for quick and easy
| projects. We realized scripting languages were suitable for
| a whole lot more than some data processing. We could much
| more effectively build huge scalable software platforms.
|
| It got so bad that the sentiment flipped, and people like
| Joel Spolsky had to go out of their way writing blog posts
| that you could in fact build successful modern web
| platforms in C#. And then he went building the world's most
| popular project management tool in Node.js anyway.
| oopsyDoodl wrote:
| Python was Googles language of choice for a while, so people
| jumped at it to work at Google.
|
| Popularity and expansive use may have nothing to do with
| quality, and a lot to do with financial influence on peoples
| agency.
|
| Business wants people templating out directories of
| performant code, not generating syntactic art for the ages.
| dimitrios1 wrote:
| > I don't know a single highly experienced multi-lingual
| developer who does not reach for a dynamic language when they
| need to deliver something quick and easy.
|
| Hi, highly experienced, multi-lingual developer here (Python,
| Ruby, PHP, JavaScript, Java, Go, C#, F#, OCaml,
| Elixir/Erlang, ReasonML, SQL, Smalltalk, Objective-C, Swift,
| leaving off quite a few prior Web 2.0 ones for brevity) nice
| to meet you. As you can see, I've done just about all of it:
| every paradigm, every syntax. It all honestly blurs together
| after a certain point, and it just becomes easier and easier
| to pick up a new language the more you learn.
|
| Anyways, these days, I never reach for a dynamic language
| when I need to deliver something quick and easy. The
| difference is just too marginal. Not worth the downsides (and
| the downsides are immense!).
|
| Because, as my experience has taught me, invariably one of
| those quick and easy ones will turn into something that
| becomes business critical, lives on for years after you are
| gone, and will be much more difficult for other, typically
| more junior, developers to update or enhance your code.
| Static typing is not only marginally less productive these
| days with all the great tools and IDE's out there (not to
| mention Go which has one of the least obtrusive static type
| checkers I've seen, or, even better if the org allows you, a
| language with a HM type system), but for a marginal
| improvement in productivity, you pay a heavy long term cost.
| Not worth it.
|
| Dynamic languages are great for rapid prototypes. After that,
| convert it to a static language. Your junior devs that join
| after you, who aren't familiar with the entire ecosystem your
| work has will thank you.
| saila wrote:
| > After that, convert it to a static language
|
| I don't think you can have a hard and fast rule like this.
| It will depend on the situation. In many instances, a
| "prototype" built in Python or similar is perfectly fine.
|
| In addition, one can make a horrendous mess in statically
| typed languages as well. One of the absolute worst projects
| I ever worked on was written in a popular statically typed
| language. This is of course an anecdote and not to say
| statically typed languages are _worse_ , but they're not
| automatically _better_ either.
|
| Just a note too that I'm a big fan of, e.g., Rust &
| TypeScript, and I often use type hints in Python, so I'm
| not anti-static typing.
| dimitrios1 wrote:
| Way easier to refactor a that statically typed mess.
|
| There is a floor for how bad you can write static typed
| code.
|
| There is no floor to how bad of JavaScript or Python or
| Ruby you can write. The madness can descend to the inner
| most circles of coding hell. Only Perl exceeds it.
| Jcampuzano2 wrote:
| Well which one do you tend to reach for then? Purely out of
| curiosity.
| dimitrios1 wrote:
| Go in the general case, Java if complex domain modeling
| is required (or requires "enterprise grade" B.S - SOAP,
| WSDLs, or other crazy XML specified madness like Adobe
| extension stuff, etc), and after that: anything
| functional if I am allowed.
|
| Explained:
|
| Go's type system is generally less rigid. It strikes a
| good balance of strict enough. A lot of Go's converts
| aren't from "systems" languages like it targeted
| originally, but rather former Python/Ruby/PHP/JavaScript
| backend devs. I love the performance and low level levers
| I can pull with Go (although to be fair Java is quite
| fast enough). But finally, Go is easy to learn (26
| language keywords?), the standard library is mostly
| great, and the worst developers I've seen still write
| mostly maintainable code that builds fast, which is what
| I optimize the most for these days.
|
| Java, for all its warts and legacy cruft, these days you
| can write fairly good java, utilizing modern libraries. I
| love most things from Codahale, who in turn I think
| pushed orgs like Spring to write better libraries, so now
| everything's pretty good. Plus all the legacy stuff comes
| in handy when you have to deal with arcane government or
| financial systems, something I have to interface with
| frequently.
|
| But if I had my choice, I'd use something where you can
| express functional programming concepts intuitively,
| without fighting the language or having to do it at a
| heavy performance cost, like Rust or better yet, just a
| full fledged FP language like OCaml (whom I understand
| heavily inspired Rust)
| nodejs_rulez_1 wrote:
| Python, like all dynamic languages, works best to bash out a
| new project into production ASAP, get a bonus and move on,
| while poor folks maintaining after you are trying to untangle
| it.
| geofft wrote:
| You're saying you have _less_ problems untangling C
| /C++/Fortran/etc. code left from previous folks who wanted to
| ship something and move on?
| david38 wrote:
| Python's "one right way to do it" is specifically designed to
| avoid this. This isn't Perl.
| woah wrote:
| Python has multiple different ways to do things even in the
| standard library lol
| Zababa wrote:
| Current Python doesn't reflect this philosophy though. You
| have lots of options for strings and string formatting.
| There is pattern matching and if statements. The standard
| library often isn't the best option for stuff (like HTTP
| requests) so you use another library. Package management
| and deployment is far from solved, with lots of different
| tools.
| dragonwriter wrote:
| > You have lots of options for strings and string
| formatting.
|
| "Strings and string formatting" is a very broad domain.
|
| For most specific tasks, there is one low-impedance
| approach and it takes very little (but not zero)
| reflection and/or experience to find it.
|
| Most new Python features directly address specific tasks
| for which there are currently multiple relatively high-
| impedance approaches taken because there is not one
| obviously correct way.
|
| Python's "one obviously correct way" is not "one possible
| way" (the latter approach is closer to Go.)
| klyrs wrote:
| Pet peeve: there is no right way to do type annotations.
| Two major projects aren't even compatible:
| https://www.cs.rpi.edu/~milanova/docs/dls2020.pdf
| zcw100 wrote:
| I think there's a lot of "Python's the greatest thing in the
| world" coming from ML/AI and it always reminds me of this. I
| wish I knew who said it.
|
| "People who know one language think it's the greatest in the
| world. People who know more than one think they all suck."
| npsimons wrote:
| > "People who know one language think it's the greatest in
| the world. People who know more than one think they all
| suck."
|
| The Blub paradox: http://www.paulgraham.com/avg.html
|
| (BTW, the TL;DR is people that know more than one language
| admit they all fall short one way or another, but there
| _are_ some languages that are more productive than others).
| streamofdigits wrote:
| Over time I've come to conclude we are not optimizing for
| language features, we are optimizing for the community around a
| programming language or stack.
|
| Sure, features are important and if critical ones are missing it
| might be a show stopper. So there is an initial thresshold that
| all candidates must pass.
|
| But problem solving is not a one-off exercise. It tends to be
| both dynamic (=facing unpredictable challenges) and recurring
| over long time horizons. Which means having a healthy, engaged,
| resourced community that will invest in adapting / solving
| _future requirements_ is essential.
|
| So the "optimization" problem includes quite a bit more than the
| presently known developer team, its software stack its hardware
| and current problem definition / user requirement.
|
| I think you see this dynamic in several cases (including python)
| where you might not think that it makes rational sense.
| throwaway_2047 wrote:
| Learning coding can be a form of premature optimization.
| Exercising can be a form of premature optimization. Your body
| ain't gonna need it. jk
| mobilemidget wrote:
| I'm a big fan of the "Make It Work Make It Right Make It Fast"
|
| If one of these steps requires a change of programming language,
| I'm never too lazy to recode :)
| Jensson wrote:
| Problem is that if you follow that strategy then it will likely
| never be as performant as if you did "Make It Fast, Make it
| work". So if you know that performance is a key factor to
| success then you should follow this path instead.
|
| And by "make it fast, make it work", I mean benchmark first.
| Kind of like test first, you write the benchmark before the
| implementation and constantly look at how fast things runs, and
| ensues every single bit you add is fast.
| DemocracyFTW wrote:
| I don't think the "Make it Right" part should or can
| reasonably be left out of the equation, otherwise I could
| just compute _something_ if all you care about is speed. That
| said, I have used a performance-first approach several times
| using own old and new solutions and libraries and have found
| early benchmarks to be a great tool to weed out untenable
| (i.e. order-of-magnitude slower) stuff. This then can mean I
| have to write fewer tests to make plausible my own or a 3rd
| party solution does in fact what I expect it to do. At a
| certain point performance becomes correctness.
| pphysch wrote:
| It's the usual trade-off of individual programmer
| freedom/expressiveness vs. rigid & predictable standards that
| enable productivity at scale.
|
| Just because you can _program_ a new feature quickly doesn 't
| mean it is _engineered_ well for the long-term & larger scales.
| firasd wrote:
| Just been thinking through this... I'm working on a personal
| project (podcast indexing/search) that involves parsing a lot of
| RSS feeds. Some years ago I had built the feed checker in Elixir
| but now when I tried to get it going again I was having too much
| trouble with version and compatibility changes in Phoenix. I
| eventually just did that part in PHP cause that's my day-to-day
| language. Wouldn't have made sense to block all development until
| I re-learned how to query a database in Phoenix. Plus later I can
| re-implement that feed checking part as as service in Elixir or
| another language. First make it work..
| StefanKiss wrote:
| well if you combine the title with the contents you reach the
| conclusion that python is not a programing language. fair enough.
| mumblemumble wrote:
| A case study from my own personal experience:
|
| We generally do everything in Java. Python is avoided because of
| all the usual complaints - dynamically typed, not fast enough,
| GIL, etc.
|
| So, as a PoC, I decided to try rewriting one of our services in
| Python. And, compared to the Java one, it is:
|
| Cheaper to write and maintain. About 1/10 as many SLOC. About
| 1/20 as many person-hours.
|
| Has better static type checks. For example, Java's type checker
| cannot statically verify for null safety. The Python one I chose
| does do that. Note that, since I did choose to put type hints on
| everything, the productivity boost in question _cannot_ be
| attributed to dynamic typing.
|
| (That said, not all 3rd-party libraries have type hints, so, if
| you want to type check everything, you may have some yak shaving
| to do.)
|
| Uses less RAM. About 1/2 as much.
|
| Is faster. Admittedly I'm leaning on numpy, Cython, and friends
| for this. It may well be much slower for projects where that is
| not possible. But still, I think that the point about premature
| operation stands in this case.
|
| (Disclaimer: I also have a colleague at $FAMOUS_COMPANY who tells
| me they are moving off of Python because they found the opposite
| of the above in many cases. Though I personally suspect that a
| mitigating factor is that their profit margins and scale are
| large enough that all the coefficients in their cost/benefit
| formula are wildly different from the norm.)
| overtomanu wrote:
| you have to also consider backward compatibility. If at all
| there is another version change in python like the one that
| happened from 2.0 to 3.0, maintenance effort would be more.
| BiteCode_dev wrote:
| Java, Perl and Ruby all had the same.
|
| In fact, in 2018, Java 8 was still massively used:
|
| https://jaxenter.com/java-8-still-strong-java-10-142642.html
|
| Then you have libs and frameworks, and they break compat too.
| The JS ecosystem basically broke everyone code for 10 years
| every 6 month.
|
| It's not a Python thing.
| mumblemumble wrote:
| According to the latest jrebel report, about 70% of Java
| shops are still on Java 8 as of earlier this year. The
| changes with Java 9 certainly created plenty of work for
| me. Java's famous claim to backward compatibility mostly
| concerns the JVM itself. They have done a killer job there.
| But the virtual machine is not the same thing as the
| platform that rests atop it.
|
| This is not by way of complaining about Java 9. The changes
| are good and needed to happen. And the reasons why most
| people are still on Java 8 are also legitimate. Everyone's
| handling the situation in a fairly mature way, from what
| I've seen. The level of histrionics about Python 3,
| though... I've got a lot less sympathy for that.
| ar_lan wrote:
| > So, as a PoC, I decided to try rewriting one of our services
| in Python.
|
| Not to your main point - but this line threw me off. I really
| thought you were discussing this as a "person of color", and I
| was so confused as the relationship of persons of color and
| Python. ^_^
| xrobledo84 wrote:
| So what does PoC mean here then?
| twobitshifter wrote:
| Proof of concept
| [deleted]
| killingtime74 wrote:
| Kotlin Scala does all these and interface with Java to boot
| TheDudeMan wrote:
| > Java's type checker cannot statically verify for null safety.
|
| Check out NullAway.
|
| > Uses less RAM. About 1/2 as much.
|
| Were you on a modern JVM?
| mumblemumble wrote:
| It's not just null. For example, Java much more frequently
| leaves you in a situation where you need to cast to/from
| `Object`. In Python, `Any` type hints can usually be avoided,
| because of union types.
| taeric wrote:
| First, I want to make it clear that if you have a
| programming style that works for you, please don't take
| this as an attack against it. Keep using what works.
|
| If you find yourself commonly having to cast to/from Object
| in Java, that is almost certainly something of your own
| doing.
| ubercore wrote:
| What's the "Java way" of accomplishing something like
| you'd represent with Union or Intersection types?
| avita1 wrote:
| Other comments have linked newer language features that
| make it easy. But for years, the Java Way of handling
| discriminated unions was to use the visitor pattern [1].
| It's very verbose, and is an insane amount of typing
| unless your IDE is doing the typing for you, but it has
| the compile time guarantees that forces each caller to
| handle every type without instanceof/Object.
|
| [1] https://dzone.com/articles/design-patterns-visitor
| dljsjr wrote:
| A common interface with all of the required functionality
| exposed as interface methods. A Java person would argue
| that if you have a function that takes an object that can
| be two totally disparate things with no shared
| functionality then that's a code smell.
| programmer_dude wrote:
| There is none, without up or down casting. This is just
| some "guideline" that gets paraded from time to time (in
| OOP circles). On second thought, I am sure there's a
| "design pattern" for sum types out there. Why take the
| easy way out, am I right?
| [deleted]
| tomtheelder wrote:
| Probably with a composite class I guess. So a union type
| of types A and B has members of types A and B, and a non-
| nullable flag to indicate which one this instance is.
| Intersection type is just the same thing without the
| flag.
|
| I don't write Java, though, just guessing.
| Someone wrote:
| You can also implement an union type as a sealed
| interface/abstract class with two classes that
| implement/extend it (https://openjdk.java.net/jeps/409,
| https://www.baeldung.com/java-sealed-classes-interfaces)
| mumblemumble wrote:
| It's one option. It has the advantage of structurally
| limiting the types that can be used in that spot. It has
| the disadvantage (compared to true union types) of
| offering no static help beyond that.
|
| If you add a case, for example, you're 100% on your own
| to make sure that all code that interacts with the type
| is updated to handle the new case. Slip-ups will produce
| run-time errors rather than static ones.
|
| It's also rather tricky (and awkward) to set things up
| such that consumers are forced to check the case value
| before attempting to coerce it.
|
| All of this can be worked around with custom linter
| rules, but then that becomes its own maintenance burden.
|
| My preference is to try and invert things such that you
| can rely on dynamic dispatch and "tell, don't ask." That
| eliminates the need to coerce things at run-time in the
| first place.
| [deleted]
| mixedCase wrote:
| Not that I'm a Java proponent, but have you tried doing the
| same rewrite effort on modern Java and with modern tools?
|
| With newer tools and most important of all: your rewrite being
| a second iteration with the benefit of hindsight, you may
| notice that the language choice makes less of an impact than
| expected. IME language differences tend to shine the most when
| writing something the first time around and its expression
| capabilities shape the design.
| mumblemumble wrote:
| I also rewrote a Python service in Java in an attempt to
| control for that. The results came out similarly in terms of
| development & maintenance effort. Unfortunately it wasn't a
| thing where a performance comparison made sense, though, so I
| didn't really do one.
|
| My sense was that the big benefit was actually down to the
| libraries and syntactic sugar. Dataclasses, comprehensions,
| and generator functions all have a big impact, but so do
| things like Python just generally having more ergonomic
| libraries for talking to databases, producting/consuming REST
| APIs, and reading/writing data files.
| xapata wrote:
| I suspect there's some algorithmic choice hidden in a
| library in this comparison, where the Python library has
| the right choice for the job and the Java library has the
| wrong one.
| twic wrote:
| You mentioned NumPy. NumPy is great, and there is basically
| no Java equivalent. So if you are doing the kind of work
| which NumPy can help with, it's going to be very hard work
| to beat it with Java, either in development speed or
| execution speed.
|
| To put it another way, perhaps what you tested here was
| having a great array library vs not having a great array
| library! It would be interesting to test Python vs Java on
| a more generic task, where there isn't a huge advantage
| from a particular library or set of libraries.
|
| It would also be great if someone would write a NumPy-style
| library for Java. My team would use that a lot!
| BiteCode_dev wrote:
| There is also no Django for Java, no Fast API for Java,
| no Typer for Java, etc.
|
| Why ? Because it's easier to write great frameworks in
| Python.
|
| It compounds.
| twic wrote:
| Equally, there's no Hibernate for Python, no Spring for
| Python. And FastAPI looks basically like JAX-RS to me.
|
| I don't think the lack of any of these things is because
| they're impossible, or even particularly hard. It's
| because the community did or did not have the need and
| energy to build them.
|
| Remember that Java had Java EE forced on it quite early.
| That defined web development for a long time. Eventually
| Spring overthrew it, and became the new tyrant. Neither
| of those are quite like Django, or Rails, but they
| occupied the space that would have had to have been empty
| for a Django or Rails to emerge. I think this is rather
| unfortunate; it would be really useful to have a vibrant
| Django- or Rails-like framework in Java.
|
| There's no Typer because there isn't much of a culture of
| building command-line tools in Java. There are some good
| command-line parsing libraries, but nothing with all the
| bells and whistles and extensions that Typer has, as far
| as i know.
| BiteCode_dev wrote:
| > Equally, there's no Hibernate for Python, no Spring for
| Python. And FastAPI looks basically like JAX-RS to me.
|
| Of course there is. In fact, SQLA is easier to use than
| Hibernate, and the ecosystem around is as rich. As for
| Spring, crossbar fits the bill.
|
| > I don't think the lack of any of these things is
| because they're impossible, or even particularly hard.
| It's because the community did or did not have the need
| and energy to build them.
|
| I've never say they are impossible. I've said they
| require less energy to build them in python :)
| [deleted]
| slotrans wrote:
| No one who writes Python has ever said to themselves, "I
| wish I had Spring." It's not necessary, and it wouldn't
| even be beneficial. The kind of dynamism that Spring
| brings to Java, Python already has.
|
| Also Hibernate is terrible and should be erased from the
| Earth.
| kwhitefoot wrote:
| > You mentioned NumPy. NumPy is great, and there is
| basically no Java equivalent.
|
| Things like this are the greatest disappointment to me
| after fifty years of writing software. It should not be
| necessary that _Java equivalent_ exists. It really ought
| to be possible to connect to libraries from any language
| by now.
| twic wrote:
| After fifty years of writing software, i would have
| thought you would understand why that wasn't possible.
| Mikeb85 wrote:
| What do you mean? JBlas exists. Colt exists. There's
| probably others but I'm not super up to date on Java
| things.
|
| Numpy is just wrappers for BLAS and LAPACK.
| plafl wrote:
| > Numpy is just wrappers for BLAS and LAPACK.
|
| Certainly not. I encourage you to have a look at the API
| of BLAS/LAPACK to perform basic operationa. And that just
| scratches the surface.
| Mikeb85 wrote:
| https://numpy.org/doc/stable/user/building.html
|
| It is. Maybe it's a very fancy wrapper.
| mumblemumble wrote:
| I would argue that numpy is a fancy wrapper for BLAS in
| the same way that a Spitfire is a fancy wrapper for a
| Rolls-Royce Merlin 61.
| mumblemumble wrote:
| Numpy was not a factor in the Python->Java PoC; it was a
| service that did no number crunching whatsoever. There,
| the specific library advantages were the psycopg, flask,
| requests, json, and csv packages. All of the (mainstream)
| Java equivalents are just kind of verbose and fiddly to
| work with compared to those.
|
| I don't think this is entirely down to the language
| itself. A lot of it is cultural. The Java community tends
| to favor a more "enterprisey" style of programming and
| API design that, at least to my tastes, tends to be
| rather overwrought. That _could_ change, in principle. I
| 'm just not optimistic about it happening any time soon.
| twic wrote:
| There is a lot of enterprisey crap, but there are also
| many decent libraries. In particular, i am pretty happy
| with the Java options for the things you mention - the
| PostgreSQL JDBC driver, JDBC itself, and HikariCP for
| pooling, the JDK HttpServer (no templating though, don't
| know if you need that here), the JDK HttpClient, Jakarta
| JSON with Joy as an implementation, and SimpleFlatMapper
| for CSV. None of them are perfect, but i am pretty happy
| churning out generic business service type things with
| them.
|
| I wish there was some way i could take a sabbatical from
| my job and come and try to rewrite your service in what i
| consider effective modern Java. I am pretty optimistic it
| would be competitive with Python in effort and
| maintainability, but i have no way to prove it!
| tmp_anon_22 wrote:
| Are you as familiar with Java as you are Python?
| mumblemumble wrote:
| I'm much more familiar with Java. It's mostly what I get
| paid to write. Python's relegated to times when I can get
| away with it, which are relatively uncommon.
| srvmshr wrote:
| I quite believe that every programming language has its strengths
| & weaknesses, and a few languages is a must in every developer's
| accoutrements.
|
| Want to build an I/O utility writing to a DB? Sure C can do it,
| but Python is better suited. Want to write a toy compiler? You
| don't want to waste your time trying to wrangle on CPython
| extensions. C works out of the box.
|
| > _'if you select a programming language based on your
| preconceived notions of how a language performs, you will never
| know if the language that might be a better, more productive
| fit'_
|
| Part of the CS education is not about recognizing homeruns but
| understanding trade-offs. The experience gain is about learning
| how tools work & which tools to choose to work in tandem. Modern
| systems use a variety of languages - JS in the webpage, SQL DBMS
| for queries, C++ to run the performance bits, Python for ML,
| introperations - maybe even Rust in the security bits of late. In
| that sense, the title was unfortunately misleading to me, since
| author tried to demonstrate a lot of usecases with Python.
|
| Python is great - but there has to be a reason why other
| languages co-exist. Not just for bankers, military or some
| enthusiastic hobbyist.
| scottcodie wrote:
| People are too shy on hardware costs. Many of my professional
| colleagues develop and optimize software full time that runs on a
| server that is as fast as the phone in their pocket.
| exyi wrote:
| I don't like how he makes it a choice between Python and
| C++/Rust. There is very many languages that are more similar to
| Python in convenience and yet run reasonably fast (and you can
| actually optimize some procedures when you need to, because there
| is a compiler). Go, Julia, C#, F#, Scala, Kotlin, even recent
| Java... all much faster than Python and much less pain to work
| with than C++.
|
| And the interoperability is not as awesome as it's painted in the
| article, it's always more pain to have more languages in a
| project that need to talk together
| raxxorrax wrote:
| You have to look at resources too. You probably find 10 Java
| developers before you find 1 Scala developer, even if they are
| related technically. Especially on long term projects you have
| at least some turnover of people.
| qaq wrote:
| I am less productive in Python vs say Go. So by default use
| Python would be a bad heuristic for people like me. Has nothing
| to do with runtime performance for the most part.
| pella wrote:
| > Prototype in Python
|
| Julia one of the target is solving the "two-language problem"
|
| _" Julia seemed to have solved the "two-language problem"--a
| conundrum often facing Python programmers, as well as users of
| other expressive, interpreted languages. You write a program to
| solve a problem in Python, enjoying its pleasant syntax and
| interactivity. The program works on a test version of your
| problem, but when you try to scale it up to something more
| realistic, it's too slow. This is not your fault. Python is
| inherently slow--something that doesn't matter for some types of
| applications but does matter for your big simulation. After
| applying various techniques to speed it up but only realizing
| modest gains, you finally resort to rewriting the most time-
| consuming parts of the calculation in C (most commonly). Now it's
| fast enough, but now you also need to maintain code in both
| languages, hence the two-language problem."_
|
| https://arstechnica.com/science/2020/10/the-unreasonable-eff...
| deckard1 wrote:
| What's the tl;dr of how Julia is solving this? Looking around
| it seems the answer is "multiple dispatch". Which seems suspect
| considering many languages have already tried this (Common
| Lisp, for example).
|
| > Clearly, multiple dispatch, or some other way around the
| expression problem, is necessary for the kind of fluent
| composability that I've described above-- _but it is not
| sufficient_. Julia has enjoyed an explosive degree of uptake in
| the scientific community because it combines this feature with
| several others that make it very attractive to numericists.
|
| That's incredibly handwavy. So what's the special sauce?
|
| There is no such thing as a free lunch when it comes to dynamic
| vs. static. It also seems like Julia _is_ trading off
| expressiveness and easy of use in favor of efficiency, based on
| comments from people that have used Julia. It 's one thing to
| be faster than any inherently slow language (Ruby, Python,
| Smalltalk, etc.), but keeping that flexibility and being as
| fast as C/C++ is a rather bold claim. Most languages hit some
| middle ground between the two, such as Java. But no one is
| under the delusion that trade-offs _weren 't_ made to get
| there.
| adgjlsfhk1 wrote:
| Julia makes a number of (in my opinion) really good tradeoffs
| here.
|
| 1. You can't add fields to a type (struct) after definition.
| This means that Julia's structs have no overhead and are
| essentially equivalent to structs in C (although they are
| parametric)
|
| 2. No local eval. Eval in Julia only happens in the global
| scope and results of eval are only visible the next time you
| visit the global scope. This may sound kind of unintuitive,
| but in practice people don't generally use this for good
| reasons. This allows Julia to never need to de-optimize code.
| Once a method is compiled that code remains valid.
|
| 3. Macros. Julia has really good macros and other code
| manipulation (since it is basically a Lisp). This makes it
| possible to generate very complicated but fast code that you
| would never write yourself. The tradeoff here is that it
| makes the language more complex, but that's a pretty good
| tradeoff. (especially compared to the C/Fortran land of using
| a preprocessor that works on text).
|
| 4. Just-In-Time (just ahead of time). Julia at it's core runs
| as if it were highly templated C++ code. If everything got
| compiled ahead of time, Julia would be generating terabytes
| of compiled code and never finish compiling. Instead, Julia
| makes the tradeoff of only compiling for the argument types
| that are actually used in the program, which means that it
| only compiles a reasonable amount of code. The tradeoff here
| is that compiling small binaries with Julia is very difficult
| (not possible to do automatically yet).
|
| The TLDR is that most expressive languages started by giving
| away as much expressiveness as possible, and then looked at
| how they could be sped up. Julia started by being a modern
| fast language and looked to see how much expressiveness could
| be added without slowing the language down.
| fault1 wrote:
| Julia combines the following:
|
| - multiple dispatch
|
| - parametricity
|
| - lightweight subtyping
|
| - staged programming
|
| In interesting ways.
|
| felleisen's class talks (partially) about it here:
| https://felleisen.org/matthias/4400-s20/lecture15.html
|
| I guess the best way to put it is that Julia encourages a
| style where 90%+ of code can go through paths that are
| static.
|
| Personally, I think Julia starts off as easy as python, but
| to get C++ or Fortran speed, you can't just code naively.
| Things go into a steep learning curve at that point, but
| perhaps there isn't yet as much know how about how to code
| "professional Julia" yet. There needs to be a book like
| Fluent Python or Effective C++ for Julia, or perhaps a
| condensed version of the Julia manual (see the 1 page zig
| manual for inspiration).
|
| The other problem I have with Julia right now is lack of
| static type checkers. "modern python" (e.g, python in
| production in the last 5 years) tends to leverage the large
| ecosystem of things that hook into mypy (I'm taking about
| tools like pydantic) to reduce the inherent brittleness of
| the language. Ruby, php, and every other dynamic language has
| also seen that trend.
|
| Right now, I've barely seen that with Julia, and it needs
| this badly for higher uptake in industry. It's why for
| example, perhaps you see a lot of Julia packages written for
| people's phds right now.
| eigenspace wrote:
| Your instinct that Julia is making tradeoffs is indeed
| correct, however I don't think it actually limits the
| expressiveness of the language. I happen to think that Julia
| is a _more_ expressive language than Python. However, it does
| require learning new patterns and paradigms and someone who
| tries to write Python code in Julia is probably bound to
| eventually get frustrated.
|
| A huge part of the design considerations for Julia
| essentially boiled down to "what sorts of dynamism and
| language semantics can we disallow while keeping the the good
| parts of dynamism"
|
| The two biggest things that had to go in order to make Julia
| fast was
|
| 1) the ability to change the memory layout of a struct in a
| running session
|
| 2) the ability to eval in the local scope (our eval always
| occurs in the global scope)
|
| These two things are huge performance problems. We might
| oneday solve 1) with Revise.jl (though it'll mean recompiling
| all your code if you do change the layout) but 2) is
| basically just a very bad idea and likely to never happen.
| Instead of a locally scoped eval, we have macros, multiple
| dispatch, parametric types, and generated functions. These
| give an incredibly powerful suite of metaprogramming tools
| that are beyond anything available in Python.
| Jtsummers wrote:
| Julia does a few things differently then Common Lisp, though
| they both offer multiple dispatch.
|
| One of the key things in CL is that it has its metaobject
| protocol which forces a lot of decisions on what gets
| executed to runtime. There are ways to speed it up, but if
| you have something like: (defgeneric foo (x))
| (defmethod foo ((x number)) (print 'number)) (defmethod
| foo ((x integer)) (print 'integer)) (defmethod foo
| :before ((x number)) (print 'also-a-number)) (foo 3)
| (foo 3.0)
|
| Then CL won't call _foo_ specialized on _number_ when given
| an integer, but _will_ call _foo :before_ specialized on
| _number_. It determines this at runtime by searching for all
| applicable methods based on the type (at least as a first
| pass, you can cache this to speed it up but then you also
| have to have cache invalidation if a definition is changed).
|
| Julia doesn't have that aspect of CL's MOP. So this helps to
| simplify the search for applicable methods and dispatch. Even
| if it did all its dispatch at runtime, it would still be
| simpler. The other thing Julia does is aggressive JIT
| compilation. So if you wrote something like (with the Julia
| equivalent of foo from above): function
| bar(x,y) foo(x) foo(y) end
|
| And, only considering floats and integers, later called it
| with each pair of float and integer then Julia would compile
| specialized versions for those 4 combinations. Now when you
| call _bar_ it still has to properly dispatch it, but once
| inside _bar_ the search for the correct _foo_ can be bypassed
| because the types will be known. CL, again thanks to the MOP,
| doesn 't make that as easy to achieve.
| GavinMcG wrote:
| Nim seems like another candidate.
| GuB-42 wrote:
| Python is not just slow, it also tends to break down for large
| projects, like all dynamic languages. It is not an absolute, you
| can do it, but the larger the code base, the more you need strong
| guarantees over flexibility, and the less relevant languages like
| Python tend to become.
|
| Prototyping in Python is a viable strategy. Personally I tend to
| use Perl for that, because I am comfortable with it and I find it
| better adapted to quick prototyping than Python. I then rewrite
| in another language, usually C++, with more care about data
| structures and optimization. The author strategy to do everything
| in Python than optimize the slow parts can make sense, though I
| tend to prefer the opposite: white your framework/engine in a
| static language (like C++) and embed an interpreter (Python, LUA,
| etc...), as it is common in game dev.
|
| But all that are language decisions! Whatever you do, there are
| going to be consequences. Choosing Python with C/C++/Rust
| optimizations is not bad, but it is a rather strong opinion, not
| the obvious default choice the author makes it.
| medo-bear wrote:
| > like all dynamic languages
|
| I think you would be surprised by the amount of type-checked
| bugs that can be eliminated in dynamic languages with standard
| software engineering practices. have a look at how common lisp,
| for example, handles type annotations and type warnings:
| https://lispcookbook.github.io/cl-cookbook/type.html
|
| instead of blaming dynamic languages, i think project managers
| need to take on better practices
|
| > but the larger the code base, the more you need strong
| guarantees over flexibility
|
| i think i disagree with this. the larger the code base the more
| flexible the program needs to be towards its input, otherwise
| your program is too sensitive and it will crash. the output
| however i think should be strict
| npsimons wrote:
| > I think you would be surprised by the amount of type-
| checked bugs that can be eliminated in dynamic languages with
| standard software engineering practices. have a look at how
| common lisp
|
| To second this, I _just_ read someone 's anecdote about how
| unit tests/TDD is eliminating the need for type checking:
| https://www.artima.com/weblogs/viewpost.jsp?thread=4639
|
| > i think i disagree with this. the larger the code base the
| more flexible the program needs to be towards its input
|
| And again, I'll second this with another anecdotal posting,
| but with the emphasis that _input_ is not just run-time, but
| also specifications for programs: http://ivy.io/common-
| lisp/2015/03/03/guerilla-lisp-opus.html
| exyi wrote:
| Not just dynamic types, but all the breaking changes they make
| in minor versions. Since the program is "compiled" on user's
| machine, not yours, you don't even control which version is it
| going to use. I mean, migrating from Scala 2.12 to 2.13 to is
| bit annoying, but I can do it whenever I have time. Python 3.9
| to 3.10 - ASAP, it's a bug suddenly when they decide to
| release.
| hsn915 wrote:
| > it also tends to break down for large projects
|
| You also end up spending a _lot_ of time and resources in the
| later stages of the project trying to work around the problems:
| hire more people to work on the code base, spend a lot of time
| investigating optimization strategies, hire a devops team to
| turn the program into a distributed system with maybe hundreds
| of instances running on CloudProvider (TM)
| api wrote:
| > turn the program into a distributed system with maybe
| hundreds of instances running on CloudProvider (TM)
|
| ... and give tens to hundreds of thousands a month to cloud
| providers when you could spend 1/100th if it were written in
| Go.
| hsn915 wrote:
| You could even save the costs by buying your own server. I
| doubt that you can't buy a very decent server hardware for
| no more than $10000. Some places pay more than that _per
| month_ to Amazon.
| api wrote:
| But that's not cloud, because cloud!
| deltaonefour wrote:
| Modern python (used in production) should be used in
| conjunction with static type checking apps. Your argument is no
| longer relevant for modern python. def div(x:
| Int, y: Int) -> Optional[Int]: return x // y if y !=
| 0 else None
|
| The above should be what production level python should look
| like nowadays.
|
| https://www.infoworld.com/article/3575079/4-python-type-chec...
|
| That is not to say the type checking situation for python is
| perfect, far from it, but the situation is good enough that
| arguments against python that utilize the dynamic nature of the
| language are mostly no longer relevant.
| globular-toast wrote:
| Division by zero should not return None. Is there an aversion
| to exceptions in "modern" Python?
| deltaonefour wrote:
| This is definitively wrong. Not to comment on "modern"
| python, but in general if this code returned an exception
| on division by zero it is definitively worse.
|
| The reason is because an exception is a runtime check. It
| caught an error that is hidden until runtime. The better
| way is to encode this logic into a static type checking
| system so this error isn't even permitted to run.
|
| With the type Optional[Int], Any other function that
| utilizes the output type of this function MUST also be an
| Optional[Int] and not just an Int, meaning that the
| function signature must handle the None case and this is
| caught statically before run time. Insofar as the function
| type signature... python type checkers should handle this
| error.
|
| However tbh I'm not sure how extensive type checkers for
| python are and whether or not they can track this type of
| error for the logic itself. For example while the below
| will catch a type error if you feed it a bad parameter, the
| signature itself could be wrong and not caught by a type
| checker: x1: Int = 2 # results in type
| error (Good) x2: Optional[Int] = None # results in
| runtime error. This is bad in terms of safety
| def addOne(x: Optional[Int]) -> Optional[Int]:
| return x + 1
|
| As far as I know the above is permitted by python type
| checkers when truly correct code is below:
| def addOne(x: Optional[Int]) -> Optional[Int]:
| return x + 1 if x is not None else None
|
| However with python 3.10 pattern matching the level of
| static type checking can increase to the point where this
| type of safety is 100% possible:
| https://www.python.org/dev/peps/pep-0636/.
| x1: Int = 2 # results in type error (Good) x2:
| Optional[Int] = None # No error (also good)
| def addOne(x: Optional[Int]) -> Optional[Int]:
| match x: case None: # if this case were
| deleted should result in static type error (also good)
| return None case _:
| return x + 1
|
| Not sure if python type checkers can handle the above code,
| but exhaustive type checking is 100% viable in the future
| if the syntax utilizes the pattern matching shown above.
| Haskell and Rust already have this level of type safety.
| See: https://rustc-dev-guide.rust-lang.org/pat-exhaustive-
| checkin...
|
| What this means in short is that you are wrong. Whatever
| your notions of "modern" python should be having division
| by zero return a runtime error instead of a None or an
| Error type is definitively worse. We are at a point in our
| technology that this type of error should be caught
| automatically BEFORE a program even runs. All type errors
| should be encoded into types and caught via static type
| checking and NOT via exceptions.
| lkitching wrote:
| For division you really should put the requirement for
| the non-zero check in the type of the divisor instead of
| propagating failure to the caller e.g.
| div(x: Int, y: NonZero[Int]) -> Int
|
| Exceptions at least have the benefit of retaining the
| location the context for the error occurred which gets
| lost without extra bookeeping when using optionals.
| Spivak wrote:
| This ends up being a super unergonomic API in practice
| because you would need to modify the type-checker to
| infer positive integers correctly. Nobody wants to go
| through the effort of NonZero =
| NewType('NonZero', int) if x != 0
| x = NonZero(x)
|
| just to call your function, especially since you still
| have to handle the case where someone does NonZero(x)
| blindly without checking. Should the constructor throw?
| lkitching wrote:
| The check has to go somewhere and the caller to div has
| the most context in the event the divisor is 0. If you
| return an optional from div then you either impose the
| check on all the callers, or just propagate None
| everywhere and some top-level function has to deal with
| mysteriously missing values.
|
| The NonZero type should be responsible for checking the
| wrapped value is non-zero, you will probably want safe
| and unsafe constructor functions Int ->
| Optional[NonZero[Int]] Int -> NonZero[Int]
|
| where the unsafe version throws.
|
| If this is overkill for your application then I'd prefer
| throwing an exception in div rather than encoding the
| failure in the return type.
| globular-toast wrote:
| This is probably all correct, but man... this just isn't
| fun at all. This isn't why I got into programming.
| deltaonefour wrote:
| I'm sorry you don't find programming fun. I find it fun.
| And if you don't like this stuff, maybe it's the wrong
| career choice for you.
| tzs wrote:
| I'm not very familiar with Python type checkers, so what
| follows may be completely off base.
|
| > The reason is because an exception is a runtime check.
| It caught an error that is hidden until runtime. The
| better way is to encode this logic into a static type
| checking system so this error isn't even permitted to run
|
| But in def div(x: Int, y: Int) ->
| Optional[Int]: return x // y if y != 0 else
| None
|
| don't you have a runtime check? It has to do a runtime
| check of the value of y in order to decide whether or not
| to return None?
|
| I don't see how this is encoding the logic of dealing
| with divide by zero into the type system or permitting
| the error to run. I'd expect that to deal with it in the
| type system to not prevent it at runtime would require y
| to be a type that does not allow 0 (does Python's type
| annotations allow constrained integers?).
|
| Handling it by returning None means the caller has to
| deal with None. If it deals with it by just return None
| too, then its caller needs to deal with it do. If
| whatever code finally really deals with it instead of
| just passing it upwards is farther up than div's caller
| or maybe div's caller's caller it seems like what you've
| ended up with is something that is less clear than
| exceptions and requires more runtime work.
|
| For those cases where the return None approach is better
| than using an exception, can you get rid of the runtime
| check for y being non-zero with something like
| def div(x: Int, y: Int) -> Optional[Int]: try:
| return x // y except ZeroDivisionError:
| return None
|
| or do exceptions in Python occur sufficient overhead when
| not taken to make explicitly checking y win?
| deltaonefour wrote:
| It encodes it into the type system at the function
| signature level. Optional[Int] is an Int with a None, so
| if you pass this type into f(x: Int) you get a type error
| because f(x: Int) is a function that deals with Ints, not
| Optional[Int]. So you get a degree of safety here.
|
| As I mentioned this safety does not usually extend to the
| function definition. I stated that it can though (and it
| is in haskell and rust), and this is usually done through
| a feature called pattern matching. Pattern matching is
| available in python 3.10 but I'm not sure whether
| exhaustive type checking on pattern matching by external
| type checkers is available yet.
| hermitdev wrote:
| Python, historically, has not had zero-overhead
| exceptions. There is a runtime cost associated with a try
| block. 3.11 will be adding zero-overhead exceptions,
| though, per https://bugs.python.org/issue40222.
| jcranberry wrote:
| Haskell throws an exception and rust panics on division
| by zero for ints.
|
| This is just defensive programming.
|
| The None state here is already an invalid state. If you
| want to use the type system to ensure correctness you
| would change the second argument to be a type
| representing multiplicatively invertible integers.
| deltaonefour wrote:
| You need to employ pattern matching for exhaustive type
| checking to be deployed.
|
| For haskell it will give you a warning. For Rust it will
| actually not compile. The key here is that you MUST use
| pattern matching.
| jcranberry wrote:
| If the compiler can figure out that you are dividing by
| zero at compile time then you will get those results.
| That is not a realistic expectation in general.
|
| Having to wrap every division call in a maybe monad or
| rust result enum is bad from a usability standpoint and
| potentially unacceptable from a performance standpoint.
|
| Exhaustive pattern matching is great. But not a
| replacement for other types of error handling.
| deltaonefour wrote:
| Performance is a trade off and usability is arguable.
| There are ML style languages where a Maybe monadic value
| is the default return value of division due to the
| division by zero issue.
|
| There are languages that employ this philosophy
| everywhere. Such languages are actually incapable of
| having a runtime error outside of edge cases like ffi.
|
| Just so you know, you don't have to wrap every type in an
| enum. Types and enums share a bijective relationship. A
| type IS an enum and vice versa. You CAN in fact define an
| Int in terms of an enum (and an enum in terms of an int,
| in C). The mechanisms behind an Int and a bool are
| roughly the same, it's just the cardinality and mappings
| that are different. Pattern matching implementations
| usually check types based off of cardinality and do not
| differentiate between enums and other built in types like
| floats or ints.
| pxc wrote:
| Why not? If there is no quotient, None seems like a good
| answer to the question 'what's the quotient?'
| deltaonefour wrote:
| I can see where he's coming from. It's the typical
| reaction of a beginner programmer.
|
| None/Null in many languages isn't type checked even if
| the language itself has static type checking. It can be
| passed to other functions mysteriously and remain a
| hidden error. Javascript and C++ are guilty of this. He's
| complaining of a safety issue most likely.
|
| However note the type signature I specified Optional[Int]
| encodes the None type into the static type checker. In
| modern programming this implies Null/None safety at the
| type level.
|
| I'm not up to date with the current state of python type
| checkers so those type checkers may let some things pass
| depending on some options you set. See my reply to the
| other guy for a more detailed expose.
| globular-toast wrote:
| > I can see where he's coming from. It's the typical
| reaction of a beginner programmer.
|
| This seems like the typical response of someone who posts
| bait ready to flex their "advanced programmer" knowledge.
| There's really no need to call out people's level of
| experience, ever. Personally I'm past the point where I
| think I've learnt everything so I just ask questions when
| I don't know.
| deltaonefour wrote:
| I am not calling anyone out. Let me be absolutely clear
| with you, there is NOTHING and I mean absolutely nothing
| wrong with asking questions or being a beginner
| programmer.
|
| The problem with your earlier post was that you weren't
| asking a question. You were stating something as if it
| were a fact.
| globular-toast wrote:
| I only stated that division by zero shouldn't return
| None, which you seem to agree with, no? The difference is
| how you ensuring that doesn't happen. You really only
| needed to say "this let's you do a static check to make
| sure you're not dividing by zero".
| deltaonefour wrote:
| No. I don't agree, I'm saying the None is better so long
| as you encode the concept of a None into the type. This
| is done by converting an Int into Optional[Int].
| globular-toast wrote:
| It's not that there's no quotient, it's that you
| shouldn't even have asked what the quotient is.
| yxhuvud wrote:
| The thing about exceptions is that they are runtime.
| Pushing the issue to the type system forces the user to
| handle the case.
| hsn915 wrote:
| The fallacy of premature optimization rears its ugly head again.
|
| When Knuth wrote that quote, he meant something very specific
| with the word optimization: low level micro optimizations.
| Spending a lot of time trying to get every little ounce of
| performance from every little CPU instruction.
|
| High level reasonable design decisions are not "optimizations" in
| this sense at all. You should think of them as non-pessimization.
|
| Choosing Python or another slow language is premature
| pessimization. You just make your program slower for no reason.
|
| So choosing a language based on how programs written in it
| perform is simply non-pessimization.
|
| If you are interested in an expansion on this idea, checkout
| Casey Muratori's lecture about philosophies of optimization:
| https://www.youtube.com/watch?v=pgoetgxecw8
|
| The article makes claims about software being "fast enough" and
| not needing further optimization. But what you call "fast enough"
| is probably 10000 times slower than what it could be.
|
| I'm not joking.
|
| People who program in slow languages have a really skewed
| perception about performance.
|
| If a page takes 5 seconds to load, they don't see that as a
| problem.
|
| If a server takes 500ms to respond to a request, they don't see
| that as a problem.
|
| This is simply unacceptable.
| dragonwriter wrote:
| > High level reasonable design decisions
|
| Viewing language choice as a high level design decision for a
| project of nontrivial scale is a symptom of bad high-level
| design decisions.
| flohofwoe wrote:
| OTH not choosing Python for tasks where the language
| performance really doesn't matter at all (for instance shell
| scripting stuff) would also be futile. It's simply a matter of
| using the right tool for a specific job (and in Python's case,
| that 'tool' isn't even the language, but the batteries-included
| stdlib).
| hsn915 wrote:
| I don't buy into this argument either.
|
| Using python for shell scripting requires extensive knowledge
| about the language and the standard library and all the
| gotchas that you can fall into.
|
| This sort of time investment does not pay off if you just
| want to write small scripts that perform simple tasks.
|
| If the task is simple, then a bash script would be better.
|
| If the task is not so simple, then a language you are really
| familar with is better.
|
| If you accept my premise about how programming in slow
| languages is generally a bad idea, you would not spend that
| much time learning any such language with all its ins and
| outs and gotchas.
|
| For background, I personally _have_ spent a lot of time
| learning python, and only later in life have a realizd what a
| mistake that was. Now I only use Go for server side
| programming. I 'm also using it for command-line programs.
| Such programs usually start small but grow over time.
| Choosing Python for them would have been a costly mistake.
| BiteCode_dev wrote:
| Talking about gotchas and then recommend bash is not a sign
| of credibility in my book.
|
| Bash has one strong point: 4 to 5 lines admin scripts for
| which concision and expressiveness matter more than
| anything else. That's it.
|
| There is nothing else it can be better at than ruby,
| python, perl, php, or even go or kotlin.
|
| The argument you need to learn x stands for bash as well,
| btw.
| hsn915 wrote:
| I only said to use bash for small scripts. Basically just
| a series of commands. Maybe setting up one or two
| environment variables. Anything more complex or involving
| "if" I would almost never write in bash.
| flohofwoe wrote:
| > If the task is simple, then a bash script would be
| better.
|
| Bash script don't work on the Windows command line, and
| bash relies on UNIX tools as its "standard library", which
| also are not available on the Windows command line. That's
| the whole point really, Python has very good cross-platform
| support (and in this case cross-platform doesn't just mean
| "Linux and macOS", but also Windows).
|
| > If the task is not so simple, then a language you are
| really familar with is better.
|
| IMHO every programmer should be familiar with at least a
| handful programming languages.
| goohle wrote:
| Just write in the header of the script: "This script
| requires Fedora Linux 33 or higher" and forget about
| outdated OS'es, unless it's your paid job to supported
| them.
|
| If you OS vendor refuses to supply you with appropriate
| tools to do your job, then just chose another vendor.
| Vote with your foots.
| hsn915 wrote:
| Bash can and does work on windows if you install it. Just
| like Python.
|
| Also doesn't windows now ship with a linux compatibility
| layer anyway (WSL)?
|
| EDIT:
|
| I just realized you are the guy who wrote sokol!
|
| If your main work station is windows and you don't work
| on linux, then replace bash from my previous comment with
| bat(ch).
| flohofwoe wrote:
| Bash is not the standard shell on Windows, and it also
| relies very much on UNIX-style filesystem conventions
| that don't map very well to the Windows environment. I
| wouldn't ask my users to install PowerShell on Linux or
| macOS either.
|
| Asking Windows users to run Bash scripts is essentially
| the same as asking them to build a C/C++ project that
| uses Autotools and Makefiles. It may work with lots of
| tinkering, but it's a massive world of pain.
|
| > If your main work station is windows and you don't work
| on linux, then replace bash from my previous comment with
| bat(ch).
|
| I usually switch between macOS, Linux and Windows at
| least several times a week, sometimes several times a
| day, maintaining different scripts to do the same thing
| wouldn't make sense.
| Supermancho wrote:
| > Bash is not the standard shell on Windows, and it also
| relies very much on UNIX-style filesystem conventions
| that don't map very well to the Windows environment.
|
| Windows was built by people who started by copying from
| Unix. This is part of the reason Bash works fine on
| Windows (ie What is your problem with it "mapping" to
| things?). You can pipe commands, you can stream text, you
| can edit binaries. There are things that slow down (or
| can't be done, modify permissions easily, etc) because of
| Windows tradeoffs, not because it's fundamentally
| different. An OS does OS things and a shell talks to the
| OS.
| majou wrote:
| OK but shell is cuter than python.
| ar_lan wrote:
| Yeah, it seems like he was referring more to bike-shedding than
| anything. Focus on the efficient, meta-goal, and make efficient
| choices for the highest value.
|
| In this case, since the entirety of the project is to be
| written in a specific language, it's usually worthy of placing
| that in the more macro-category - where that choice affects all
| other decisions.
|
| Don't spend 4 hours of your time on a `for` loop but half an
| hour of your time deciding on the language and architecture.
| wiremine wrote:
| > When Knuth wrote that quote, he meant something very specific
| with the word optimization: low level micro optimizations.
|
| Sure, but that isn't to say his advice can't be applied in
| other contexts. Applying optimizations before knowing where a
| bottleneck is just a bad idea, regardless of the situation. To
| constrain the advice to the original context is sort of odd,
| but maybe there's more to it than that?
|
| I always thought the risk of premature optimization was similar
| to the lean principle of deferring commitment: you should to
| wait until the last possible moment to a decision that has high
| switching costs. In both cases you want to have as much
| information as possible to taking action.
|
| As an aside, this bugged me for a long time: how _much_
| information do you need? Recently I co-presented with the CEO
| of a successful manufacturing company. At his firm they use the
| military's rule of thumb of taking action when you have 70% of
| the information. Deciding sooner than that and you risk making
| a bad decision; Waiting for more information after 70% and you
| likely are missing an opportunity.
| dragontamer wrote:
| > Sure, but that isn't to say his advice can't be applied in
| other contexts. Applying optimizations before knowing where a
| bottleneck is just a bad idea, regardless of the situation.
| To constrain the advice to the original context is sort of
| odd, but maybe there's more to it than that?
|
| Donald Knuth writes entire volumes of books in assembly
| language, because the details matter. If you want to have
| that point, then you probably shouldn't be quoting Donald
| Knuth.
|
| Donald Knuth is the guy who points out that "for(int
| i=size-1; i>=0; i--)" is faster than "for(int i=0; i<size;
| i++)". When he says "don't sweat the small stuff", its
| because his books are filled with _incredibly_, detailed
| stuff. Far more detail than any other programming book /
| algorithms book.
|
| IIRC, the "premature optimization is the source of evil"
| statement was about how GOTO statements and how efficient
| they were. The __context__ is that GOTO is indeed a very fast
| construct (especially in the context of replacing recursion).
| rvbissell wrote:
| If one cared _that_ much about speed, they 'd use `--i`,
| not `i--`
| dragontamer wrote:
| Knuth wrote that discussion point in assembly language.
|
| I translated it to C so that people got the gist of it.
| Jtsummers wrote:
| Per godbolt with GCC 11 and clang 13, there is no
| difference between those two. Both end up compiled to the
| same machine code (though GCC and clang both compile them
| differently, both are consistent with themselves with
| respect to pre or post decrement).
|
| https://godbolt.org/z/qbEo9GzEx
| Jtsummers wrote:
| He made the comment a few times, apparently, in the context
| of GOTO it was this:
|
| > Programmers waste enormous amounts of time thinking
| about, or worrying about, the speed of noncritical parts of
| their programs, and _these attempts at efficiency actually
| have a strong negative impact when debugging and
| maintenance are considered._ We should forget about small
| efficiencies, say about 97% of the time: premature
| optimization is the root of all evil. Yet we should not
| pass up our opportunities in that critical 3%.
|
| From: "Structured Programming with Goto Statements",
| emphasis mine.
|
| Archive link: https://web.archive.org/web/20131023061601/ht
| tp://cs.sjsu.ed... (I haven't read it in a while, and it's
| 41 pages, so I probably won't get to it today.)
| hsn915 wrote:
| The advice doesn't generalize to high level "optimizations".
| The problem with optimizations (low level) is they make the
| code more complicated and more difficult to understand. This
| is a problem. You better have really good payout. (It also
| takes a lot of time to apply micro optimizations to
| everything, so if you agonize about every CPU instruction you
| will never get anything done).
|
| But, general high level design decisions that don't produce
| slow code by default do not suffer from this. They don't make
| the code any complicated or hard to understand. Quite the
| opposite, they usually simplify things. Because most of the
| time, the way to get good performance is to do the minimal
| thing that needs to be done to perform the task at hand. The
| code you write will be very straight forward. There's no
| switching cost.
|
| On the other hand, choosing a slow language has a high
| switching cost. You can't just take a bad design and try to
| optimize it. It usually doesn't work. You have to rework the
| whole design.
|
| EDIT:
|
| I have one other important point to add:
|
| > Applying optimizations before knowing where a bottleneck is
| just a bad idea, regardless of the situation.
|
| Ignoring any and all performance concerns using this kind of
| reasoning will result in a codebase that is permeated with
| slowness. The program is slow but you can't identify any
| single bottleneck. You end up with a monster that simply
| can't be fixed.
|
| Worse, you may not even realize how slow it actually is. If
| you have profiled it and found no particular bottle neck, you
| may think this simply the limit of computer performance and
| the task at hand cannot be performed any faster than this.
| taeric wrote:
| I love that you can replace "optimization" in your first
| paragraph with "abstraction" and have the same results. :D
|
| This actually leads me to see the concept of "micro
| abstractions", and I'm going to see if I see those in hard
| to deal with programs, now.
| Jtsummers wrote:
| > the lean principle of deferring commitment: you should to
| wait until the last possible moment to a decision that has
| high switching costs.
|
| Last _responsible_ moment, not last _possible_ , moment.
| That's a big difference. You shouldn't start on the work at
| the last minute, that's too late. But you also don't
| (necessarily) need to start on something 2 years before it's
| due (unless it's big and/or there are anticipated problems
| that will need to be addressed that the time would permit;
| that's what makes it the last _responsible_ moment).
| wiremine wrote:
| Agreed, I should clarify that in my original comment.
| xapata wrote:
| How do you know when you've had 7/10ths of "the information"?
|
| There's a great deal of scientific literature about this
| topic, BTW. Especially in machine learning.
| taeric wrote:
| My favorite aspect of Knuth's take, is that he definitely
| concerns himself with the low level aspects of programming when
| writing things.
|
| I don't mean that as a criticism, either. I keep meaning to
| strive to having the level of cognizance to memory usage that
| he just drips into his software.
| cletus wrote:
| 100% this.
|
| It's now right up there with "X considered harmful", which 80%
| of the time translates to "I don't like X".
|
| Specific to Python from the article, I'll add that I personally
| am done with dynamically typed languages for anything where
| more than one person needs to work on it or it needs to be
| long-lived. Static typing is simply too useful. Dynamic typing
| is (IMHO) a false economy. I've often said "Python means
| writing unit tests for spelling mistakes". Refactoring a
| dynamically typed code base of any reasonable size is a
| comparative nightmare.
| goohle wrote:
| It's same story again and again.
|
| Shell good for simple scripts. -- Shell scripts are pain to
| support!
|
| Perl is astonishingly good for text manipulation. -- Perl is
| write-only language!
|
| Python is easy to learn and start coding. -- Python2 to
| Python3 migration is decades long hell!
|
| JavaScript is the language of the World Wide Web. --
| JavaScript is nightmare to refactor!
|
| And so on.
|
| If language easily accepts anything that developer pops out
| of his head, then program will be hard to support and
| refactor.
|
| If language limits developer to certain good, safe,
| performant solutions, then it hard to write programs in it,
| because you will need to learn good patterns first, which may
| take years.
| deckard1 wrote:
| > hard to write programs in it, because you will need to
| learn good patterns first, which may take years.
|
| People really underestimate the time it takes for devs to
| figure this out. My company has been on TypeScript for over
| three years now and it's a giant mess. Everything is
| arbitrarily typed with no design philosophy or organization
| behind anything. Which leads to these massive interfaces
| with every single field being optional. Which means you're
| _still_ doing run-time checking all over the place.
|
| I've always maintained that if your team sucks at
| organization then no amount of tooling will save them.
| You're not so much working as a team but as bitter enemies.
| Possibly the turnover is too high in the company. You have
| mercenaries rather than ideologues. Not that I'd want to
| work with a team of pure zealots. But there should be
| enough people holding the architecture in place. It helps
| morale as well. You don't want to work on a project that no
| one gives a shit about.
| jefb wrote:
| From my experience most of your assertions have a hidden
| assumption that the software requirements are complete and they
| are good. Computing something 10000 times faster is useless if
| it turns out to be the wrong computation. Furthermore, things
| that run 10000 times faster than other things tend to be harder
| to adapt and change.
|
| Requirements that are good enough to warrant a 5 second page
| load time being "unacceptable" must have, at some point, been
| prototyped with real use cases, real data, and in real
| environments.
|
| Getting to that point - where the business case is connected
| with such a strong through-line to the user experience can
| literally take years. Early in a project lifecycle it's more
| important to design software that is easy to change than
| software that is fast.
|
| Language choice does not matter, performance does not matter;
| the design philosophy of the developer(s) when building
| something new is everything. That philosophy should be "I'll
| likely have to change this" not "I need to make sure this runs
| quickly". Any high level design decisions should be made with
| the former in mind, not the latter.
|
| Once you've earned that product wisdom, sure - carve out the
| slow thing and write it in CUDA or something. If things are
| easy to change, that's not a scary proposition; it's just part
| of the natural project lifecycle and Knuth lives to die another
| day.
| [deleted]
| lucasyvas wrote:
| In the current age of software, we spend way less time
| designing with any future thought whatsoever. In that respect,
| I concur - the article's position is damaging.
|
| You can design slow crap in any language, but I'd rather use
| something that has the potential to be robust and fast for my
| problem domain so I don't have to rewrite the entire app in the
| future to language "X", when I could have easily started with
| "X".
|
| Make no mistake - if you do this, you made an upfront error in
| selecting your language for the business problem. It may not be
| an unrecoverable error, but it is a failure in judgement
| regardless.
| BiteCode_dev wrote:
| I have a half million users a day streaming website coded in
| Python. It does machine learning, uploading, encoding, dynamic
| load balancing, live search, comments, votes and most of the
| code I had to write was in django or rq endpoints.
|
| On first load, not only the page is ready, but the video has
| started to auto play (including buffering), under 3 seconds.
| Less than 500ms on a reload.
|
| The whole project has been running for 10 years on 7 servers.
| Hosting costs under $1K/mo.
|
| And even better, the code base is a mere 11K LoC, so a single
| dev can manage it. And make a living out of it.
|
| That's because:
|
| - Python is almost never the bottleneck in a website
|
| - Python wraps compiled languages for slow stuff. That's the
| whole point.
|
| - your archi matters much more than the language you chose.
| Because good db providers, proper caching and optimized data
| fetching will be the pareto moves, unless you are google size.
|
| - you will not code most things anyway. You are not going to
| code the video encoding, you will use ffmpeg. You will not
| recode a database, you will use an existing one. You will not
| code a server, you will use nginx or something else. What you
| will spend time on is creating logic for your clients.
|
| So I think your are extrapolating this rare use case where your
| POV makes sense, and ignore the 99% of the case were, indeed, a
| slow language doesn't matter.
|
| In fact, if electron success tells anything, it's that while I
| do value speed a lot, the market has voted.
| rixed wrote:
| > I have a half million users a day streaming website coded
| in Python. It does machine learning, uploading, encoding,
| dynamic load balancing, live search, comments, votes and most
| of the code I had to write was in django or rq endpoints.
|
| Sounds much more likely that you wrote in python the code
| that bundle together the libraries that peform the actual
| network streaming, machine learning, encoding, search, etc,
| and that those libraries were not written in python.
| BiteCode_dev wrote:
| Yes, that's my whole point.
|
| Where in the stack do you see that Python performances
| matter?
|
| Only one software out of a 1000 will actually need it. The
| others will mostly do plumbing or automation.
|
| And scripting languages excel at both.
|
| If you don't fall in that category, then yes, do use a fast
| language. If you are coding a chat server, use Erlang. If
| you want to do a rendering engine, use Rust. But if you
| most likely want to code your company CRUD software and
| glue it to some external services and providers, use
| Python.
| shepardrtc wrote:
| > The article makes claims about software being "fast enough"
| and not needing further optimization. But what you call "fast
| enough" is probably 10000 times slower than what it could be.
|
| As a fun project, I started writing some code to do
| calculations on matrices. Wrote it in Python and used Numpy.
| Then I wrote it in Julia. Almost immediately it was a 100x
| improvement. I spent time doing "premature optimization" and
| netted a 10x improvement. So overall it went from minutes to
| milliseconds.
|
| I guess if you're writing a website or something then I
| wouldn't worry, but for other things, doing optimization from
| the get-go can save you a ton of time and effort later on. And
| really, such optimizations can affect the architecture in ways
| you wouldn't think of without it. Reworking your architecture
| down the road is a huge time sink. I like to make sure mine is
| close to where I want it to be in the beginning.
| [deleted]
| davesnx wrote:
| The author: Selecting a programming language can be a form of
| premature optimization
|
| Also the author: Pick python no matter the problem, the team, the
| libraries, the deployment targets, etc.
| eesmith wrote:
| Where does the author imply something like "Pick python no
| matter the .. etc.?"
|
| I read it as the much more milder "don't reject Python because
| of vague concerns about run-time performance".
|
| I can't find anywhere which suggests the author things people
| should use Python to, for example, code up their web app front-
| ends or to implement a 'hard' real-time operating system.
| davesnx wrote:
| Mi bad english played me bad here. I meant to write "Picks
| python... blabla"
|
| Looked to me like a "Python isn't than bad for perf" article.
| IshKebab wrote:
| "Guys, our Python prototype is finished. It works really well and
| has proved our concept. I know we've put a ton of work into it
| but now I suggest we scrap it and rewrite it in a language that
| isn't dog slow."
|
| - Nobody
| geofft wrote:
| Largely because nobody has ever found Python to be dog slow.
| Put your unoptimized Python into production, you'll be fine.
| gpderetta wrote:
| We run pylint as part of our build, including as a
| prerequisite of merging a branch into master.
|
| Our codebase is well over 1M lines of C++. We have about 100k
| lines of python. Running pylint takes the same order of
| magnitude of time (half as much IIRC) as running a full
| optimizing build + linking of the C++ code base. We run
| pylint over all cores, while we run the C++ build only on a
| subset of cores on the build machine.
|
| I would call that dog slow, unless you think python is not
| the appropriate language write pylint. And no, it is not
| fine.
| IshKebab wrote:
| Dropbox is the biggest user of Python that I know of, and
| they went as far as writing their own JIT compiler before
| giving up and switching to faster languages.
| dragonwriter wrote:
| I agree with the broad concept (with the caveat that there are
| exceptions to all generalizations), but note that "Python" can
| generally be replaced with "whatever general purpose language you
| are comfortable and familiar with that doesn't have any clear
| insurmountable barriers like 'is not available for the desired
| target platform'."
| outsomnia wrote:
| Thesis is dogmatic selection of language is bad: article is about
| using Python under all circumstances.
| 4pkjai wrote:
| Yes, I'd say replace python with [whatever language your team
| is most comfortable with]
| birdstheword5 wrote:
| I've met several people who learned Python and refuse to learn
| any other language. I wonder if this mentality exists for other
| languages
| ohCh6zos wrote:
| I know right around seven languages I would be comfortable
| claiming I knew on a job application. I am probably more
| opinionated about language features, but less picky about the
| specific language for it.
| exyi wrote:
| Oh yes, it's sooo common. I met many C# web developers who
| just refuse to even learn a bit of JS, "because it's soo bad
| language"
| Semaphor wrote:
| I think JS and PHP have a special status here and don't
| count as an answer to what GP was asking. Many people hate
| JS and/or PHP, often for legacy reasons.
| exyi wrote:
| But it's not only JS, I used it as an example because
| these people are doing websites, yet don't even want to
| learn JS. They also hate to write SQL and are stuck with
| ORMs...
| Semaphor wrote:
| That is different, but SQL is also not a general
| programming language. I see what you say as an issue with
| those people, but a different issue ;)
| rhn_mk1 wrote:
| I'm seeing a lot of resistance from C programmers to switch
| to anything else for which I can't find a rational
| explanation.
| npsimons wrote:
| > I wonder if this mentality exists for other languages
|
| Visual Basic, PHP and Java come to mind. Probably C# as well.
| tcbasche wrote:
| Not even at a language level. There are Java developers who
| only want to work with particular frameworks (Spring) and
| don't know anything else or want to learn anything else.
| eesmith wrote:
| The article is not suggesting to use Python under all
| circumstances.
|
| Here's the conclusion, which summarizes the essay:
|
| > All of this is to say while some companies do extract massive
| value from squeezing every CPU cycle out of their code, those
| companies also typically build data centres. So if you don't
| need a dedicated building to host your machines, please
| consider doing the math to see if it's truly worth making your
| developers less productive in the name of computational
| efficiency when you don't even know if that perceived
| efficiency is even necessary.
|
| As an example, the kernel of my software is only a few hundred
| lines long. It's in C with AVX2 intrinsics to eek out the last
| bit of performance. The Python overhead is <1%, which means
| that even swapping out Python with a 1,000x faster language
| won't be noticeable. This is the '"glue" code' mentioned in the
| section on 'Use language bindings'.
|
| Had I started with "It needs to be fast therefore I can't use
| Python", then it probably would have been a lot more work to
| do. (See, "numerous anecdotes of where someone implemented
| something in Python in 1/3 the time a competing team did
| creating the same thing in e.g. C++ or Java".)
| pyrale wrote:
| > The article is not suggesting to use Python under all
| circumstances.
|
| I would say the article is very much doing that, it's just
| stopping short from spelling it out clearly.
|
| It starts from the flawed premises that Python is much more
| effective to come up with actual production code than other
| languages [1].
|
| The author then moves on to telling you that you should try
| at least a couple of times to code something in python before
| obvious performance shortcomings are glaring, then keep
| python as a binding language.
|
| At no point does the author acknowledge that there may be
| other reasons to pass over python than performance.
|
| From the way he approaches the subject, the author seems to
| live in the paradise where you only have to sling half-baked
| prototypes to prod, and then move on to another project while
| poor souls handle maintenance.
|
| The take on why performance matters [2] also completely
| overlooks the fact that optimizing for infra costs is likely
| the least common reason to write for performance. Sometimes,
| if you don't reach a given threshold, your code simply _doesn
| 't_ work. Sometimes, your performance is your users' time
| (and sometimes, these users are even developers). Such
| companies don't necessarily build datacenters.
|
| [1]: "This is what leads to numerous anecdotes of where
| someone implemented something in Python in 1/3 the time a
| competing team did creating the same thing in e.g. C++ or
| Java." -- of course, it's just anecdotes and not a claim, so
| the author won't defend it, but you're still expected to
| believe it.
|
| [2]: "All of this is to say while some companies do extract
| massive value from squeezing every CPU cycle out of their
| code, those companies also typically build data centres."
| eesmith wrote:
| I agree with those who say "software engineering" is not
| "engineering." We as a field are more comfortable with our
| beliefs, supported by anecdotal evidence, than with
| empirical results.
|
| That said, _the few and limited empirical results which do
| exist_ , like Prechelt's empirical work from 20 years ago,
| _support the author 's premise_.
|
| As another example, Boehm's work across a range of projects
| (<100KLoc) shows that code size has the biggest impact on
| development effort, and slightly non-linear scaling in KLoC
| (KLoC^~1.1 in COCOMO); and various publications show Python
| generally requires fewer lines of code.
|
| (As one example picked semi-arbitrarily from a Google
| Scholar search, http://jucs.org/jucs_19_3/a_comparison_of_f
| ive/jucs_19_03_04... on a small project has Python at the
| smallest effective LOC, smallest effective byte size, and
| nearly the smallest (after Java) eByte/eLOC size.)
|
| Modern C++ is a much more compact (and complicated!)
| language than 20 years ago, so of course all of these
| previous results must be reviewed and reconsidered.
|
| As a field, we have not provided that funding for solid
| empirical research!
|
| > At no point does the author acknowledge that there may be
| other reasons to pass over python than performance.
|
| Shrug. Okay. And from that you conclude the author is
| saying that Python should be used in all cases? "I'm going
| to talk about why you should eat more vegetables" doesn't
| mean "you should only ever eat vegetables." Similarly, "I'm
| going to talk about why you shouldn't reject Python from
| the get-go for a project you think is performance
| intensive" doesn't mean "you should only ever use Python."
|
| The author even talks about Rust in an earlier blog entry,
| from August of this year, at https://snarky.ca/introducing-
| the-python-launcher-for-unix/ saying "And so over 3 years
| ago I set out to re-implement the Python Launcher for Unix
| in Rust. On July 24, 2021, I launched 1.0.0 of the Python
| Launcher for Unix".
|
| So the evidence is that Cannon does not believe that Python
| is the best language for all projects.
|
| > Sometimes, your performance is your users' time
|
| Cannon's essay is about the case where people select a
| programming language based strongly on expected
| computational cost, and forget other factors in the cost
| model. Here's the relevant quote:
|
| ] I think the jump to selecting a programming language
| based on potential performance needs often comes from a
| place where people think their computation costs are more
| important to optimize for than their developer time. I
| don't think that always holds, though, as software
| developers are expensive.
|
| In your model, people are choosing a language based in part
| of the cost of users' time, which means they they are _not_
| primarily focused on computation costs, and so are outside
| the scope of this essay. That doesn 't contradict the
| essay, only show that it's incomplete. Which we know
| already from the text itself.
|
| > Sometimes, if you don't reach a given threshold, your
| code simply doesn't work.
|
| And sometimes you can reach a given threshold with an off-
| the-shelf use of Pandas, rather than build your own
| analysis routine. If you start off thinking "Python must be
| slow" and don't even know what your thresholds or
| performance characteristics are, then selecting (say)
| assembly over Python is probably premature optimization.
|
| Cannon didn't even argue that if you aren't building data
| centers then you must use Python. Your [2] trims the full
| paragraph I quoted earlier. You left out: a) the "please
| consider", which makes this a request, and not a rejection
| of any alternatives; b) the request to also _consider_
| developer costs, because essay argues people should
| consider more than computational costs; and c) the "you
| don't even know if that perceived efficiency is even
| necessary", which ties the essay back to Knuth's "premature
| optimization is the root of all evil" quote.
| stonemetal12 wrote:
| I took python more as a motivating example than an absolute. As
| far as I can tell the article suggests developer speed over
| application performance. Then FFI to a "fast" language to
| overcome your performance problems. That way you only pay the
| overhead of a high performance, slow to develop language where
| it actually matters. Which is usually much less than the whole
| program, so not paying for performance where it isn't needed is
| a big win.
| everyone wrote:
| Just use what you know. Something you've used before.
| KingOfCoders wrote:
| This is always the case with me, once with Scala, now with Rust.
| agilob wrote:
| >Selecting a programming language can be a form of premature
| optimization
|
| >Prototype in Python.
|
| Stopped reading here.
|
| Wait, what? The only thing I know about python is that it's
| indentation-sensitive, no idea about syntax or libraries.
| Suggesting me to use python is premature optimisation.
| geofft wrote:
| If you continued reading past the point where you didn't know
| things, you would know more things. :)
| j4mie wrote:
| My (perhap controversial) take is that static typing in Python is
| also a form of premature optimisation. Code should be written
| without static types first, and static types should only be added
| when absolutely necessary. 99% of the time, it never will be.
| IshKebab wrote:
| Why though? Static types make it easier to write code overall,
| so you're slowing yourself down for no real benefit.
|
| You could just as easily say "Descriptive variable names are a
| form of premature optimisation. Code should be written with
| single-letter names, and descriptive names only added when
| absolutely necessary."
___________________________________________________________________
(page generated 2021-11-15 23:03 UTC)