[HN Gopher] Command-line Tools can be 235x Faster than your Hado...
___________________________________________________________________
Command-line Tools can be 235x Faster than your Hadoop Cluster
(2014)
Author : tosh
Score : 274 points
Date : 2026-01-18 08:58 UTC (14 hours ago)
(HTM) web link (adamdrake.com)
(TXT) w3m dump (adamdrake.com)
| killingtime74 wrote:
| Hadoop, blast from the past
| ejoebstl wrote:
| Great article. Hadoop (and other similar tools) are for datasets
| so huge they don't fit on one machine.
| vjerancrnjak wrote:
| https://www.scylladb.com/2019/12/12/how-scylla-scaled-to-one...
|
| I like this one where they put a dataset on 80 machines only
| then for someone to put the same dataset on 1 Intel NUC and
| outperform in query time.
|
| https://altinity.com/blog/2020-1-1-clickhouse-cost-efficienc...
|
| Datasets never become big enough...
| literalAardvark wrote:
| Well yeah, but that's a _very_ different engineering decision
| with different constraints, it's not fully apples to apples.
|
| Having materialised views increases insert load for every
| view, so if you want to slice your data in a way that wasn't
| predicted, or that would have increased ingress load beyond
| what you've got to spare, say, find all devices with a
| specific model and year+month because there's a dodgy lot,
| you'll really wish you were on a DB that can actually run
| that query instead of only being able to return your
| _precalculated_ results.
| DetroitThrow wrote:
| >Datasets never become big enough...
|
| Not only is this a contrived non-comparison, but the
| statement itself is readily disproven by the limitations
| basically _everyone_ using single instance ClickHouse often
| run into if they actually have a large dataset.
|
| Spark and Hadoop have their place, maybe not in rinky dink
| startup land, but definitely in the world of petabyte and
| exabyte data processing.
| zX41ZdbW wrote:
| When a single server is not enough, you deploy ClickHouse
| on a cluster, up to thousands of machines, e.g.,
| https://clickhouse.com/blog/how-clickhouse-powers-ahrefs-
| the...
| saberience wrote:
| Well, at my old company we had some datasets in the 6-8 PB
| range, so tell me how we would run analytics on that dataset
| on an Intel NUC.
|
| Just because you don't have experience of these situations,
| it doesn't mean they don't exist. There's a reason Hadoop and
| Spark became synonymous with "big data."
| dapperdrake wrote:
| These situations are _rare_ not difficult.
|
| The solutions are well known even to many non-programmers
| who actually have that problem:
|
| There are also sensor arrays that write 100,000 data points
| per millisecond. But again, that is a hardware problem not
| a software problem.
| PunchyHamster wrote:
| And we can have pretty fucking big single machines right now
| dapperdrake wrote:
| https://yourdatafitsinram.net/
| rcarmo wrote:
| This has been a recurring theme for ages, with a few companies
| taking it to extremes--there are people transpiring COBOL to bash
| too...
| paranoidrobot wrote:
| A selection of times it's been previously posted:
|
| (2018, 222 comments)
| https://news.ycombinator.com/item?id=17135841
|
| (2022, 166 comments)
| https://news.ycombinator.com/item?id=30595026
|
| (2024, 139 comments)
| https://news.ycombinator.com/item?id=39136472 - by the same
| submitter as this post.
| nasretdinov wrote:
| And now with things like DuckDB and clickhouse-local you won't
| have to worry about data processing performance ever again. Just
| kidding, but especially with ClickHouse it's so much better to
| handle the large data volume compared to the past, and even a
| single beefy server is often enough to satisfy all data analytics
| needs for a moderate-to-large company.
| torginus wrote:
| When I worked as a data engineer, I rewrote some Bash and Python
| scripts into C# that were previously processing gigabytes of JSON
| at 10s of MB/s - creating a huge bottleneck.
|
| By applying some trivial optimizations, like streaming the
| parsing, I essentially managed to get it to run at almost disk
| speed (1GB/s on an SSD back then).
|
| Just how much data do you need when these sort of clustered
| approaches really start to make sense?
| embedding-shape wrote:
| > I rewrote some Bash and Python scripts into C# that were
| previously processing gigabytes of JSON
|
| Hah, incredibly funny, I remember doing the complete opposite
| about 15 years ago, some beginner developer had setup a whole
| interconnected system with multiple processes and what not in
| order to process a bunch of JSON and it took forever. Got
| replaced with a bash script + Python!
|
| > Just how much data do you need when these sort of clustered
| approaches really start to make sense?
|
| I dunno exactly what thresholds others use, but I usually say
| if it'd take longer than a day to process (efficiently), then
| you probably want to figure out a better way than just running
| a program on a single machine to do it.
| commandersaki wrote:
| How do you stream parse json? I thought you need to ingest it
| whole to ensure it is syntactically valid, and most parsers
| don't work with inchoate or invalid json? Or at least it
| doesn't seem trivial.
| rented_mule wrote:
| I don't know what the GP was referring too, but often this is
| about "JSONL" / "JSON Lines" - files containing one JSON
| object per line. This is common for things like log files.
| So, process the data as each line is deserialized rather than
| deserializing the entire file first.
| shakna wrote:
| There's a whole heap of approaches, each with their own
| tradeoffs. But most of them aren't trivial, no. And most end
| up behaving erratically with invalid json.
|
| You can buffer data, or yield as it becomes available before
| discarding, or use the visitor pattern, and others.
|
| One Python library that handles pretty much all of them, as a
| place to start learning, would be:
| https://github.com/daggaz/json-stream
| giovannibonetti wrote:
| You assume it is valid, until it isn't and you can have
| different strategies to handle that, like just skipping the
| broken part and carrying on.
|
| Anyway, you write a state machine that processes the string
| in chunks - as you would do with a regular parser - but the
| difference is that the parser is eager to spit out a stream
| of data that matches the query as soon as you find it.
|
| The objective is to reduce the memory consumption as much as
| possible, so that your program can handle an unbounded JSON
| string and only keep track of where in the structure it
| currently is - like a jQuery selector.
| torginus wrote:
| I used Newtonsoft.Json which takes in a stream, and while it
| can give you objects, it can also expose it as a stream of
| tokens.
|
| The bulk of the data was in big JSON arrays, so you basically
| consumed the array start token, then used the parser to
| consume an entire objects which could be turned into a C#
| object by the deserializer, then you consumed a comma or end
| array token until you ran out of tokens.
|
| I had to do it like this because DS-es were running into the
| problem that some of the files didn't fit into memory. The
| previous approach took 1 hour, involved reading the whole
| file into memory and parsing it as JSON (when some of the
| files got over 10GB, even 64GB memory wasnt enough and the
| system started swapping).
|
| It wasn't fast even before swapping (I learned just how slow
| Python can be), but then basically it took a day to run a
| single experiment. Then the data got turned into a dataframe.
|
| I replaced that part of the Python code processing and
| outputted a CSV which Pandas could read without having to
| trip through Python code (I guess it has an internal
| optimized C implementation).
|
| The preprocessor was able to run on the build machines and
| DSes consumed the CSV directly.
| briHass wrote:
| This sounds similar to how in C#/.NET there are (at least)
| 3 methods to reading XML: XmlDocument, XPathDocument, or
| XmlReader. The first 2 are in-memory object models that
| must parse the entire document to build up an object
| hierarchy, which you then access object-oriented
| representations of XML constructs like elements and
| attributes. The XmlReader is stream-based, where you handle
| tokens in the XML as they are read (forward-only.)
|
| Any large XML document will clobber a program using the in-
| memory representations, and the solution is to move to
| XmlReader. System.Text.Json (.NET built-in parsing) has a
| similar token-based reader in addition to the standard
| (de)serialization to objects approach.
| zahlman wrote:
| Would for example https://pypi.org/project/json-stream/
| have met your needs?
| torginus wrote:
| I'm going to go out on a limb and say no - this library
| seems to do the parsing in Python, and Python is _slow_ ,
| like many times slower than Java, C# or languages in this
| class - which you find out if you try to do heavy data
| processing with it, which is one of the reasons I dislike
| the language. It's also very hard to parallelize - in C#
| if you feed stuff into LINQ and entries are independent,
| you can make the work parallel with PLINQ very quickly,
| while threads aren't really a thing in Python (or at
| least they werent back then).
|
| I've seen so many times that data processing quickly
| became a bottleneck and source of frustration with Python
| that stuff needed to be rewritten, that I came to not
| bother writing stuff in Python in the first place.
|
| You can make Python fast by relying on NumPy and pandas
| with array programming, but doing so can be quite
| challenging to format and massage the data so that the
| things that you want can be expressed as array
| programming ops, that usually it became too much of a
| burden for me.
|
| I wish Python was at least as fast as Node (which also
| can have its own share of performance cliffs)
|
| It's possible that nowadays Python has JITs that improve
| performance to Java levels while keeping compatibility
| with most existing code - I haven't used Python
| professionally in quite a few years.
| zahlman wrote:
| From the README, features include:
|
| > native code parsing speedups for most common platforms
|
| Which is to say, roughly analogous to "relying on NumPy".
| (A well-designed system avoids repeatedly calling from
| Python to C and prefers to let loops live within the C
| code; that applies at least as much to tree-like data as
| array-like data.)
|
| > I wish Python was at least as fast as Node (which also
| can have its own share of performance cliffs) It's
| possible that nowadays Python has JITs that improve
| performance to Java levels while keeping compatibility
| with most existing code - I haven't used Python
| professionally in quite a few years.
|
| No guarantees, but have you tried PyPy? It's existed
| since 2007 and definitely improved over time.
|
| I would say that "performance cliffs" are just endemic to
| programming. Even in C you find people writing bad
| algorithms because better ones seem (at least
| superficially) much harder to write -- especially if the
| good algorithm requires, say, a hash table. (C++ standard
| library containers definitely ameliorate this effect, but
| you pay in code complexity, especially where templates
| are needed.) And on the other hand you sometimes see big
| improvements from dropping to assembly (cf. ffmpeg).
| bob1029 wrote:
| https://devblogs.microsoft.com/dotnet/the-convenience-of-
| sys...
|
| https://learn.microsoft.com/en-
| us/dotnet/standard/serializat...
| zjaffee wrote:
| It's not about how much data you have, but also the sorts of
| things you are running on your data. Joins and group by's scale
| much faster than any aggregation. Additionally, you have a
| unified platform where large teams can share code in a
| structured way for all data processing jobs. It's similar in
| how companies use k8s as a way to manage the human side of
| software development in that sense.
|
| I can however say that when I had a job at a major cloud
| provider optimizing spark core for our customers, one of the
| key areas where we saw rapid improvement was simply through
| fewer machines with vertically scaled hardware almost always
| outperformed any sort of distributed system (abet not always
| from a price performance perspective).
|
| The real value often comes from the ability to do retries, and
| leverage left over underutilized hardware (i.e. spot instances,
| or in your own data center at times when scale is lower),
| handle hardware failures, ect, all with the ability for the
| full above suite of tools to work.
| dapperdrake wrote:
| Other way around. Aggregation is usually faster than a join.
| sgarland wrote:
| Disagree, though in practice it depends on the query,
| cardinality of the various columns across table, indices,
| and RDBMS implementation (so, everything).
|
| A simple equijoin with high cardinality and indexed columns
| will usually be extremely fast. The same join in a 1:M
| _might_ be fast, or it might result in a massive fanout. In
| the case of the latter, if your RDBMS uses a clustering
| index, and if you've designed your schemata to exploit this
| fact (e.g. a table called UserPurchase that has a PK of
| (user_id, purchase_id)) can still be quite fast.
|
| Aggregations often imply large amounts of data being
| retrieved, though this is not necessarily true.
| dapperdrake wrote:
| That level of database optimization is rare in practice.
| As soon as a non-database person gets decision making
| authority there goes your data model and disk layout.
|
| And many important datasets never make it into any kind
| of database like that. Very few people provide "index
| columns" in their CSV files. Or they use long variable
| length strings as their primary key.
|
| OP pertains to that kind of data. Some stuff in text
| files.
| jitl wrote:
| unconvinced. any join needs some kind of seek on the
| secondary relation index, or a bunch of state if ur
| stream joining to build temporary index sizes O(n) until
| end of batch. on the other hand summing N numbers needs
| O(1) memory and if your data is column shaped it's like
| one CPU instruction to process 8 rows. in "big data"
| context usually there's no traditional b-tree index to
| join either. For jobs that process every row in the input
| set Mr Join is horrible for perf to the point people end
| up with a dedicated join job/materialized view so
| downstream jobs don't have to re do the work
| rented_mule wrote:
| I like the peer comment's answer about a processing time
| threshold (e.g., a day). Another obvious threshold is data that
| doesn't conveniently fit on local disks. Large scale processing
| solutions can often process directly from/to object stores like
| S3. And if it's running inside the same provider (e.g., AWS in
| the case of S3), data can often be streamed much faster than
| with local SSDs. 10GB/s has been available for a decade or
| more, and I think 100GB/s is available these days.
| betaby wrote:
| > data can often be streamed much faster than with local
| SSDs. 10GB/s has been available for a decade or more, and I
| think 100GB/s is available these days.
|
| In practice most AWS instances are 10Gbps capped. I have seen
| ~5Gbps consistently read from GCS and S3. Nitro based images
| are in theory 100Gbps capable, in practice I've never seen
| that.
| sgarland wrote:
| Also, anything under 16 vCPUs generally has baseline /
| burst bandwidth, with the burst being best-effort, 5-60
| minutes.
|
| This has, at multiple companies for me, been the cause of
| surprise incidents, where people were unaware of this fact
| and were then surprised when the bandwidth suddenly
| plummeted by 50% or more after a sustained load.
| KolmogorovComp wrote:
| > Just how much data do you need when these sort of clustered
| approaches really start to make sense?
|
| I did not see your comment earlier, but to stay with Chess see
| https://news.ycombinator.com/item?id=46667287, with ~14Tb
| uncompressed.
|
| It's not humongous and it can certainly fit on disk(s), but not
| on a typical laptop.
| noufalibrahim wrote:
| I remember a panel once at a PyCon where we were discussing, I
| think, the anaconda distribution in the context of packaging
| and a respected data scientist (whose talks have always been
| hugely popular) made the point that he doesn't like Pandas
| because it's not excel. The latter was his go to tool for most
| of his exploratory work. If the data were too big, he'd sample
| it and things like that but his work finally was in Excel.
|
| Quick Python/bash to cleanup data is fine too I suppose and
| with LLMs, it's easier than ever to write the quick throwaway
| script.
| dapperdrake wrote:
| Whenever I had to use anaconda it was slow as molasses. Was
| that ever fixed?
| zahlman wrote:
| What _tasks_ were slow?
| greazy wrote:
| This has been fixed for ages. The dep solver was changed to
| Libmamba a few years ago.
| acomjean wrote:
| I took a bio statistic class. The tools were Excel/ R or
| Stata.
|
| I think most people used R. Free and great graphing. Though
| the interactivity of Excel is great for what ifs. I never got
| R till I took that class. Though RStudio makes R seem like
| scriptable excel.
|
| R/Python are fast enough for most things though a lot of
| genomic stuff (Blast alignments etc..) are in compiled
| languages.
| dapperdrake wrote:
| Adam Drake's example (OP) also streams from disk. And the unix
| pipeline is task-parallel.
| toast0 wrote:
| > Just how much data do you need when these sort of clustered
| approaches really start to make sense?
|
| You really need an enormous amount of data (or data processing)
| to justify a clustered setup. Single machines can scale up
| rather quite a lot.
|
| It'll cost money, but you can order a 24x128GB ram, 24x30TB ssd
| system which will arrive in a few days and give you 3 TB ram,
| 720 TB (fast) disk. You can go bigger, but it'll be a little
| exotic and the ordering process might take longer.
|
| If you need more storage/ram than around that, you need
| clustering. Or if the processing power you get in your single
| system storage isn't enough, you would need to cluster, but ~
| 256 cores of cpu is enough for a lot of things.
| greazy wrote:
| What motherboard supports this much ram?
| jtbaker wrote:
| you didn't need to read to rewrite to C# to do that - python
| should be able to handle streaming that amount/velocity of data
| fine, at least through a native extension like msgspec or
| pydantic. additionally, you made it much harder for other data
| engineers that need to maintain/extend the project in the
| future to do so.
| saberience wrote:
| The C# is probably far more maintainable and less error prone
| than Python. At least in my experience that's almost always
| the case.
|
| The amount of Python jobs I've had which run fine for several
| hours and then break with runtime errors, whereas with C# you
| can be reliably sure that if it starts running it will finish
| running.
| jtbaker wrote:
| Not a language problem, it's a dev culture problem. You can
| hold your devs accountable to the quality of their code.
| Strong er typing support via static analysis as well as
| runtime validation with untrusted input/data has really
| helped python alot.
|
| I'm not necessarily the biggest fan of python, but writing
| a data engineering tool in a non-data engineering focused
| language seems like a bad decision. Now when the OP leaves
| the organization is in a much tougher position.
| groundzeros2015 wrote:
| Bash is built around streaming though. You have to know how to
| use the tools to get the gains.
| MarginalGainz wrote:
| The saddest part about this article being from 2014 is that the
| situation has arguably gotten worse.
|
| We now have even more layers of abstraction (Airflow, dbt,
| Snowflake) applied to datasets that often fit entirely in RAM.
|
| I've seen startups burning $5k/mo on distributed compute clusters
| to process <10GB of daily logs, purely because setting up a
| 'Modern Data Stack' is what gets you promoted, while writing a
| robust bash script is seen as 'unscalable' or 'hacky'. The
| incentives are misaligned with efficiency.
| petcat wrote:
| > a robust bash script
|
| These hardly exist in practice.
|
| But I get what you mean.
| sam_lowry_ wrote:
| Yoy don't. It's bash only because the parent process is bash,
| but otherwise it's all grep, sort, tr, cut and othe textutils
| piped together.
| mjevans wrote:
| awk can do some heavy lifting too if the environment is too
| locked down to import a kitchen sink of python modules.
| RobinL wrote:
| Worse in some ways, better in others. DuckDB is often an
| excellent tool for this kind of task. Since it can run
| parallelized reads I imagine it's often faster than command
| line tool, and with easier to understand syntax
| mrgoldenbrown wrote:
| IMHO the main point of the article is that typical unix
| command pipeline pipeline IS parallelized already.
|
| The bottleneck in the example was maxing out disk IO, which I
| don't think duckdb can help with.
| chuckadams wrote:
| Pipes are parallelized when you have unidirectional data
| flow between stages. They really kind of suck for fan-out
| and joining though. I do love a good long pipeline of do-
| one-thing-well utilities, but that design still has major
| limits. To me, the main advantage of pipelines is not so
| much the parallelism, but being streams that process
| "lazily".
|
| On the other hand, unix sockets combined with socat can
| perform some real wizardry, but I never quite got the hang
| of that style.
| Linux-Fan wrote:
| UNIX provides the Makefile as go-to tool if a simple
| pipeline is not enough. GNUmake makes this even more
| powerful by being able to generate rules on-the-fly.
|
| If the tool of interest works with files (like the UNIX
| tools do) it fits very well.
|
| If the tool doesn't work with single files I have had
| some success in using Makefiles for generic processing
| tasks by creating a marker file that a given task was
| complete as part of the target.
| mdavidn wrote:
| Pipelines are indeed one flow, and that works most of the
| time, but shell scripts make parallel tasks easy too. The
| shell provides tools to spawn subshells in the background
| and wait for their completion. Then there are utilities
| like xargs -P and make -j.
| briHass wrote:
| More importantly, you have your data in a structured format
| that can be easily inspected at any stage of the pipeline
| using a familiar tool: SQL.
|
| I've been using this pattern (scripts or code that execute
| commands against DuckDB) to process data more recently, and
| the ability to do deep investigations on the data as you're
| designing the pipeline (or when things go wrong) is very
| useful. Doing it with a code-based solution (read data into
| objects in memory) is much more challenging to view the data.
| Debugging tools to inspect the objects on the heap is painful
| compared to being able to JOIN/WHERE/GROUP BY your data.
| groundzeros2015 wrote:
| Yep. It's literally what SQL was designed for, your
| business website can running it... the you write a shell
| script to also pull some data on a cron. It's beautiful
| reval wrote:
| I've seen this pattern play out before. The pushback on simpler
| alternatives seems from a legitimate need for short time to
| market from the demand some of the equation and a lack of
| knowledge on the supply side. Every time I hear an engineer
| call something hacky, they are at the edge of their abilities.
| networkadmin wrote:
| > Every time I hear an engineer call something hacky, they
| are at the edge of their abilities.
|
| It's just like the systemd people talking about sysvinit.
| "Eww, shell scripts! What a terrible hack!" says the guy with
| no clue and no skills.
|
| It's like the whole ship is being steered by noobs.
| acdha wrote:
| systemd would be a derail even if you weren't
| misrepresenting the situation at several levels.
| Experienced sysadmins in my experience were the ones
| pushing adoption because they had to clean up the messes
| caused by SysV's design limitations and flaws, whereas in
| this case it's a different scenario where the extra
| functionality is both unneeded and making it worse at the
| core task.
| networkadmin wrote:
| > Experienced sysadmins in my experience were the ones
| pushing adoption because they had to clean up the messes
| caused by SysV's design limitations and flaws
|
| That's funny. I used to have to clean up the messes
| caused by _systemd 's_ design limitations and flaws,
| until I built my own distro with a sane init system
| installed.
|
| Many of the noobs groaning about the _indignity_ of shell
| scripts don 't even realize that they could write init
| 'scripts' in whatever language they want, including
| Python (the language these types usually love so much, if
| they do any programming at all.)
| chuckadams wrote:
| It's entirely possible that both SysV init and systemd
| suck for different reasons. I'm still partial to systemd
| since it takes care of daemons and supervision in a way
| that init does not, but I'll take s6 or process-compose
| or even supervisord if I have to. Horses for courses.
| plagiarist wrote:
| I want to love s6 but every time I see the existence of
| s6-rc-compile I get heated. I'm sure there are excellent
| reasons behind it but I personally don't want services to
| work that way.
| plagiarist wrote:
| You can write arbitrary scripts into systemd... or like
| one step removed at most? That's not really a difference
| unless you have some nuance in mind that I don't.
|
| I honestly do not like systemd, either. It is okay for
| managing processes but I wish it didn't spread into
| everything else in the machine.
|
| Or if it must, could it actually work cohesively across
| their concepts? Would be nice to have an obvious and easy
| way to run Quadlet as its own user to isolate further,
| would be nice to have systemd-sysusers present in
| /etc/subuid so they can run containers.
|
| I like what they are doing with atomic distros. It would
| be great to have a single file declarative setup for
| something like running a containerized reverse HTTP proxy
| with an isolated user. Instead of "atomic" but you
| manually edit files in /etc after install.
| acdha wrote:
| I think you'd have a more fruitful discussion if you
| stopped trying to call people noobs when they don't agree
| with you.
|
| For example, I've been dealing with SysV since the early
| 90s and while it's gotten better since we no longer have
| to support the really bizarre Unix variants, my problem
| with init scripts wasn't "indignity" but the lack of
| consistency across distributions and versions, which
| affects anyone shipping software professionally ("can't
| do this easily until $distro upgrades coreutils"), and
| from an operator's perspective using Python doesn't make
| that better because instead of supporting one consistent
| thing you'd end up with the subset of features each
| application team felt like implementing, consistent only
| to the extent that they care to follow other projects.
| One virtue of systemd is that having a single common way
| to specify dependencies, restarts, customization, etc.
| avoids the ops people having to learn dozens of different
| variations of the same ideas and especially how to deal
| with their gaps. A few years back, a data center power
| outage at one place I worked really highlighted that: the
| systemd-based servers recovered quickly because they
| actually had working retries; all of the older stuff
| using SysV had to be manually reviewed because there were
| all kinds of problems like races on dependencies like DNS
| or NFS, retry logic which failed hard after a short
| period of time, failures because a stale PID file wasn't
| removed, or cases where a vendor had simply never
| implemented retries in their init scripts. While in
| theory you can handle all of those in SysV most people
| never did.
|
| After a couple decades of that, a lot of us don't want to
| spend time on problems Microsoft solved in Bill Clinton's
| first term.
| networkadmin wrote:
| I just created my own OS, with my own init system that
| does things how I think it should be done--and it does it
| every time, without the bizarre bugs that come from Linux
| Puttering's shitware code.
|
| It's the same thing any corporation should be doing if
| they were smart, instead of outsourcing everything to
| RedHat, Microsoft, Google, etc.
| array_key_first wrote:
| The reality is unit files are more portable than init
| scripts, regardless of what anyone says.
|
| Systemd unified and simplified administration across a
| lot of distributions. Before, it was a hodge podge, and
| there was a lot of knowledge lost going from rhel to
| Debian.
| whatwhaaaaat wrote:
| I hate to blather on about systemd in this decade but how
| in the world does creating something completely different
| than sysv init help people shipping software? Now they
| have to support yet another init scheme.
| acdha wrote:
| Prior to all of the important distributions consolidating
| on systemd, you had to support each distribution's
| convention for customization, overrides, dependencies,
| conventions for things like changing users or locations
| for PID files, not to mention the differences in various
| shell tools.
|
| Nothing insurmountable but it meant init files were
| inevitably much longer than the corresponding Upstart or
| systemd files despite doing less, and anytime we shipped
| a new version you had more testing since you had to
| implement a lot of functionality which is built in to
| other things.
| bitwize wrote:
| Specifying system processes and their dependencies
| declaratively, rather than in a tangle of arbitrary
| executable code, is cleaner, more efficient, easier to
| use, and more auditable. And that's not even getting into
| the additional process management duties systemd assumes.
| dapperdrake wrote:
| Eternal September
| networkadmin wrote:
| +1, Insightful
| dapperdrake wrote:
| Best reply my LLM had. Sorry.
| hdgvhicv wrote:
| +5 Troll
| pocketarc wrote:
| I agree - and it's not just what gets you promoted, but also
| what gets you hired, and what people look for in general.
|
| You're looking for your first DevOps person, so you want
| someone who has experience doing DevOps. They'll tell you about
| all the fancy frameworks and tooling they've used to do Serious
| Business(tm), and you'll be impressed and hire them. They'll
| then proceed to do exactly that for your company, and you'll
| feel good because you feel it sets you up for the future.
|
| Nobody's against it. So you end up in that situation, which
| even a basic home desktop would be more than capable of
| handling.
| jrjeksjd8d wrote:
| I have been the first (and only) DevOps person at a couple
| startups. I'm usually pretty guilty of NIH and wanting to
| develop in-house tooling to improve productivity. But more
| and more in my career I try to make boring choices.
|
| Cost is usually not a huge problem beyond seed stage. Series
| A-B the biggest problem is growing the customer base so the
| fixed infra costs become a rounding error. We've built the
| product and we're usually focused on customer enablement and
| technical wins - proving that the product works 100% of the
| time to large enterprises so we can close deals. We can't
| afford weird flakiness in the middle of a POC.
|
| Another factor I rarely see discussed is bus factor. I've
| been in the industry for over a decade, and I like to be able
| to go on vacation. It's nice to hand off the pager sometimes.
| Using established technologies makes it possible to delegate
| responsibility to the rest of the team, instead of me owning
| a little rats nest fiefdom of my own design.
|
| The fact is that if 5k/month infra cost for a core part of
| the service sinks your VC backed startup, you've got bigger
| problems. Investors gave you a big pile of money to go and
| get customers _now_. An extra month of runway isn't going to
| save you.
| woooooo wrote:
| The issue is when all the spending gets you is more
| complexity, maintenance, and you don't even get a
| performance benefit.
|
| I once interviewed with a company that did some machine
| learning stuff, this was a while back when that typically
| meant "1 layer of weights from a regression we run
| overnight every night". The company asked how I had solved
| the complex problem of getting the weights to inference
| servers. I said we had a 30 line shell script that ssh'd
| them over and then mv'd them into place. Meanwhile the
| application reopened the file every so often. Zero problems
| with it ever. They thought I was a caveman.
| ffsm8 wrote:
| The issue with solutions like that is usually that people
| don't know how it works and how to find it if it ever
| stops working...
|
| Basically discoverability is where shell script fail
| chuckadams wrote:
| Those scripts have logs, right? Log a hostname and path
| when they run. If no one thinks to look at logs, then
| there's a bigger problem going on than a one-off script.
| woooooo wrote:
| That becomes a problem if you let the shell script mutate
| into an "everything" script that's solving tons of
| business problems. Or if you're reinventing kubernetes
| with shell scripts. There's still a place for simple
| solutions to simple problems.
| justsomehnguy wrote:
| > Basically discoverability is where shell script fail
|
| No, it's lack of documentation and no amount of $$$$/m
| enterprise AI solutions (R)(TM) would help you if there
| is no documentation.
| LevGoldstein wrote:
| Which is why you take the time to put usage docs in the
| repo README, make sure the script is packaged and
| deployed via the same methods that the rest of the
| company uses, and ensure that it logs success/failure
| conditions. That's been pretty standard at every
| organization I've been at my entire professional career.
| Anyone who can't manage that is going to create worse
| problems when designing/building/maintaining a more
| complex system.
| mlyle wrote:
| Yah. A lot of the complexity in data movement or
| processing is unneeded. But decent standardized
| orchestration, documentation, and change management isn't
| optional even for the 20 line shell script. Thankfully,
| that stuff is a lot easier for the 20 line standard shell
| script.
|
| Or python. The python3 standard library is pretty
| capable, and it's ubiquitous. You can do a lot in 50-100
| lines (counting documentation) with no dependencies. In
| turn it's easy to plug into the other stuff.
| roncesvalles wrote:
| You can literally have a 20 line Python script on cron
| that verifies if everything ran properly and fires off a
| PagerDuty if it didn't. And it looks like PagerDuty even
| supports heartbeat so that means even if your Python
| script failed, you could get alerted.
| cheema33 wrote:
| I work for a small company with a handful of devs. We
| don't have a dedicated devops person, so I do it all.
| Everything is self-hosted. Been that way for years. But,
| yeah, if I go on vacation and something foes screwy, the
| business is hosed. However, even if it were hosted on AWS
| or elsewhere, it would not be any better. If anything, it
| may be worse. Instead of a person being well versed in
| standards based tech, they'd have to be an AWS expert.
| Why would we want that?
|
| I have recently started using terraform/tofu and ansible
| to automate nearly all of the devops operations. We are
| at a point where Claude Code can use these tools and our
| existing configs to make configuration changes, debug
| issues by reviewing logs etc. It is much faster at
| debugging an issue than I am and I know our stuff inside
| and out.
|
| I am beginning to think that AI will soon force people to
| rethink their cloud hosting strategy.
| SJC_Hacker wrote:
| In my experience, that $5k/month easily blows up into
| $100k/month
| pragma_x wrote:
| I've seen the ramifications of this "CV first" kind of
| engineering. Let's just say that it's a bad time when you're
| saddled with tech debt solely from a handful of influential
| people that really just wanted to work elsewhere.
| bandrami wrote:
| I'm largely a stranger to the js world but from the outside
| it sure looks like projects are sharded so as to maximize
| npm contribution count
| atomicnumber3 wrote:
| I'm convinced k8s is a conspiracy by bigtech to suppress
| startups.
| lormayna wrote:
| For a dasaset that live in RAM, the best solution are DuckDB or
| clickhouse-local. Using SQLish data is easier than a bunch of
| bash script and really powerful.
| zX41ZdbW wrote:
| Though ClickHouse is not limited to a single machine or local
| data processing. It's a full-featured distributed database.
| nicoburns wrote:
| > datasets that often fit entirely in RAM.
|
| Yep, and a lot more datasets fit entirely into RAM now.
| Ignoring the recent price spikes for a moment, 128GB of RAM in
| a laptop is entirely achievable and not even the limit of what
| is possible. That was a pipe dream in 2014 when computers with
| only 4GB were still common. And of course for servers the max
| RAM is much higher, and in a lot of scenarios streaming data
| off a fast local SSD may be almost as good.
| dapperdrake wrote:
| Oldie-but-goldy:
|
| https://yourdatafitsinram.net/
| newyankee wrote:
| I have actually worked in a company as a consultant data guy
| in a non technical team, I had a 128 GB PC 10 years back, and
| did everything with open source R then, and it worked ! The
| others thought it was wizardry
| plagiarist wrote:
| You don't really need to ignore the price spikes even. You
| can still buy more than 128Gb RAM on a machine with the $5k
| from one of the months.
| wccrawford wrote:
| I've spent my last 2 decades doing what's right, using the
| technologies that make sense instead of the techs that are cool
| on my resume.
|
| And then I got laid off. Now, I've got very few modern
| frameworks on my resume and I've been jobless for over a year.
|
| I'm feeling a right fool now.
| fHr wrote:
| This exactly, actual doers are most of the time not rewarded
| meanwhile the AWS senior sucking Jeffs wiener specialist gets
| a job doing nothing but generating costs and leave behind
| more shit after his 3 years moving the ladder up to some even
| bigger bs pretend consulting job at an even bigger company.
| It's the same bs mostly for developers. I rewrite their
| library from TS to Rust and it gains them 50x performance
| increases and saves them 5k+ a week over all their compute
| now but nobody gives a shit and I do not have a certification
| for that to show off on my LinkedIn. Meanwhile my PM did
| nothing got paid to do some shity certificate and then gets
| the credit and the certificate and pisses of to the next
| bigger fish collecting another 100k more meanwhile I get a 1k
| bonus and a pat on the shoulder. Corporate late stage
| capitalism is complete fucking bs and I think about becoming
| a PM as well now. I feel like a fool and betrayed. Meanwhile
| they constantly threaten our Team to lay it off or outsource
| it as they say we are to expensive in a first world country
| and they easily find as good people in India etc. What a time
| to be alive.
| antonvs wrote:
| > saves them 5k+ a week over all their compute
|
| If you're willing and able to promote yourself internally,
| you can make people give a shit, or at least publicly claim
| they do. That's 260k+ per year, and even big businesses are
| going to care about that at some level, especially if it's
| something that can be replicated. Find 10 systems you can
| do that with, and it's 2.6m+ per year.
|
| But, if you don't want to play the self-promotion game,
| yeah someone else is going to benefit from your work.
| hackthemack wrote:
| I have hung on to my job for many years now because of being
| in a similar situation in regards to trying to do the right
| thing and the fear of not being hire-able.
|
| There is something wrong with the industry in chasing fads
| and group think. It has always been this way. Businesses
| chased Java in the late 90s, early 00s. They chased CORBA,
| WSDL, ESB, ERP and a host of other acronyms back in the day.
|
| More recently, Data Lake, Big Data, Cloud Compute, AI.
|
| Most of the executives I have met really have no clue. They
| just go with what is being promoted in the space because it
| offers a safety net. Look, we are "not behind the curve!". We
| are innovating along with the rest of the industry.
|
| Interviews do not really test much for ability to think and
| reason. If you ran an entire ISP, if you figured out, on your
| own, without any help, how to shard databases, put in
| multiple layers of redundancy, caching... well, nobody cares
| now. You had to do it in AWS or Azure or whatever stack they
| have currently.
|
| Sadly, I do not think it will ever be fixed. It is something
| intrinsic to human nature.
| ahartmetz wrote:
| Try Rust? The system programming world isn't very bullshit-
| infested and Rust is trendy (which is good for a change),
| also employers can't realistically expect many years of Rust
| experience.
|
| Need training and something to show? Contribute to some FOSS
| project.
| willtemperley wrote:
| Yep. The cloud providers however always get paid, and get paid
| twice on Sunday when the dev-admins forget to turn stuff off.
|
| It's the same story as always, just it used to be Oracle
| certified tech, now it's the AWS tech certified to ensure you
| pay Amazon.
| 1vuio0pswjnm7 wrote:
| "I've seen startups burning $5k/mo on distributed compute
| clusters to process <10GB of daily logs, purely because setting
| up a 'Modern Data Stack' is what gets you promoted, while
| writing a robust bash script is seen as 'unscalable' or
| 'hacky'."
|
| Also seen strange responses from HN commenters when it's
| mentioned that bash is large and slow compared to ash and bash
| is better suited for use as an interactive shell whereas ash is
| better suited for use as a non-interactive shell, i.e., a
| scripting shell
|
| I also use ash (with tabcomplete) as an interactive shell for
| several reasons
| shiandow wrote:
| If airflow is a layer of abstraction something is wrong.
|
| Yes it is an additional layer, but if your orchestration starts
| concerning itself with _what_ it is doing then something is
| wrong. It is not a layer on top of other logic, it is a
| _single_ layer where you define how to start your tasks, how to
| tell when something is wrong, and when to run them.
|
| If you don't insist on doing heavy compitations within the
| airflow worker it is dirt cheap. If it's something that can
| easily be done in bash or python you can do it within the
| worker as long as you're willing to throw a minimal amount of
| hardware at it.
| mritchie712 wrote:
| happy middle ground: https://www.definite.app/ (I'm the
| founder).
|
| datalake (DuckLake), pipelines (hubspot, stripe, postgres), and
| dashboards in a single app for $250/mo.
|
| marketing/finance get dashboards, everyone else gets SQL + AI
| access. one abstraction instead of five, for a fraction of your
| Snowflake bill.
| attractivechaos wrote:
| On the contrary, the key message from the blog post is _not_ to
| load the entire dataset to RAM unless necessary. The trick is
| to stream when the pattern works. This is how our field
| routinely works with files over 100GB.
| data-ottawa wrote:
| Airflow and dbt serve a real purpose.
|
| The issue is you can run sub tib jobs on a few small/standard
| instances with better tooling. Spark and Hadoop are for when
| you need multiple machines.
|
| Dbt and airflow let you represent your data as a DAG and
| operate on that, which is critical if you want to actually
| maintain and correct data issues and keep your data transforms
| timely.
|
| edit: a little surprised at multiple downvotes. My point is,
| you can run airflow and dbt on small instances, and you can do
| all your data processing on small instances with tools like
| duckdb or polars.
|
| But it is very useful to use a tool like dbt that allows you to
| re-build and manage your data in a clear way, or a tool like
| airflow which lets you specify dependencies for runs.
|
| After say 30 jobs or so, you'll find that being able to re-run
| all downstreams of a model starts to payoff.
| adammarples wrote:
| Agreed, airflow and dbt have literally nothing to do with the
| size of the data and can be useful, or overkill, at any size.
| Dbt just templates the query strings we use to query the data
| and airflow just schedules when we query the data and what we
| do next. The fact that you can fit the whole dataset in
| duckdb without issue is kind of separate to these tools, we
| still need to be organised about how and when we query it.
| hmokiguess wrote:
| This reminds me of this reddit comment from a long time ago:
| https://www.reddit.com/r/programming/comments/8cckg/comment/...
| rawgabbit wrote:
| Well. I try for a middle ground. I am currently ditching both
| airflow and dbt. In Snowflake, I use scheduled tasks that call
| stored procedures. The stored procedures do everything I need
| to do. I even call external APIs like Datadog's and Okta's and
| pull down the logs directly into snowflake. I do try to name my
| stored procedures with meaningful names. I also add generous
| comments including urls back to the original story.
| rawgabbit wrote:
| I forgot to mention in Snowflake, besides chron scheduled
| tasks, you can add dependent tasks that only run if the
| previous task succeeded. I have 40 tasks chained together
| that way. Each of my task calls a stored procedure. Within
| each procedure, I have Try Catch and a catch-all clause that
| Raiseerror.
| jesse__ wrote:
| I've done a handful of interviews recently where the 'scaling'
| problem involves something that comfortably fits on one
| machine. The funniest one was ingesting something like 1gb of
| json _per day_. I explained, from first principals, how it
| fits, and received feedback along the lines of "our engineers
| agreed with your technical assessment, but that's not the
| answer we wanted, so we're going to pass". I've had this
| experience a good handful of times.
|
| I think a lot of people don't realize machines come with TBs of
| RAM and hundreds of physical cores. One machine is fucking huge
| these days.
| coliveira wrote:
| Yes, but then how are these people going to justify the money
| they're spending on cloud systems?... They need to find only
| reasons to maintain their "investment", otherwise they could
| be held as incompetent when their solution is proven to be
| ineffective. So, they have to show that it was a unanimous
| technical decision to do whatever they wanted in the first
| place.
| kevmo314 wrote:
| The wildest part is they'll take those massive machines,
| shard them into tiny Kubernetes pods, and then engineer
| something that "scales horizontally" with the number of pods.
| jesse__ wrote:
| Yeah man, you're running on a multitasking OS. Just let the
| scheduler do the thing.
| mystraline wrote:
| Its all fun and games, until the control plane gets
| killed by the OOMkiller.
|
| Naturally, that detaches all your containers. And theres
| no seamless reattach for control plane restart.
| dgxyz wrote:
| Or your CNI implementation is made of rolled up turds and
| you lose a node or two from the cluster control plane
| every day.
|
| (Large EKS cluster)
| zacmps wrote:
| Until you need to schedule GPUs or other heterogenous
| compute...
| jesse__ wrote:
| Are you saying that running your application in a pile of
| containers somehow helps that problem ..? It's the same
| problem as CPU scheduling, we just don't have good
| schedulers yet.. Lots of people are working on it though
| dgxyz wrote:
| Yeah this. As I explain many times to people, processes
| are the only virtualisation you need if you aren't
| running a fucked up pile of shit.
|
| The problem we have is fucked up piles of shit not that
| we don't have kubernetes and don't have containers.
| jesse__ wrote:
| Hahhah, yuuuup.
|
| I can maybe make a case for running in containers if you
| need some specific security properties but .. mostly I
| think the proliferation of 'fucked up piles of shit' is
| the problem.
| jchw wrote:
| Containers _are_ just processes plus some namespacing,
| nothing really stops you from running very huge tasks on
| Kubernetes nodes. I think the argument _for_ containers
| and Kubernetes is pretty good owing to their operational
| advantages (OCI images for distributing software,
| distributed cron jobs in Kubernetes, observability tools
| like Falco, and so forth).
|
| So I totally understand why people preemptively choose
| Kubernetes before they are scaling to the point where
| having a distributed scheduler is strictly necessary.
| Hadoop, on the other hand, you're definitely paying a
| large upfront cost for scalability you very much might
| not need.
| Gud wrote:
| Disagree.
|
| Different processes can need different environments.
|
| I advocate for something lightweight like FreeBSD jails.
| ahartmetz wrote:
| I think my brain hurts
| andai wrote:
| I had to re-read this a few times. I am sad now.
| cyberpunk wrote:
| To be fair each of those pods can have dedicated, separate
| external storage volumes which may actually help and it's
| def easier than maintaining 200 iscsi or more whatever
| targets yourself
| yieldcrv wrote:
| "there's no wrong answer, we just want to see how you think"
| gaslighting in tech needs to be studied by the EEOC,
| Department of Labor, FTC, SEC, and Delaware Chancery Court to
| name a few
|
| let's see how _they_ think and turn this into a paid
| interview
| ahartmetz wrote:
| Every one of these cores is really fast, too!
| jesse__ wrote:
| yeah man, computers are completely bananacakes
| yndoendo wrote:
| I recently had to parse 500MB to 2GB daily log files into
| analytical information for sales. Quick and dirty, the
| application would of needed 64GB RAM and work laptop only has
| 48GB RAM. After taking time cleaning it up, it was using
| under 1GB of RAM and worked faster by only retaining records
| in RAM if need be between each day.
|
| It is not about what you are doing, it is always about how
| you do it.
|
| This was the same with doing OCR analysis of assembly and
| production manuals. Quick and dirty, it would of took over 24
| hours of processing time, after moving to semaphores with
| parallelization it took less than two hours to process all
| the information.
| bauerd wrote:
| In interviews just give them what they are looking for. Don't
| overthink it. Interviews have gotten so stupidly standardized
| as the industry at large copied the same Big Tech DSA/System
| Design/Behavioral process. And therefore interview processes
| have long been decoupled from the business reality most
| companies face. Just shard the database and don't forget the
| API Gateway
| jesse__ wrote:
| Meh .. I've played that game; it doesn't work out well for
| anyone involved.
|
| I optimize my answers for the companies I _want_ to work
| for, and get rejected by the ones I don 't. The hardest
| part of that strategy is coming to terms with the idea that
| I constantly get rejected by people that I think are mostly
| <derogatory_words_here>, but I've developed thick skin over
| the years.
|
| I'd much rather spend a year unemployed (and do a ton of
| painful interviews) and find a company who's values align
| with mine, than work for a year on a team I disagree with
| constantly and quit out of frustration.
| bauerd wrote:
| The company's values may align to yours, even though they
| reject you. It's because the interview process doesn't
| need to have anything to do with their real-world
| process. Their engineers probe you for the same "best
| practices" that they themselves were constantly probed
| for in their own interviews. Interviewing is its very own
| skill that doesn't necessarily translate into real-life
| performance.
| jesse__ wrote:
| I agree with your observation. My issue is (from
| experience) it's really hard to tell from the outside if
| a teams' values align with mine. Many teams talk the
| talk, but don't walk the walk, as the saying goes. It's
| just easier to not participate than it is to guess, and
| be wrong.
|
| I also believe that running a broken interview process
| actively selects for qualities you actually don't want,
| so it's much more likely that teams conducting those
| interviews aren't teams I want to work on.
|
| Edit: As credence for my claims, the best team I've ever
| worked on was a team I did 90%+ of the hiring for, and we
| didn't do any of the 'typical' interview bullshit most
| companies do.
|
| What we did instead was sit people down and have deep
| technical conversations about systems they'd worked on in
| the past. The candidate would explain, in as much detail
| as they could muster, a system they'd worked on in the
| past, down to the lowest level details. Usually, they
| would talk to us for at least 20-30 minutes, then, we
| (the interviewers) would pose questions, usually starting
| with the form 'if we changed X, what effect would it
| have'. Doing interviews in this style make a few things
| immediately obvious:
|
| 1. Did the candidate have a deep, systemic understanding
| of the system they worked on?
|
| 2. Does the candidate have a good mental model for
| evaluating change in the system?
|
| That's how I conduct interviews, and unsurprisingly, when
| I get interviewed like that, my success rate is 100%. I
| don't think I've ever done an interview like that which
| did not result in an offer.
|
| Anyways, there's some rambling and unsolicited opinions
| for you :)
| mystifyingpoi wrote:
| This. Most interviewers don't want to do interviews, they
| have more important job to do (at least, that's what they
| claim). So they learn questions and approaches from the
| same materials and guides that are used by candidates.
| Well, I'm guilty of doing exactly this a few times.
| tshaddox wrote:
| > In interviews just give them what they are looking for
|
| Unless, of course, you have multiple options and you don't
| want to work for a company that's looking for dumb stuff in
| interviews.
| jmye wrote:
| 100%. Interviews should be a two-way filter. I'm
| sympathetic to unemployed-and-just-need-something, but
| also: boy are there a lot of companies hiring data
| engineers.
| groundzeros2015 wrote:
| Meh. as an interviewer I would always make it clear if we
| wanted to switch to "let's pretend it doesn't fit on a
| machine now".
|
| Demonstrating competency is always good.
| dehrmann wrote:
| > but that's not the answer we wanted
|
| You could have learned this if you were better about
| collecting requirements. You can tell the interviewer "I'd do
| it like this for this size data, but I'd do it like this for
| 100x data. Which size should I design this for?" If they're
| looking for one direction and you ask which one, interviewers
| will tell you.
| jesse__ wrote:
| I've done that too and, in my experience, people that ask a
| scaling question that fits on a single machine don't have
| the capacity to have that nuanced conversation. I usually
| try to help the interviewer adjust the scale to something
| that actually requires many machines, but they usually
| don't get it.
|
| Said another way, how do you have a meaningful conversation
| about scaling with a person who thinks their application is
| huge, but in reality only requires a tiny fraction of a
| single machine? Sometimes, there's such a massive gulf
| between perception and reality that the only thing to do is
| chuckle and move on.
| badgersnake wrote:
| This kind of bad interview is rife. It's often more a case of
| guess what the interviewer thinks than come up with a good
| solution.
| colechristensen wrote:
| Yeah I had this problem at a couple of times in startup
| interviews where the interviewer asked a question I happened
| to have expertise in and then disagreed with my answer and
| clearly they didn't know all that much about it. It's ok,
| they did me a favor.
|
| It may or may not be related that the places that this
| happened were always very ethnically monotone with narrow age
| ranges (nothing against any particular ethnic group, they
| were all _different_ ethnic monotones)
| jesse__ wrote:
| Hah yeah, that's a funny one, being able to run circles
| around the interviewer.
| jitl wrote:
| 1gb of json u can do in one parse -\\_(tsu)_/- big batches
| are fast
| ytoawwhra92 wrote:
| > I explained, from first principals, how it fits, and
| received feedback along the lines of "our engineers agreed
| with your technical assessment, but that's not the answer we
| wanted, so we're going to pass". I've had this experience a
| good handful of times.
|
| Probably a better outcome than being hired onto a team where
| everyone know you're technically correct but they ignore your
| suggestions for some mysterious (to you) reason.
| czhu12 wrote:
| I think it's not so much engineers actually setting up a
| distributed compute, as it is dropping a credit card into a
| paid cloud service, which behind the scenes sets up a
| distributed compute cluster and bills you for the compute in an
| obfuscated way, then gives a 20% discount + SSO if you sign up
| for annual enterprise plan.
|
| This kind of practice is insidious because early on, they
| charge $20/month to get started on the first 100mb of log
| ingestion, and you can have it up and running in 30 seconds
| with a credit card. Who would turn that down?
|
| Revisit that set up 2 years later and it's turned into a 60k/y
| behemoth that no one can unwind
| jitl wrote:
| On the other hand, now we have duckdb for all the "small big
| data", and a slew of 10-100x faster than Java equivalent stuff
| in the data x rust ecosystem, like DataFusion, Feldera,
| ByteWax, RisingWave, Materialize etc
| groundzeros2015 wrote:
| The point of the article is those don't actually work that
| well.
|
| I guarantee those rust projects have spent more time playing
| with rust and library design than the domain problem they are
| trying to solve.
| jitl wrote:
| None of the systems I mentioned existed at the time the
| article was published. I think the author would love duckdb
| which is a very speedy CLI SQL thingy that reads and writes
| data in all sorts of formats. It fits in great with other
| Unix CLI stuff.
|
| Many of the projects I mentioned you could see as a
| response to OP and the 2015 "Scalability, but at what
| COST?" paper which benchmarked distributed systems to see
| how many cores they need to beat a single thread.
| (https://news.ycombinator.com/item?id=26925449)
| dgxyz wrote:
| Our lot burns a fortune on snowflake every month but no one is
| using it. Not enough data is being piped into it and the shitty
| old reports we have which just run some SQL work fine.
|
| It looked good on someone's resume and that was it. They are
| long gone.
| jmye wrote:
| > because setting up a 'Modern Data Stack' is what gets you
| promoted
|
| It's not just that, it's that you better know their _specific_
| tech stack to even get hired. It's a lot of dumb engineering
| leaders pretending that AWS, Azure and Snowflake are such
| wildly different ecosystems that not having _direct_ experience
| in theirs is disqualifying (for pure DE roles, not talking
| broader sysadmin).
|
| The entire data world is rife with people who don't have the
| faintest clue what they're doing, who really like buzzwords,
| and who have never thought about their problem space
| critically.
| awesome_dude wrote:
| ENG are building what MGMT has told them to build for, the
| scale they want, not the scale they have
| rented_mule wrote:
| A little bit of history related to the article for any who might
| be interested...
|
| mrjob, the tool mentioned in the article, has a local mode that
| does not use Hadoop, but just runs on the local computer. That
| mode is primarily for developing jobs you'll later run on a
| Hadoop cluster over more data. But, for smaller datasets, that
| local mode can be significantly faster than running on a cluster
| with Hadoop. That's especially true for transient AWS EMR
| clusters -- for smaller jobs, local mode often finishes before
| the cluster is up and ready to start working.
|
| Even so, I bet the author's approach is still significantly
| faster than mrjob's local mode for that dataset. What MapReduce
| brought was a constrained computation model that made it easy to
| scale way up. That has trade-offs that typically aren't worth it
| if you don't need that scale. Scaling up here refers to data that
| wouldn't easily fit on disks of the day -- the ability to
| seamlessly stream input/output data from/to S3 was powerful.
|
| I used mrjob a _lot_ in the early 2010s -- jobs that I worked on
| cumulatively processed many petabytes of data. What it enabled
| you to do, and how easy it was to do it, was pretty amazing when
| it was first released in 2010. But it hasn 't been very relevant
| for a while now.
| KolmogorovComp wrote:
| > The first thing to do is get a lot of game data. This proved
| more difficult than I thought it would be, but after some looking
| around online I found a git repository on GitHub from rozim that
| had plenty of games. I used this to compile a set of 3.46GB of
| data, which is about twice what Tom used in his test. The next
| step is to get all that data into our pipeline.
|
| It would be interesting to redo the benchmark but with a (much)
| larger database.
|
| Nowadays the biggest open-data for chess must comes from Lichess
| https://database.lichess.org, with ~7B games and 2.34 TB
| compressed, ~14TB uncompressed.
|
| Would Hadoop win here?
| woooooo wrote:
| If you get all the data on fast SSDs in a single chassis, you
| probably still beat EMR over S3. But then you have a whole
| dedicated server to manage your 14TB of chess games.
|
| The "EMR over S3" paradigm is based on the assumption that the
| data isn't read all that frequently, 1-10x a day typically, so
| you want your cheap S3 storage but once in a while you'll want
| to crank up the parallelism to run a big report over longer
| time periods.
| dapperdrake wrote:
| Probably not.
|
| The compressed data can fit onto a local SSD. Decompression can
| definitely be streamed.
| djsjajah wrote:
| Not only can it be streamed, but lz4 will probably make
| things quicker.
| _zoltan_ wrote:
| what would you calculate in the data?
|
| I could be tempted to do some fun on an NVL72 ;-)
| fmajid wrote:
| I've contributed to PrestoDB, but the availability of DuckDB and
| fast multi core machines with even faster SSDs makes the need for
| distribution all the more niche, or even cargo-culting Google or
| Meta.
| fifilura wrote:
| The benefit of prestodb is that it can be used without even
| starting one of these expensive instances in AWS Athena.
| EdwardCoffin wrote:
| This makes me think of Bane's rule, described in this comment
| here [1]:
|
| _Bane 's rule, you don't understand a distributed computing
| problem until you can get it to fit on a single machine first._
|
| [1] https://news.ycombinator.com/item?id=8902739
| forinti wrote:
| I think many devs learn the trade with Windows and don't get
| exposure to these tools.
|
| Plus, they require a bit of reading because they operate on a
| higher level of abstraction than loops and ifs. You get implicit
| loops, your fields get cut up automatically, and you can apply
| regexes simultaneously on all fields. So it's not obvious to the
| untrained eye.
|
| But you get a lot of power and flexibility on the cli, which
| enable you to rapidly put together an ad hoc solution which can
| get the job done or at least serve as a baseline before you reach
| for the big guns.
| benrutter wrote:
| This times a zillion! I think there's been a huge industry push
| to convince managers and more junior engineers that spark and
| distributed tools are _the correct way to do data engineering_.
|
| I think its a similar pattern to web dev influencers have
| convinced everyone to build huge hydrated-spa-framework-craziness
| where a static site would do.
|
| My advice to get out of this mess:
|
| - Managers, don't ask for specific solutions (spark, react). Ask
| for clever engineers to solve problems and optimise / track what
| you vare about (cost, performance etc). You hired them to know
| best, and they probably do.
|
| - Technical leads, if your manager is saying "what about
| hyperscale?" You don't have to say "our existing solution will
| scale forever". It's fine to say, "our pipelines handle dataset
| up to 20GB, we don't expect to see anything larger soon, and if
| we do we'll do x/y/z to meet that scale". Your manager probably
| just wants to know scaling isn't going to crash everything, not
| that you've optimised the hell out of everything for your excel
| spreadsheet processing pipeline.
| zug_zug wrote:
| Absolutely, when I worked at (semi-well-known unicorn) a half-
| dozen years ago on the data-engineering team the manager told
| me "Hey we want to use spark next quarter, that's a huge
| initiative."
|
| And I immediately asked, "in what capacity?" And the answer was
| don't-know/doesn't-matter, it's just important that we can say
| we're using it. I really wish I understood where that was
| coming from (his manager resume-building? somebody getting a
| kickback?)
| spauldo wrote:
| That's when you rewrite your codebase in the SPARK dialect of
| Ada and play innocent when your management questions you
| about it.
| coliveira wrote:
| They'll never say it's resume building or kickbacks, they'll
| invent some technically sounding and/or business reason to
| achieve the same result.
| thwarted wrote:
| The most interesting part is that you can say you're
| doing/using something entirely independent of if you actually
| are. Sure, that's a lie, but so is only using something so
| you can say you're using it (sure, they admitted to _you_
| that was the reason, but that won 't be the reason they put
| on LinkedIn).
| woeirua wrote:
| Here's the thing though, most companies work with small data.
| The distribution of data set size follows a power law which
| means that few engineers get to work with petabyte sized
| datasets. That said, the job market definitely incentivizes
| people to have that kind of experience on their resume if they
| want to keep progressing in salary. This incentivizes over
| engineering.
| groundzeros2015 wrote:
| yes, but engineers also suck at communicating costs and
| benefits (and understanding them).
| cryptoboy2283 wrote:
| Earlier in 2010 -
| http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...
| phyzix5761 wrote:
| It's easy to overlook how often straightforward approaches are
| the best fit when the data and problem are well understood. Large
| expensive tools can become problems in their own right creating
| complexity that then requires even more tooling to manage. (Maybe
| that's the intent?) The issue is that teams and companies often
| adopt optimization frameworks earlier than necessary. Starting
| with simpler tools can get you most of the way there and in many
| cases they turn out to be all that's needed.
| jonathanhefner wrote:
| And since AI agents are extremely good at using them, command-
| line tools are also probably 235x more effective for your data
| science needs.
| fifilura wrote:
| No joins in that article?
|
| The comments here smell of "real engineers use command line". But
| I am not sure they ever actually worked with analysing data more
| than using it as a log parser.
|
| Yes Hadoop is 2014.
|
| These days you obviously don't set up a Hadoop cluster. You use
| the cloud provider service provided (BigQuery or AWS Athena for
| example).
|
| Or map your data into DuckDB or use polars if it is small.
| ziml77 wrote:
| > But I am not sure they ever actually worked with analysing
| data more than using it as a log parser.
|
| It really feels that way. Real data analysis involves a lot
| more than just grepping logs. And the reason to be wary of
| starting out unprepared for that kind of analysis is that
| migrating to a better solution later is a nightmare.
| noo_u wrote:
| In many ways HN is Reddit in denial at this point :) Comments
| and upvotes that are based mostly on vibes, with depth and
| discussion usually happening somewhere towards the middle of
| the comment tree.
| dapperdrake wrote:
| Where else would you JOIN in?
| jeffbee wrote:
| Something to note here is that the result of xargs -P is unlikely
| to be satisfactory, since all of the subprocesses are simply
| connected to the terminal and stomp over each other's outputs. A
| better choice would be something like rush or, for the Perl fans,
| parallel.
| hmokiguess wrote:
| Tangential, but this reminds of the older K website when it used
| to be shakti.com that had an intro like this in their about
| section:
|
| 1K rows: use excel
|
| 1M rows: use pandas/polars
|
| 1B rows: use shakti
|
| 1T rows: only shakti
|
| Source:
| https://web.archive.org/web/20230331180931/https://shakti.co...
| meken wrote:
| I'm curious about the memory usage of the cat | grep part of the
| pipeline. I think the author is processing many small files?
|
| In which case it makes the analysis a bit less practical, since
| the main use case I have for fancy data processing tools is when
| I can't load a whole big file into memory.
| dapperdrake wrote:
| Memory footprint is tiny:
|
| Unix shell pipelines are task-parallel. Every tool gets spun up
| as its own unix process -- think "program" (fork-exec).
| Standard input and standard output (stdin, stdout) get hooked
| up to pipes. Pipes are like temporary files managed by the
| kernel (hand-wave). Pipe buffer size is a few KB. Grep does a
| blocking read on stdin. Cat writes to stdout. Both on a kernel
| I/O boundary. Here the kernel can context-switch the process
| when waiting for I/O.
|
| In the past there was time-slicing. Now with multiple cores and
| hardware threads they actually run concurrently.
|
| This is very similar to old-school approach to something like
| multiple threads, but processes don't share virtual address
| spaces in the CPU's memory management unit (MMU).
|
| Further details: look up McIlroy's pipeline design.
| srcreigh wrote:
| MapReduce is from a world with slow HDDs, expensive ram,
| expensive enterprise class servers, fast network.
|
| In that case to get best performance, you'd have to shard your
| data across a cluster and use mapreduce.
|
| Even in the authors 2014 SSDs multi-core consumer PC world, their
| aggregate pipeline would be around 2x faster if the work was
| split across two equivalent machines.
|
| The limit of how much faster distributed computing is comes down
| to latency more than throughput. I'd not be surprised if this
| aggregate query could run in 10ms on pre sharded data in a
| distributed cluster.
| dapperdrake wrote:
| Confusing the concept and the implementation.
| srcreigh wrote:
| Somebody has to go back to first principles. I wrote pig
| scripts in 2014 in Palo Alto. Yes, it was shit. IYKYK. But
| the author, and near everybody in this thread, are wrong to
| generalize.
|
| PCIe would have to be millions of times faster than Ethernet
| before command line tools are actually faster than
| distributed computing and I don't see that happening any time
| soon.
| jeswin wrote:
| The same thing is true with Sqlite vs Postgres. Most startups
| need Sqlite, not Postgres. Many queries run an order of magnitude
| faster. Not only is it better for your users, it's life changing
| to see the test suites (which would take minutes to run) complete
| in mere seconds
| shimman wrote:
| Feels like quibbling over the differences between two databases
| that are going to act the same for 90% of projects out there
| doesn't really matter.
|
| If you want speed, just have your database stored in the same
| place as your application, locally, rather than hopping across
| the world to retrieve data that can be located next to the
| code.
|
| That would probably be the easiest thing to do to get a real
| measured performance gains.
|
| As other commentators pointed out, computers are extremely
| powerful. This isn't 1995, you can easily host everything in
| the same local area and get a very responsive application with
| very minimal needs to worry about resource constraints.
| mbb70 wrote:
| The bigness of your data has always depended on the what you are
| doing with it.
|
| Consider the following table of medical surgeries:
| date,physician_name, surgery_name,success.
|
| "What are the top 10 most common surgeries?" - easy in bash
|
| "Who are the top physicians (% success) in the last year for
| those surgeries?" - still easy in bash
|
| "Which surgeries are most affected by physician experience?" -
| very hard in bash, requires calculating for every surgery how
| many times that physician had performed that surgery on that day,
| then compare low and high experience outcomes.
|
| A researcher might see a smooth continuum of increasingly complex
| questions, but there are huge jumps in computational complexity.
| At 50gb dataset might be 'bigger' than a 2tb one if you are
| asking tough questions.
|
| It's easier for a business to say "we use Spark for data
| processing", than "we build bespoke processing engines on a case
| by case basis".
| adamdrake wrote:
| Author here!
|
| It's great to see this post I wrote years ago still being useful
| for people.
|
| I agree with many here that the situation is arguably worse in
| many ways. However, along similar lines, I've been pleased to see
| a move away from cargo culting microservices (another topic I
| addressed in a separate post on that site).
|
| To all those helping companies and teams improve performance,
| keep it up! There is hope!
| dapperdrake wrote:
| Adam,
|
| Thank you very much!
|
| Been re-reading your post multiple times.
|
| You inspired me to port Waters-Series (kind-of streams) to
| JavaScript to get pipelining for stream processing.
| olq_plo wrote:
| And now you can do this with polars in parallel on all your cores
| and the GPU, using almost the same syntax as in pyspark.
| jgord wrote:
| highly recommend xsv by BurntSushi [ csv parser / wrangler
| written in rust ]
___________________________________________________________________
(page generated 2026-01-18 23:00 UTC)