[HN Gopher] Explorer - A library to bring dataframes to Elixir
___________________________________________________________________
Explorer - A library to bring dataframes to Elixir
Author : ahamez
Score : 119 points
Date : 2022-04-28 09:24 UTC (2 days ago)
(HTM) web link (www.cigrainger.com)
(TXT) w3m dump (www.cigrainger.com)
| newskfm wrote:
| [deleted]
| ahamez wrote:
| Added '(Elixir and Polars)' to the title as 'Introducing
| Explorer' alone is not clear enough in my opinion.
| rapnie wrote:
| > we've been using Elixir in production since 2018. We switched
| over to LiveView in early 2020 and haven't looked back.
|
| What does "switched to LiveView" mean? I thought LiveView was
| some collaborative, live documentation / interactive tutorial
| tool. Or does it provide a whole programming environment and/or
| development process?
| Existenceblinks wrote:
| I think you mean https://livebook.dev/ which is built on top of
| LiveView. LV can be used instead of "Dead View" which is a non
| diff-tracking view.
| rapnie wrote:
| Ah stupid. Thank you. I will try to get my brain in higher
| gear today.
| pbowyer wrote:
| > Second, there are callbacks! The second argument in
| DataFrame.filter/2 takes a callback function against the
| dataframe. There's some cool stuff we can do with that, and I
| think callbacks against the dataframe are a natural way to work.
|
| I'm not enamoured with how DataFrame.filter/2 reads compared to
| the dplyr example. I've written very little R but read code by
| others and find dpylr excellent for that; and I use a JS 'port'
| of dplyr [1] for projects. I'm in the "hate pandas" camp and find
| it painful to both read and write - and these callback functions
| are coming too close to Pandas for my liking. I hope an
| alternative can be found
|
| 1. https://pbeshai.github.io/tidy/
| throwamon wrote:
| For further inspiration, this is a pretty good-looking "dplyr
| for Python": https://github.com/machow/siuba
|
| There's precedent in Ecto for a "magic" operator (`^` if I'm
| not mistaken), so it wouldn't be a stretch to implement it here
| as well.
| jasonpbecker wrote:
| Flow/Broadway + how nice Ecto is already had me thinking, "Elixir
| is super interesting for data engineering." Explorer and Livebook
| have me thinking, "Elixir has the best shot at unseating Python."
| sergiomattei wrote:
| Doubt it. Elixir is a leaky abstraction over Erlang, so OTP
| concepts are a huge barrier of entry for non-software
| engineers.
|
| e.g. to this day starting a small library requires learning
| supervisors, genservers, etc. Unless you're doing a one off
| .exs, you're going to be really confused by what "mix new"
| does.
|
| Saying this as a passionate Elixirist -- there's nothing wrong
| with the leaky abstraction part, it's what makes it powerful --
| but dethroning Python isn't happening anytime soon.
| ravi-delia wrote:
| Maybe I've just forgotten what learning Elixir was like, but
| I feel like OTP never gave me all that much trouble? I guess
| it might be a little weird, but conceptually it's not that
| different from how objects work and presumably Python data
| people do fine with that. I feel like the sticking point
| might be the syntax. Elixir has a very developer-friendly
| syntax, but the flexibility (all the ways to call functions,
| implicit returns, arity stuff, collection syntax, even
| macros) makes it way harder to just learn "incantations" and
| be able to read other people's code.
| josevalim wrote:
| I am not sure I agree with the OTP bits.
|
| We often tend to compare having to learn OTP with "not having
| to learn OTP" but the reality is that we need to compare OTP
| with concurrency tools in other languages (e.g. threads,
| locks, mutexes).
|
| Of course, one can say: "I wouldn't use threads in other
| languages" but that's potentially a bigger problem, as you
| end up with software which can't easily exploit concurrency
| (which is especially important in the data domain).
|
| I still think what you said has some truth to it, but I think
| the frame of reference is a bit off. I would rather learn a
| bit of OTP to get concurrency and fault tolerance rather than
| learn nothing and get none of it. :)
|
| PS: "mix new" does not generate supervision trees by default.
| sergiomattei wrote:
| > compare OTP with concurrency tools in other languages
| (e.g. threads, locks, mutexes)
|
| That's a very fair point, but is the average data scientist
| going to be learning this or offloading that task to a
| software engineer?
|
| > I would rather learn a bit of OTP to get concurrency and
| fault tolerance rather than learn nothing and get none of
| it. :)
|
| Agreed --- I've been using Elixir for a bit and am only now
| learning about supervision trees and genservers. It's such
| an exciting part of the language.
|
| > PS: "mix new" does not generate supervision trees by
| default.
|
| Yup and that was confusing for me as a beginner -- I recall
| being confused over where the entrypoint of the app was.
| Coming from a language like Python.
|
| Hi Jose, huge fan. Thanks for dropping by this thread!
| 3jckd wrote:
| It's interesting what/when/if will unsettle Python. And what
| the adoption would be like.
|
| Julia has been <designed> to unsettle Python in the data space
| but to no avail.
|
| -\\_(tsu)_/-
| jasonpbecker wrote:
| Julia was designed to unsettle a creeping set of python in
| the domain of Perl, C++, and Matlab in scientific computing
| and various quant analysis that's pretty distant, IMO, from
| what industry data engineering, ML, and analytics work was
| doing.
| rch wrote:
| Factor in Rust integration via Rustler and it starts to look
| like a 'when not if' dynamic.
| tfp137 wrote:
| I thought it would be Clojure, but that was 10 years ago and
| Python is still (inexplicably?) dominant.
| peoplefromibiza wrote:
| Amazing!
|
| Rustler precompiled[1] is exactly what I was looking for to start
| writing NIFs in Rust.
|
| Having to bring in the entire Rust toolchain stopped me every
| time I thought about it - not mentioning that at my current day
| job I'm not completely at liberty of deciding what tools I
| can/cannot use.
|
| Combined with Livebook[2] this is the perfect combo to lure into
| the Elixir ecosystem my fellow data scientists colleagues.
|
| [1] https://dashbit.co/blog/rustler-precompiled
|
| [2] https://livebook.dev/
| pdimitar wrote:
| > _Having to bring in the entire Rust toolchain_
|
| I relate to the general sentiment, mind you, but Rust is
| brainlessly easy to install.
|
| But even if you seriously don't want to even do that (two shell
| commands) you can still pull a Docker image and work inside it
| -- but that's a bit more involved, granted.
|
| I applaud the Rustler Precompiled effort because apparently it
| lowers the barrier to entry for many -- yourself included --
| but at least for me installing Rust was never an actual
| problem.
|
| As for Explorer, I have a few friends that curse at Python and
| Pandas every week, I might be able to "sell" them this project
| instead!
| peoplefromibiza wrote:
| > I relate to the general sentiment, mind you, but Rust is
| brainlessly easy to install.
|
| True, my issue is with releasing libraries that need Rust.
|
| It forces the users of the library to install Rust, adding
| friction, and possibly deal with issues/errors that they are
| not used to.
|
| Especially on heterogeneous systems: library developed on
| Linux, user is on Mac/Windows.
|
| One example is my day job where the CI platform is controlled
| by another department and you can't create your own custom
| Dockerfile(s), because "security".
|
| On the developers' side Rust is completely fine, in fact IMO
| Python is much more problematic, despite being an order of
| magnitude more popular.
| pdimitar wrote:
| Oh yeah, agreed, distribution becomes little more involved.
| In these cases Rustler Precompiled is a complete bliss
| indeed. ^_^
| umren wrote:
| Your friends haven't tried R and Julia?
|
| I don't see how elixir can replace python any time soon
| because it's DS ecosystem is lacking rn sadly
| peoplefromibiza wrote:
| of course they do, R much more than Julia tbh, I think on
| the prototyping side of the venture Elixir experience is
| more polished, even though slower ( Nx IMO is promising in
| that space)
| pdimitar wrote:
| Oh, he definitely tried R but was not very impressed. Says
| it's only moderately better than Python's otherwise
| excellent bindings to a lot of libraries, and that he had
| no patience for Julia. He isn't exactly a programmer, he's
| just a very practical guy that is open to learning some
| programming to get his stuff done quicker and with less
| mistakes but that's about it. He has a short fuse for
| ecosystem deficiencies -- tbf I think we the programmers
| have to learn from such people. :D
|
| And yeah, Elixir's scene in this area is just now starting.
| The NX project opened a lot of possible doors but the
| community just started stepping up. It's going to be a
| while, granted, but I am optimistic because Elixir's
| overall community values quality and centralization (with
| all the sometimes problematic collaboration efforts that
| entails) over rushing to have 10 competing implementations.
|
| Time will tell. I have no horse in the race either way,
| just sharing various observations really.
| jasonpbecker wrote:
| As an R user who can't stand working in Python, where
| Elixir I think has the most potential is not the data
| science part but data engineering. The whole world of
| ELT/ETL pipeline and workflow tools written in Python is a
| mess IMO, and Broadway/Flow + Ecto + what's going on with
| Explorer etc I think can quickly replace a whole mess of
| python/scala/EMR type workflows in a language and
| environment that feels way better suited to the problem
| space, in my experience.
| pdimitar wrote:
| Yeah, can confirm, I did a ton of Elixir work in the ETL
| space and it's amazingly well-suited for it. If you don't
| get an OCD over the fact that it obviously can't work as
| fast as Rust (realistically it's likely anywhere from 30x
| to 100x slower even) then the ergonomic and intuitive
| syntax combined with the transparent parallelism create
| an all-around win.
|
| So far I've never regretted using Elixir for ETL, but I
| do confess that for two projects I was mighty tempted to
| try a rewrite in Rust because ingesting several dozens of
| billions of records in a dynamic language isn't exactly
| fast.
|
| But, again, the ergonomics trump everything else 99% of
| the time for me. Plus for one-off ingestion tasks nobody
| cares if it takes 30 minutes or the whole night.
___________________________________________________________________
(page generated 2022-04-30 23:01 UTC)