[HN Gopher] Farewell Pandas, and thanks for all the fish
___________________________________________________________________
Farewell Pandas, and thanks for all the fish
Author : nojito
Score : 187 points
Date : 2024-08-29 11:47 UTC (11 hours ago)
(HTM) web link (ibis-project.org)
(TXT) w3m dump (ibis-project.org)
| softwaredoug wrote:
| One thing I do like about pandas is it's pretty extensible to
| columns of new types. Maybe I'm missing something, but does
| Polars allow this? Last time I checked there wasn't a clear path
| forward.
| datadrivenangel wrote:
| Ibis allows this with custom datatypes.
| porker wrote:
| I'm curious, when do you want to add a new column type? It's
| not a situation I've been in.
| softwaredoug wrote:
| GeoPandas is one popular library
| (https://geopandas.org/en/stable/)
|
| I added a column type for full text search, searching
| tokenized text (https://github.com/softwaredoug/searcharray)
|
| The pandas support for extensions is very robust with
| extensive testing to make sure you can do all the data
| aggregations functionality you expect in and pandas column.
| deadlypointer wrote:
| OTOH the list and struct types allow lots of complex
| operations, e.g with `.eval`, which run blazingly fast.
| maxdo wrote:
| curious to know what they use instead of Dask ?
| datadrivenangel wrote:
| DuckDB. It's faster on a single computer than Dask.
| codyvoda wrote:
| Ibis still supports over a dozen backends, including DuckDB
| and DataFusion and Polars (all great single-node OLAP query
| engine options that are often faster than trying something
| distributed) and PySpark, Snowflake, BigQuery, Trino,
| ClickHouse, Druid, and more
|
| plenty of options for fast and scalable computation. in the
| fullness of time, it would be great to have a Dask backend
| but it'd need to be rewritten to use dask-expr, which isn't
| something the core Ibis team has the capacity for
| lopatin wrote:
| As a potential user of Ibis, how do I know they won't
| eventually drop support for other backends until it just
| becomes a frontend for DuckDB?
| codyvoda wrote:
| Great question, a bunch of reasons:
|
| 1. The value prop of Ibis is to avoid vendor lock-in and
| allow users to more easily swap out execution engines that
| meet their needs. So it's valuable to the project to have
| "competing" backends supported (e.g. DuckDB, DataFusion,
| and Polars for local, and Snowflake, ClickHouse, PySpark,
| Databricks, etc. for distributed) -- the idea is a standard
| like Apache Arrow that everyone adopts to stop this cycle
| of tightly coupling a dataframe interface to an execution
| engine 2. Ibis is primarily supported by Voltron Data (my
| employer) which is itself a vendor, so the real worry would
| be that Ibis drops all backends until it only support
| Voltron Data's engine (Theseus)... 3. ... but Voltron Data
| cannot do that for a number of reasons. First, Ibis is an
| independently governed open source project. Second, it
| would not make any business sense. You can read some more
| on that here: https://ibis-project.org/posts/why-voda-
| supports-ibis.
|
| The general idea is to add more backends, not remove them,
| but pandas is a special case where it introduces a lot of
| technical complexity with (arguably) negative benefit to
| potential users -- the DuckDB backend can do everything the
| pandas backend can do, but better
| 2-3-7-43-1807 wrote:
| even vanilla is faster than dask on a single computer ...
| _OFFICIALLY_ ... cause dask is and only is for clusters
| whimsicalism wrote:
| if you're at cluster scale, i would skip right over dask
| and go to spark
| 2-3-7-43-1807 wrote:
| or hadoop
| cpcloud wrote:
| I'll also suggest inscribing stone tablets with a chunked
| version of your computation and distributing the
| described chunked computation to your friends, and then
| gathering the tablets and finishing up the compute.
|
| It's scalable to ~billions of nodes.
| 2-3-7-43-1807 wrote:
| but that exists already and it's called databricks
| mistrial9 wrote:
| can you expand or give more context on this ?
| datadrivenangel wrote:
| Dask is designed for scaling to multiple compute nodes, and
| is often slower than vanilla pandas on a single machine
| because of the overhead.
|
| If your data is larger than memory though, at a certain
| point Dask is faster than pandas (which will often just
| fail), and DuckDB (Which is super fast for in-memory data,
| and still quite fast for out of memory data).
|
| Rule of thumb these days though is that if your data isn't
| in the terabytes per run, a single processing node is
| probably best.
| spywaregorilla wrote:
| What is this exactly?
|
| The front page doesn't make it clear to make what ibis is besides
| being an alternative to pandas/polars.
| prepend wrote:
| Check the project's left nav, https://ibis-project.org/why
|
| I didn't know either. It's a python dataframe library/api.
| smu3l wrote:
| Never heard of ibis before but the front page give a pretty
| good overview of what it is to me at least. Looks like a
| dataframes api that can be executed on a variety of backend
| engines, both distributed and local.
|
| Somewhat analogous to how the pandas api can be used in pyspark
| via pyspark pandas, but from the api -> implementation
| direction rather than (pandas) implementation -> api ->
| implementation maybe?
| giovannibonetti wrote:
| As far as I can tell, it is an ORM for data engineers. They can
| write Python code that gets translated to either SQL or some
| other language understood by the DB engine that actually runs
| it.
| joelschw wrote:
| Huge fan of Ibis, the value isn't that you can now use DuckDB...
| it's that your syntax will work when the next cool thing arrives
| too
| infecto wrote:
| I will try it out but honestly the pain-point for me is the
| library api to pandas is not always intuitive/natural to how
| things should be in Python. The NaN/None bit is annoying but I
| find that to be a minor annoyance.
| mananaysiempre wrote:
| > NULL indicates a missing value, and NaN is Not a Number.
|
| That's actually less true than it sounds. One of the primary
| functions of NaN is to be the result of 0/0, so there it means
| that there could be a value but we don't know what it is because
| we didn't take the limit properly. One of the primary functions
| of NULL is to say that a tuple satisfies a predicate except we
| don't know what this one position is--it's certainly is something
| out in the real world, we just don't know what. These ideas are
| what motivates the comparison shenanigans both NaN and NULL are
| known for.
|
| There's certainly an argument to be made that the actual
| implementation of both of these ideas is half-baked in the
| respective standards, and that they are half-baked _differently_
| so we shouldn't confuse them. But I don't think it's fair to say
| that they are just completely unrelated. If anything, it's
| Python's None that's doesn't belong.
| cpcloud wrote:
| 100% this.
|
| We'd love to have a docs contribution that lays this out in
| detail if you'd be up for it!
|
| Disclaimer: (I lead the project and work on it full time).
| aeonik wrote:
| Why don't we build a portable numeric system that just has
| these mathematical constants and definitions built in, in a
| portable and performance manner?
| verandaguy wrote:
| We just don't have the technology for that, else there'd be
| libraries floating around that let you do that.
| miohtama wrote:
| Related to this, below someone posted a link to the blog post
| of Wes McKinney where he discussed Pandas limitations and how
| PyArrow works around these
|
| > 4. Doing missing data right > All missing data in Arrow is
| represented as a packed bit array, separate from the rest of
| the data. This makes missing data handling simple and
| consistent across all data types. You can also do analytics on
| the null bits (AND-ing bitmaps, or counting set bits) using
| fast bit-wise built-in hardware operators and SIMD.
|
| > The null count in an array is also explicitly stored in its
| metadata, so if data does not have nulls, we can choose faster
| code paths that skip null checking. With pandas, we cannot
| assume that arrays do not have null sentinel values and so most
| analytics has extra null checking which hurts performance. If
| you have no nulls, you don't even need to allocate the bit
| array.
|
| > Because missing data is not natively supported in NumPy, over
| time we have had to implement our own null-friendly versions of
| most key performance-critical algorithms. It would be better to
| have null-handling built into all algorithms and memory
| management from the ground up.
|
| https://wesmckinney.com/blog/apache-arrow-pandas-internals/
| jononor wrote:
| Note that the post is from 2017, and pandas now has
| (optional) support for PyArrow backed dataframes. So there is
| movement away from the critiques that were presented there.
| miohtama wrote:
| Yes I think that early post discussed about upcoming Pandas
| 2.0 which now realised 7 years later.
| OptionOfT wrote:
| > One of the primary functions of NULL is to say that a tuple
| satisfies a predicate except we don't know what this one
| position is--it's certainly is something out in the real world,
| we just don't know what.
|
| I'm not sure I understand this. Can you explain it with an
| example?
|
| I'm thinking something like: let (x, y) =
| NULL;
|
| But then I'm stuck.
| kstrauser wrote:
| In the broader DB theory context and not strictly Pandas,
| think of a `users` table with a `middle_name` column. There's
| a difference between `middle_name is null` (we don't know
| whether they have a middle name or what it is if they do) and
| `middle_name = ''` (we know that they don't have a middle
| name).
|
| In that case, `select * from users where middle_name is null`
| gives us a set of users to prod for missing information next
| time we talk to them. `...where middle_name = ''` gives us a
| list of people without a middle name, should we care.
|
| Edit:
|
| A related aside is that Django's ORM's handling of NULL gives
| me freaking hives. From https://docs.djangoproject.com/en/5.1
| /ref/models/fields/#nul...:
|
| > If a string-based field has null=True, that means it has
| two possible values for "no data": NULL, and the empty
| string. In most cases, it's redundant to have two possible
| values for "no data;" the Django convention is to use the
| empty string, not NULL.
|
| I've been around numerous Django devs when they found out
| that other DBs and ORMs do not remotely consider NULL and
| empty string to be the same thing.
| emmelaich wrote:
| In Oracle, "" is converted to NULL so maybe that's where
| they get that from.
| kstrauser wrote:
| Whoa, you're right.[0] I wouldn't have expected that at
| all.
|
| [0] https://docs.oracle.com/en/database/oracle/oracle-
| database/1...
| rbanffy wrote:
| You can almost count on Oracle doing the wrong thing...
|
| It never ceases to amaze me they got themselves into a
| position where they had to introduce a VARCHAR2 type.
| mananaysiempre wrote:
| What I mean is, the ivory-tower relational model is that each
| table ("relation") represents a predicate (Boolean-valued
| function) accepting as many arguments as there are columns
| ("attributes"), and the rows ("tuples") of the table are an
| exhaustive listing of those combinations of arguments for
| which the predicate yields true ("holds"). E.g. the relation
| Lived may contain a tuple (Edgar Codd, 1923-08-19,
| 2003-04-18) to represent the fact that Franklin was born on
| 19 August 1923 and died on 18 April 2003.
|
| One well-established approach[1] to NULLs is that they
| represent the above "closed-world assumption" of
| exhaustiveness to encompass values we don't know. For
| example, the same relation could also contain the tuple
| (Leslie Lamport, 1941-02-07, NULL) to represent that Lamport
| was born on 7 February 1941 and lives to the present day.
|
| We could then try to have some sort of three-valued logic to
| propagate this notion of uncertainty: define NULL = x to be
| (neither true nor false but) NULL ("don't know"), and then
| any Boolean operation involving NULL to also yield NULL;
| that's more or less what SQL does. As far as I know, it's
| possible to make this consistent, but it'll always be weaker
| than necessary: a complete solution would instead assign a
| variable to each unknown value of this sort and recognize
| that those values are, at the very least, equal to
| themselves, but dealing with this is NP-complete, as it
| essentially amounts to implementing Prolog.
|
| [1] http://www.esp.org/foundations/database-
| theory/holdings/codd...
| ironmagma wrote:
| Man, these frontend people are constantly churning through
| dependencies, a new framework every year. Don't reinvent the
| wheel, just be like backend folks and pick a dependency and stick
| with it. Oh wait...
| vundercind wrote:
| ETL and data analysis "notebook" world been warming up for
| years and now it's red-hot due to every company down to
| Grandma's Cookie Bakery LLC wanting to "do AI". And most of the
| standard tools are surprisingly terrible so churn is increasing
| for reasons similar to why that happened in frontend (the
| underlying tools are ass and the existing interfaces bad and
| almost all of it's being overcomplicated for bad reasons and a
| million more developers just jumped into the pool)
| ironmagma wrote:
| Yes, and so really what we are seeing is just the effects of
| a high quality bar. If you want high-quality software, you do
| more iterations. And iterations produce breaking changes.
| vundercind wrote:
| Mmm. A lot of it's motivated by pain, with the output
| filtered through a web of organizational friction,
| principal agent problems, self-promotional nonsense, and
| other harmful factors. If the web's any indication, there's
| no guarantee this activity is efficient at getting us
| closer to meeting a high quality bar.
| dr_mee6 wrote:
| Just as an FYI Ibis is being around for 10 years
| approximately...
| frakt0x90 wrote:
| Curious if you considered Polars. That's become the defacto
| standard in my group as we all dislike pandas.
| cpcloud wrote:
| We support Polars as a backend.
|
| DuckDB was chosen as the default because Polars was too
| unstable to be the default backend two years ago.
| amelius wrote:
| Good riddance.
|
| Pandas turns 10x programmers into 1x programmers.
| data-ottawa wrote:
| I've been using pandas for over a decade and I'm very
| efficient with it.
|
| The Modern Pandas blog series from Tom Augsburger is
| excellent, and the old Wes McKinney book (creator of pandas)
| gave a good glimpse into how to effectively use it.
|
| I don't think many people learn it correctly so it gets used
| in a very inefficient and spaghetti script style.
|
| It's definitely a tool that's showing it's age, but it's
| still a very effective swiss army knife for data processing.
|
| I am glad we're getting better alternatives, Hadley Wickams
| tidyverse really showed what data processing can be.
| codyvoda wrote:
| fun fact if you weren't aware, Ibis was also originally
| created by Wes McKinney and takes a lot of inspiration from
| the tidyverse
| data-ottawa wrote:
| I was not aware of that, the influence from the tidyverse
| is very clear
|
| I've tried to make mockups of similarly inspired API and
| I think they did a great job with clever tricks like the
| `_` reference (does that clash with IPython/Jupyter's
| use?).
|
| I do wonder how they manage compatibility with so many
| backends, it seems like many features will be directly
| tied to your backend (e.g. Trino using Java Pattern for
| regex vs BigQuery using Re2) in hard to explain ways. But
| maybe that's not a big concern, because very often you're
| only going to be using a couple of backends.
|
| I'll have to try Ibis out for myself, it looks like it
| can unify a lot of the work I have to do. I've moved away
| from pandas for all but the last mile of computation, so
| this might be a direct good SQL replacement.
| cpcloud wrote:
| `_` definitely clashes with IPython and Jupyter's use of
| it.
|
| However, if you import it from Ibis then it ceases to be
| used as "most recent result" and remains the ibis
| underscore object (unless of course you explicitly assign
| it to something else).
|
| Regarding backend compatibility, there are definitely a
| few kinds of things that we don't currently abstract
| over. One is regular expression syntax and another is
| floating point math (e.g., various algebraic properties
| that are violated that result in slightly different
| outputs).
|
| Hope you give it a go, and please report issues at
| https://github.com/ibis-project/ibis.
| data-ottawa wrote:
| Thanks for the reply!
|
| I think that's the right policy to take, and I did notice
| the support matrix on your website which addresses my
| earlier question:
|
| https://ibis-project.org/backends/support/matrix
| throwway_278314 wrote:
| Just imagine how much more efficient you would be if you
| were using R's DataTable.
|
| Look, I applaud your skill, but at some point even a master
| craftsman realizes that the swiss army knife may not be the
| best tool, and a leatherman offers certain advantages.
| cpcloud wrote:
| Sometimes I write data analysis code in R to make myself
| aware of how it is occasionally possible to have nice
| things.
| data-ottawa wrote:
| I really like R's library and I'll use them any chance I
| get (libraries like lmer are still orders of magnitude
| more efficient than the same model in Statsmodels).
|
| From my experience the biggest impediment to using R in
| production is many orgs don't have a blessed way to run
| it.
|
| R is my favourite language for data processing, the
| manual section Computing on the Language[1]is why R is
| such an ergonomic tool. I had hoped Julia would catch up,
| but Julia's macros are not comparable in their depth.
|
| I think pandas is probably the data equivalent of editing
| files using default vim or processing data with awk.
|
| [1] https://rstudio.github.io/r-manuals/r-lang/Computing-
| on-the-...
| cpcloud wrote:
| As a joke, I wrote an Ibis backend
| (https://github.com/cpcloud/ibish) that processes
| expressions using shell commands strung together with
| named pipes. It supports joins using the coreutils join
| command, projections, filters and some aggregations with
| awk.
|
| It's faster than pandas in some cases and folks should
| put it into production immediately!
| scrlk wrote:
| I also recommend Matt Harrison's talks ("Idiomatic Pandas")
| and book ("Effective Pandas").
|
| Once I adopted method chaining, a lot of the issues that I
| had with pandas in the past due to poor style (e.g.
| SettingWithCopyWarning) pretty much disappeared.
| __mharrison__ wrote:
| Curious if you could expand on this?
| Kalanos wrote:
| Lol. As if different tools for filtering rows/columns
| determines a programmer's capability.
| fifilura wrote:
| What is the difference, other than parallel execution?
| magnio wrote:
| - Much faster and more memory efficient.
|
| - Consistent Expression and SQL-like API.
|
| - Lazy execution mode, where queries are compiled and
| optimized before running.
|
| - Sane NaN and null handling.
|
| - Much faster.
|
| - Much more memory efficient.
| cpcloud wrote:
| Did it get more memory efficient during the time between
| authoring point 1 and authoring the last point?
| dgfitz wrote:
| You may think this was clever, but it is a common
| literary technique to emphasize a point.
|
| I'd suggest less snark, you're not doing yourself any
| favors.
| itsoktocry wrote:
| > _I 'd suggest less snark, you're not doing yourself any
| favors._
|
| Why do people feel the need to jump in and police tone
| like this? Who are _you_? You 're not doing yourself any
| favours, either.
|
| > _common literary technique to emphasize a point._
|
| "Common" is a stetch, and who cares.
| temp_praneshp wrote:
| > Why do people feel the need to jump in and police tone
| like this?
|
| From the community guidelines(https://news.ycombinator.co
| m/newsguidelines.html): "Be kind. Don't be snarky.
| Converse curiously; don't cross-examine. Edit out
| swipes."
|
| > Who are you?
|
| A member of the community.
| dgfitz wrote:
| > Why do people feel the need to jump in and police tone
| like this?
|
| Did you know that more information is communicated via
| tone than words?
| fifilura wrote:
| Do you use it over many machines (RAMs)?
|
| I dont care so much about the memory and CPU stuff, I
| mostly leave the heavy lifting to an SQL engine.
|
| Although the Null handling seems very compelling, I guess
| it comes at a cost of incompatibility with existing
| libraries, otherwise Pandas would have implemented it as
| well?
|
| I am curious about the SQL api though.
| cpcloud wrote:
| Polars' support for SQL is pretty nascent and missing a
| lot of functionality.
|
| If it were better, we'd use it internally in Ibis for the
| Polars backend implementation.
|
| If you're going down the mixed SQL, DataFrame API route
| then Ibis is probably the best solution out there for
| that.
|
| I work on Ibis, so take what I say with a grain of salt.
| There may yet be other libraries out that there that have
| similar functionality.
| oreilles wrote:
| When using Pandas appropriately, that is with method
| chaining, lambda expressions (instead of intermediate
| assignments) and pyarrow datatypes, you also get much
| faster speed and null values handling.
| codyvoda wrote:
| Polars is a great choice but, like pandas, locks you into its
| execution engine(s). Polars is a supported backend for Ibis,
| though depending on your use case DuckDB or DataFusion may
| scale better. we did some benchmarking recently: https://ibis-
| project.org/posts/1tbc/
| ang_cire wrote:
| I wouldn't switch to Polars as a replacement to Pandas, given
| that Polars also seems likely to die soon. I'd also recommend
| avoiding Manatees.
| cpcloud wrote:
| On the flip side, Ibises (Ibisii ??) have been around for
| years and aren't going anywhere.
| highfrequency wrote:
| Why do you think polars is likely to die soon?
| dr_mee6 wrote:
| Global warming...
| ang_cire wrote:
| Sadly, I don't think many caught the joke :)
| mrbluecoat wrote:
| Dask was really cool back in the day. Farewell, friend.
| lmc wrote:
| Back in the day? What are people using now instead?
| ZeroCool2u wrote:
| Ray seems to be a popular choice now if you need multi node.
| But Polars is really nice if you're okay with just vertically
| scaling on one machine.
| cpcloud wrote:
| I've been working with pencil and paper recently. Thinking
| about starting an Ibis backend for it.
| dr_mee6 wrote:
| Abacuses?
| thsgtu wrote:
| About time. It always surprises me how long pandas has been able
| to hold on. Wes McKinney talked about pandas' limitations back in
| 2017 https://wesmckinney.com/blog/apache-arrow-pandas-internals/
| __mharrison__ wrote:
| Not surprising. There are much better compute engines than
| pandas.
|
| Folks then ask why not jump from pandas to [insert favorite
| tool]?
|
| - Existing codebases. Lots of legacy pandas floating about.
|
| - Third party integration. Everyone supports pandas. Lots of
| libraries work with tools like Polars, but everything works with
| pandas.
|
| - YAGNI - For lots of small data tasks, pandas is perfectly fine.
| cpcloud wrote:
| All great reasons to stick with Pandas. If a thing works for
| you, then that's one less thing to manage.
|
| Fully agree that if pandas is working for you, then you're
| better off sticking with it!
| whimsicalism wrote:
| If your task is too big for pandas, you should probably skip
| right over dask and polars for a better compute engine.
| __mharrison__ wrote:
| Jump straight to cluster and skip Dask?
|
| Not sure what "big" means here, but a combination of .pipe,
| pyarrow, and polars can speed up many slow Pandas operations.
|
| Polars streaming is surprisingly good for larger than RAM. I
| get that clusters are cool, but I prefer to keep it on a
| single machine if possible.
|
| Also, libraries like cudf can greatly speed up Pandas code on
| a single machine, while Snowpark can scale Pandas code to
| Snowflake scale.
| cpcloud wrote:
| In my experience, Polars streaming runs out of memory at
| much smaller scales than both DuckDB and DataFusion and
| tends to use much more memory for the same workload when it
| doesn't outright segfault.
|
| Polars is faster than those two once you get to less than a
| few GB, but beyond that you're better off with DuckDB or
| DataFusion.
|
| I would love for this to improve in Polars, and I'm sure it
| will!
| __mharrison__ wrote:
| My understanding is that the Polars team is working on a
| new streaming engine. It looks like you will get your
| wish.
| ritchie46 wrote:
| Do you mean segfault or OOM? I am not aware of Polars
| segfaulting on high memory pressure.
|
| If it does segfault, would you mind opening an issue?
|
| Some context; Polars is building a new streaming engine
| that will eventually be ready to run the whole Polars API
| (Also the hard stuff) in a streaming fashion. We expect
| the initial release end of this year/early next year.
|
| Our in-memory engine isn't designed for out-of-core
| processing and thus if you benchmark it on restricted
| RAM, it will perform poorly as data is swapped or you go
| OOM. If you have a machine with enough RAM, Polars is
| very competitive in performance. And in our experience it
| is tough to beat in time-series/window functions.
| cpcloud wrote:
| Segmentation violations are often the result of different
| underlying problems, one of which can be running out of
| memory.
|
| We (the Ibis team) have opened related issues and the
| usual response is to not use streaming until it's ready,
| or to fix the problem if it can be fixed.
|
| Not sure what else there is to do, seems like things are
| working as expected/intended for the moment!
|
| We'll definitely be the first to try out any improvements
| to the streaming engine.
| ritchie46 wrote:
| They have different implications for us. An abort due to
| an OOM isn't a bug in our program, as SEGFAULT is a
| serious bug we want to fix.
| 0cf8612b2e1e wrote:
| Even medium tasks, the slow step is rarely computations, but
| the squishy human entering them.
|
| When I work on a decent 50GB+ dataset, I have to do something
| fairly naive before I get frustrated at computation time.
|
| Edit: pandas is now Boring Technology (not a criticism). It is
| a solid default choice. In contrast, we are still in a Cambrian
| explosion of NeoPandas wannabes. I have no idea who will win,
| but there is a lot of fragmentation which makes it difficult me
| for to jump head first into one of these alternatives. Most of
| them are faster or less RAM pressure which is really low on my
| list of problems.
| mrguyorama wrote:
| Pandas made me think I hated python.
| e10v_me wrote:
| Funny, the same) When I was switching from R (data.table) to
| Python, it was painful. Not only because it was slow, but
| because of the API. At that time, I thought that maybe it's
| because of switching to something new. Several years later,
| switching from Pandas to Polars API was a real joy (Ibis is
| also good is that sense). So, I learned that it had been Pandas
| fault all along))
| whimsicalism wrote:
| Frankly, pandas/dask/polars - all are trying to recreate
| something that has existed for years (ie. sql, spark) and with a
| terrible API.
|
| I truly thought I was terrible at python for a long time because
| of pandas - turns out it just has an absolutely terrible API
| surface
| tech_ken wrote:
| I would not say pandas is a recreation of SQL/Spark, they have
| very different use-cases in my experience. SQL/Spark is like a
| bulk data management tool: I use it if I need to load massive
| data from some remote store, perform light preprocessing, join
| up a couple dimension tables, etc. Having normalized and joined
| my data, then pandas enters as a 'last-mile' processing engine,
| particularly when paired with ex. SKLearn or whatever other
| inference lib you're using. Pandas is awesome if you need to
| ex. apply string manipulation to a data table or daisy-chain
| some complicated computations together. Honestly in my opinion
| the API is really nice, I came over from R tidyverse and the
| 'chained methods' approach to pandas let's me carry over all my
| old patterns and paradigms. I find it far easier to use that
| approach than having to write like 20 dependent subqueries or
| staging tables
| cmollis wrote:
| true spark has existed for years and is a great toolset.. i use
| it ever day. it's also a huge hassle spinning clusters up and
| down and configuration is complex.
|
| I can execute some pretty hairy scans against a huge s3 parquet
| dataset in Duckdb that I would typically have to run in either
| spark or athena.. it's a little slower, but not ridiculously
| slower. And, it does all of that from my desktop.. no clusters,
| no mem or task configs.. just run the query. Being able to
| integrate all of the expensive historical scanning and knitting
| that back into an ML pipeline with desktop python is pretty
| nice.
| dammaj wrote:
| Personnally, I tend to use Pandas because it is integrated
| everywhere, because of the ecosystem that uses it. Let's say I
| want to read data from json file (csv file, python dict, etc.)
| and I want to plot it using plotly. If Ibis is compatible with
| whatever Pands dataframe is compatible with, then for most of my
| usage I don't really care much about the "backend".
| kremi wrote:
| Pandas has been working fine for me. The most powerful feature
| that makes me stick to it is the multi-index (hierarchical
| indexes) [1]. Can be used for columns too. Not sure how the cool
| new kids like polars or ibis would fare in that category.
|
| [1]
| https://pandas.pydata.org/docs/user_guide/advanced.html#adva...
| cpcloud wrote:
| Multi-indexes definitely have their place. In fact, I got
| involved in pandas development in 2013 as part of some work I
| was doing in graduate school, and I was a heavy user of multi-
| indexed columns. I loved them.
|
| Over time, and after working on a variety of use cases, I
| personally have come to believe the baggage introduced by these
| data structures wasn't worth it. Take a look at the indexing
| code in pandas, and the staggering complexity of what's
| possible to put inside square brackets and how to decipher its
| meaning. The maintenance cost alone is quite high.
|
| We don't plan to ever support multi-indexed rows or columns in
| Ibis. I don't think we'd fare well _at all_ there,
| intentionally so.
| kremi wrote:
| > Take a look at the indexing code in pandas
|
| As the end-user, not quite my concern.
|
| > and the staggering complexity of what's possible to put
| inside square brackets and how to decipher its meaning
|
| I might not be aware of everything that's possible -- the
| usage I have of it doesn't give me an impression of
| staggering complexity. In fact I've found the functionality
| quite basic, and have been using pd.MultiIndex.from_* quite
| extensively for anything slightly more advanced than
| selecting a bunch of values at some level of the index.
| hansvm wrote:
| > As the end-user, not quite my concern.
|
| Complicated code is (probabilistically) slow, buggy,
| infrequently updated code. By all means, if it looks like a
| good enough tool for the job (especially if the
| alternatives don't) then use it anyway, but that's slightly
| different from it not being your concern.
|
| I've seen enough projects need "surprise" major revisions
| because some team tried to sneak a dataframe into a 10M QPS
| service that my default is keeping pandas far away from
| anything close to a user-facing product.
|
| I've also seen costs balloon as the data's scale grows
| beyond what pandas can handle, but basically all the
| alternatives suck for myriad reasons, so I don't try to
| push "not pandas" in the data backend. People can figure
| out what works for themselves, and I kind of like just
| writing it from scratch in a performant language when I
| personally hit that bottleneck.
| highfrequency wrote:
| Is your workload mostly single-threaded? If so, is that due to
| dataset size, or machine core count?
| kremi wrote:
| Sorry I don't know what to answer. I don't think what I do
| qualifies as "workload".
|
| I have a process that generates lots of data. I put it in a
| huge multi-indexed dataframe that luckily fits in RAM. I then
| slice out the part I need and pass it on to some computation
| (at which point the data usually becomes a numpy array or a
| torch tensor). Core-count is not really a concern as there's
| not much going on other than slicing in memory.
|
| The main gain I get of this approach is prototyping velocity
| and flexibility. Certainly sub-optimal in terms of
| performance.
| jononor wrote:
| I work a lot with IoT data, where basically everything is
| multi-variate time-series from multiple devices (at different
| physical locations and logical groupings). Pandas multi index
| is very nice for this, at least having time+space in the index.
| UncleOxidant wrote:
| Related to this pandas?: https://pandas.pydata.org/
| cpcloud wrote:
| On the contrary, very related to that pandas!
| Kalanos wrote:
| I've only heard about Ibis maybe three times in the past two
| years and I pay pretty close attention to the space. If Ibis
| moves away from pandas, then it just means that I am less likely
| to try Ibis because there is no bridge.
|
| Sure, hip new frameworks are moving away from pandas/numpy, but
| I'll wait 5 years for the dust to settle here while the
| compatibility and edge cases sort themselves out. The
| pydata/numfocus ecosystem is extensive.
|
| It's just tabular data. So what if I have to wait a few more
| milliseconds to get my result.
| cpcloud wrote:
| Sounds like a solid plan. No reason to incur switching costs if
| things are working!
|
| I think you're misunderstanding what "removing pandas" means.
|
| You can still compute on DataFrames, it'll just be with
| something other than Pandas itself, probably DuckDB, Polars, or
| DataFusion.
|
| So, the bridge was there, is there, and isn't going anywhere.
| codyvoda wrote:
| > If Ibis moves away from pandas, then it just means that I am
| less likely to try Ibis because there is no bridge.
|
| the bridge is that Ibis accepts pandas as input and has a
| `to_pandas()` method as output
|
| Ibis also still depends on pandas (and thus numpy) internally.
| also, Ibis was created by the creator of pandas, and the lead
| developer (and other contributors) have also worked on pandas.
| the Ibis team understands pandas and the broader Python data
| ecosystem very well
|
| > It's just tabular data. So what if I have to wait a few more
| milliseconds to get my result.
|
| usually as scale grows, milliseconds -> seconds -> minutes ->
| hours -> days. at some point along the way having a dataframe
| library that can scale up without rewriting your code might be
| useful. but if you're dealing with small tabular data and
| pandas meets your needs, it's a great library to use and stick
| with!
| riezebos wrote:
| Nice to see, over the past months I've replaces pandas with ibis
| in all new projects and I am a huge fan!
|
| - Syntax in general feels more fluid than pandas
|
| - Chaining operations with deferred expressions makes code
| snippets very portable
|
| - Duckdb backend is super fast
|
| - Community is very active, friendly and responsive
|
| I'm trying to promote it to all my peers but it's not a very well
| known project in my circles. (Unlike Polars which seems to be the
| subject of 10% of the talks at all Python conferences)
| glial wrote:
| In my experience, the best thing about Pandas is how much it made
| me appreciate using dplyr and the tidyverse. If it wasn't for
| Pandas, I may not be the avid R user I am today.
___________________________________________________________________
(page generated 2024-08-29 23:01 UTC)