[HN Gopher] Siuba - A Dplyr Port to Python
___________________________________________________________________
Siuba - A Dplyr Port to Python
Author : grst
Score : 73 points
Date : 2021-01-21 13:13 UTC (2 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| mangecoeur wrote:
| > A killer feature of siuba is that the same analysis code can be
| run on a local DataFrame, or a SQL source.
|
| Everyone thinks this is a good idea until they discover that SQL
| is not actually very portable, and any attempt to make it so
| neutralizes whatever benefits your SQL DB has.
|
| E.g. if you have a Postgres db, that has tons of excellent
| features that you actually want to use, sticking to a lowest-
| common-denominator flavor of SQL basically nukes all that.
| disgruntledphd2 wrote:
| I actually disagree, given that this is a dplyr port. dplyr has
| always had this feature, and I agree that it can make things
| difficult, but it can also provide a really, really useful
| abstraction over your SQL, especially if you need to change
| databases for whatever reason.
|
| In general, this actually looks like the best dplyr port I've
| seen, and may actually get me to do some exploratory analysis
| in Python.
| ellisv wrote:
| I think this is a port of the dplyr syntax/verbs only. None
| of the dplyr C++ code is used, it's all syntactic sugar on
| top of Pandas. So any SQL translation is going to be similar
| to what Pandas does.
| disgruntledphd2 wrote:
| Yeah, I probably used the wrong word, I didn't think it
| used the dplyr code, just that it looked much more like
| dplyr
| closed wrote:
| siuba does the SQL translation :). pandas is used for local
| data, since it does a lot of optimization in c++, similar
| to dplyr's low level code.
|
| Thanks for bringing this up--the docs could be clearer here
| [deleted]
| spinningslate wrote:
| kudos for doing this. Whilst I haven't used Suiba, dplyr in R is
| a very compelling way to do data munging. If Siuba brings the
| same thing to python, it's a very welcome addition to the
| ecosystem.
|
| I know pipes split opinion; I'm definitely in the 'pro' camp.
| Chaining ops in a dataflow pipeline fits my mental model well.
| jphoward wrote:
| What I think would be a nice approach would be someone creating a
| new Dataframe class that merely inherits from a Pandas dataframe
| but just implementing .filter(), .summarise(), .select() etc. as
| methods, using the dplyr syntax. If they return their own
| dataframes then chaining follows naturally.
|
| I know that isn't how dplyr works, but it feels more Pythonic,
| and this solution isn't entirely analogous to dplyr anyway.
|
| This approach seems a little complicated, though I'm sure with
| some use I could learn to enjoy it.
| ellisv wrote:
| > What I think would be a nice approach would be someone
| creating a new Dataframe class that merely inherits from a
| Pandas dataframe but just implementing .filter(), .summarise(),
| .select() etc. as methods, using the dplyr syntax. If they
| return their own dataframes then chaining follows naturally. >
| I know that isn't how dplyr works, but it feels more Pythonic,
| and this solution isn't entirely analogous to dplyr anyway.
|
| I think I agree. Pandas has a lot of overhead/baggage that I
| don't want 90% of the time. Being able to chain _simple_ verbs
| on a data frame would be great -- something like
| mtcars.groupby(cyl).summarize(avg_hp = hp.mean())
| closed wrote:
| I'm still debating chaining vs piping, but you can do..
| from siuba import _ from siuba.data import mtcars
| # mtcars is a pandas DataFrame mtcars \
| .groupby("cyl") \ .siu_summarize(avg_hp=_.hp.mean())
| closed wrote:
| Hey, author of siuba here, I totally agree that subclassing
| would be a natural choice in python. One challenge there is
| that users will often get a DataFrame back (e.g. from
| pd.read_csv), so it requires a lot of casting to the child
| class.
|
| Right now, a compromise I've been exploring is just attaching
| siuba's DF functions to a pandas DataFrame, e.g.
| df.siu_mutate(...). This seems to be what pandas wants people
| to do [1]! One obstacle here is that the DataFrame has 300+
| methods, which can be overwhelming to learners.
|
| I've spent a lot of time wondering whether the piping syntax
| feels like too much vs chaining. It's still an open question in
| my mind, so it's really helpful to hear what feels most
| natural!
|
| https://pandas.pydata.org/pandas-docs/stable/development/ext...
| jphoward wrote:
| All good points! Great work on the project and thank you for
| working to make Pandas more friendly!
| Ambroisie wrote:
| What about monkey patching the class at runtime?
|
| Edit: oh that's pretty much what the linked decorators do.
| jmount wrote:
| Neat. I've been working on my own "piped-Codd" style system I
| call the "data algebra" https://github.com/WinVector/data_algebra
|
| I use method chaining as the composing notation.
| civilized wrote:
| This is, what, the fourth? Attempt to port dplyr to Python?
| Unclear why this one will be any more successful than the others
| closed wrote:
| Hey, creator of siuba here. I think siuba's big advantage is
| that it can generate SQL code.
|
| The architecture necessary to pull off executing either pandas
| or SQL also makes it very extensible (e.g. to spark or dask in
| the future :).
|
| https://siuba.readthedocs.io/en/latest/key_features.html
| disgruntledphd2 wrote:
| This one looks much, much better than the others, as a long-
| time user of dplyr who's now a refugee in the Python universe.
| javitury wrote:
| Dplyr and data.table are two libraries that make R shine in terms
| of data manipulation.
|
| Python however has static analysis tools that are unavailable in
| R. I wonder if there is a data manipulation library in python
| that takes advantage of this. Looking at this library, it doesn't
| appear to use type hints. Other libraries, like pandas, have some
| basic support for typings but they are still far from being fully
| typed.
| closed wrote:
| Siuba uses type hints to dispatch the appropriate versions of
| custom functions!
|
| For example, siuba allows users to create custom functions
| using a thin wrapper around functools.singledispatch.
|
| When deciding how to run... df >>
| filter(my_custom_func(_.x))
|
| It requires that the return type be compatible with the backend
| being run (e.g. pandas, a SQL dialect).
|
| Would be super interesting to try and lay out what would be
| needed to do static analysis via mypy. I think it'd require
| some plugins for singledispatch at least, probably some
| reworking things in ways myoy expects.
|
| https://nbviewer.jupyter.org/github/machow/siuba/blob/master...
| unnah wrote:
| Don't forget ggplot2, another shiny tidyverse product. It has
| also been ported to python:
| https://plotnine.readthedocs.io/en/stable/
| closed wrote:
| Hey y'all, creator of siuba here--happy to answer any questions!
|
| One piece of context I try to bring into discussions is that the
| way I test and develop siuba is by livecoding data analyses for
| an hour [1]. I encounter a lot of arguments like "X is possible
| with pandas", but when I sit down with analysts in realistic
| settings (e.g. time constrained) it turns out X works in more
| limited ways then they thought [2][3].
|
| I'm a big fan of pandas though. It's what siuba is built on!
|
| [1]: https://m.youtube.com/c/chowthedog
|
| [2]: https://mchow.com/posts/2020-02-11-dplyr-in-python/
|
| [3]: https://siuba.readthedocs.io/en/latest/developer/pandas-
| grou...
| ellisv wrote:
| How's the performance? I'm certainly willing to give up a
| little computational performance for being able to write my
| code faster.
| closed wrote:
| Using the experimental fast grouped pandas functions, it
| should run at the speed of optimized pandas code!
|
| Since siuba functions just run on pandas DataFrames, you can
| always hand tune for performance, but imo most of the time
| pandas code runs slow it's because of something like
| .agg(lambda ...) somewhere.
|
| There's an example with timings here:
|
| https://siuba.readthedocs.io/en/latest/developer/pandas-
| grou...
| rogue7 wrote:
| This looks neat, the _ trick is similar to the Self [0] of
| fastcore (from fastai).
|
| However many things are possible with vanilla pandas. I use it a
| lot for data munging, usually with the fluent interface (method
| chaining) style [1], e.g.:
|
| df.loc[lambda f: ...].groupby(...).agg(["mean", "count"])
|
| It also plays nicely with the black autoformatter.
|
| Anonymous functions are verbose and limited in Python, but you
| can still do many things and use a regular function when a lambda
| won't do.
|
| I guess one of the thing I need the most when doing data analysis
| is column name autocompletion, inside groupby, lambdas, for
| column selection...
|
| I wonder if one could do it in IPython, similar to the string
| autocompletion of file/directory paths. Basically a parsing of
| dataframe column names in order to autocomplete strings.
|
| [0]: https://fastcore.fast.ai/basics.html#Self-(with-an-
| uppercase...
|
| [1]: https://tomaugspurger.github.io/method-chaining
| bobbylarrybobby wrote:
| IPython already does tab completion of data frame column names.
| E.G., `df["col<tab>` will do what you'd hope.
| lordgrenville wrote:
| As long as there isn't a space in the column name. This is
| riding on a Pandas trick of making the column name accessible
| as an attribute of the dataframe, which breaks down when
| there's a space.
| ZeroCool2u wrote:
| If the necessary information is there, as in you've mentioned a
| column name in a dataframe at least once, PyCharm will now do
| column name autocompletion. It's actually pretty solid in my
| experience.
| alexilliamson wrote:
| Didn't know pycharm would do that... thanks for the info!
| Yeah the best things about dplyr in opinion are 1) less
| verbose than pandas 2) much better autocomplete.
| closed wrote:
| Hey, thanks for pointing out Self--I definitely need to dig
| into fastcore more!
|
| One motivation for developing siuba is that the grouped agg you
| show requires users specify only one operation on one column.
|
| E.g.
|
| 1. Calculate mean of x
|
| However, common operations like demeaning a column are multiple
| operations:
|
| 1. Calculate mean of x
|
| 2. Subtract result of (1) from x
|
| In siuba you can just write mutate(res = _.x -_.x.mean()). This
| isn't possible from something like gdf.x.agg("mean"), and from
| what I can tell deeply confusing to analysts :/.
|
| In vanilla pandas I really like to use the chaining method you
| laid out, and siuba to me is mostly a utility library for
| making the approach a little more succinct / performant[1].
|
| siuba has experimental autocompletion (thanks to Tim Mastny!),
| and there's a pretty hefty technical write up on how it uses
| IPython machinery for that in siuba's architectural desicion
| record folder[2].
|
| [1]: https://siuba.readthedocs.io/en/latest/developer/pandas-
| grou...
|
| [2]:
| https://github.com/machow/siuba/blob/master/examples/archite...
___________________________________________________________________
(page generated 2021-01-23 23:01 UTC)