[HN Gopher] Discovering copy-on-write in R
___________________________________________________________________
Discovering copy-on-write in R
Author : franklin_p_dyer
Score : 39 points
Date : 2023-12-19 17:08 UTC (2 days ago)
(HTM) web link (franklin.dyer.me)
(TXT) w3m dump (franklin.dyer.me)
| karencarits wrote:
| The data.table package may also make a huge difference in
| performance, and often simplifies the code as well
| https://github.com/Rdatatable/data.table
| JZL003 wrote:
| Copy on write is really nice, especially when I often face a very
| very large read-only matrix (200+GB) and want to do some
| embarrassingly parallel processes on subsets of it. I haven't
| found a language which makes it as easy, not python (although not
| unexpected), not Julia even
| mplewis9z wrote:
| If you haven't tried Swift, copy-on-write is one of the core
| tenets of its value types (`struct`s, basically), and it's
| almost entirely transparent.
| GrumpySloth wrote:
| It's only transparent for types that already implement copy-
| on-write. For custom types you need to implement it yourself,
| using reference typed private properties.
| sampo wrote:
| > not python
|
| Pandas has a global option to turn on copy-on-write.
|
| https://pandas.pydata.org/docs/dev/user_guide/copy_on_write....
| fbdab103 wrote:
| News to me! Will definitely break some of my current code
| (chained assignments no longer work), but is probably a more
| sensible default.
|
| To be default mode in Pandas 3, but seeing as how long it
| took them to pull the trigger on Pandas 2, that could be a
| while.
| tpoacher wrote:
| Matlab/Octave are also copy-on-write. It's quite a powerful
| mechanism to take advantage of when you're aware of it.
| 2devnull wrote:
| >" How to prevent this? The obvious way is to just not use
| dataframes, at least not while doing aggregation. Rather than
| allocating a huge dataframe and loading our partial results into
| its columns bit by bit, we can just store our partial results in
| a plain list."
|
| A lot to be said for not defaulting to data frames, in both r and
| python. Or, if you must, using something like r data.table or
| python's polars if you don't think in other data structures
| easily or just want convenience.
| franklin_p_dyer wrote:
| For sure! Definitely an good thing to know for an R newbie like
| me who is handling large datasets naively.
|
| Thanks for mentioning polars, I hadn't heard of it before but
| it looks neat.
| IKantRead wrote:
| > A lot to be said for not defaulting to data frames, in both r
| and python
|
| I would even add _especially_ in Python. The main issue I have
| found is that pandas heavy code is just not as easy to
| integrate into other Python tools /features/abstractions as
| code using mostly numpy, dictionaries and various
| comprehensions to do the vast majority of your work.
|
| As a heavy pandas user for several years, I decided about a
| year ago to not import pandas by default and instead treat most
| data problems like regular python problems. I've been genuinely
| surprised as how much easier it is to create useful
| abstractions with the code I've been writing, and also how much
| easier it's been to onboard non-DS devs into the code base.
|
| There are a few obvious cases when Pandas is very helpful, and
| I'll pull it out in those places, but I've been able to do a
| tremendous amount of data work in the last year and used _very_
| little pandas. The result is that I have an actual codebase to
| work with now rather than a billion broken notebooks.
| franklin_p_dyer wrote:
| In what cases have you found it worthwhile to use pandas?
| isoprophlex wrote:
| Resampling, regularizing, binning and forward/backward
| filling time series data is an absolute pain in the ass
| using only SQL and/or vanilla python. It does its thing
| well, there.
|
| (Note that in general, I'm the biggest pandas hater I know)
| canjobear wrote:
| It can be nice for groupby-aggregate logic. And it feeds
| into plotnine.
| kristjansson wrote:
| > The result is that I have an actual codebase to work with
| now rather than a billion broken notebooks.
|
| This is the biggest part. Giving yourself permission to make
| real abstractions, rather than forcing yourself to go
| directly from data-on-disk to pandas (or whatever) makes it
| that much easier to test, repeat, modify, and extend whatever
| analysis you're working on.
| ryanmonroe wrote:
| Good article. Some smaller changes you could make to the final
| function: In the last line `as.data.frame(do.call(cbind,
| out_list))` is used to convert the list to a data.frame. Passing
| it to `cbind` converts the list to a matrix (i.e. combines it
| into one long vector internally), and then `as.data.frame`
| converts it back to a list (as noted in the article data frames
| are lists). Instead, you can use `as.data.frame(out_list)` to
| make your list a data frame directly, to avoid converting the
| list to a matrix and back to a list again. The
| `unlist(lapply(split(cvec, groups), aggfun))` is also doing a lot
| of work, if you don't mind using an external package*,
| `collapse::BY(cvec, groups, aggfun)` is much faster (doesn't
| require converting `groups` to factor, doesn't copy `cvec`'s
| contents like `split`).
|
| Here's some completely not-the-point of the article code review
| since I can't help myself. If you can set up earlier steps give
| you a named list for `col_grouping`, and use `lapply`, the code
| is a little more concise: efficient_flow_agg <-
| function(dat, col_grouping, gpcol_name="GroupMembership") {
| make_postproc <- function(gp, groups) {
| gp$preproc(dat[gp$which_cols]) |>
| lapply(collapse::BY, groups, gp$aggfun) |>
| gp$postproc() } col_grouping |>
| lapply(make_postproc, groups = dat[[gpcol_name]]) |>
| as.data.frame() }
|
| * I had previously written here that `tapply` is probably faster,
| but apparently `tapply` does exactly `unlist(lapply(split(x, g),
| f)))` anyway? wtf R. Strange there's not something like
| `collapse::BY` in base R.
| franklin_p_dyer wrote:
| Thanks for the feedback! The business with `cbind` is a
| facepalm, I'll definitely fix that. I don't think it will
| affect performance much since that last step won't be repeated
| many times, but it makes me cringe now knowing how redundant it
| is.
|
| Good advice on `col_grouping` as well, accessing those
| components of an aggregation rule by index rather than by name
| is a bad code smell and decreases readability for sure.
| kazinator wrote:
| There is a macro like this for Common Lisp: _modf_.
|
| https://github.com/smithzvk/modf
|
| With _modf_ , you use the existing place syntax to refer to part
| of an object. It looks like you're mutating that object, but in
| fact it will return a clone of the entire containing object, with
| the modification, while the original remains untouched.
___________________________________________________________________
(page generated 2023-12-21 23:01 UTC)