[HN Gopher] Tidyverse 2.0.0
___________________________________________________________________
Tidyverse 2.0.0
Author : l2dy
Score : 118 points
Date : 2023-04-08 06:40 UTC (1 days ago)
(HTM) web link (www.tidyverse.org)
(TXT) w3m dump (www.tidyverse.org)
| fithisux wrote:
| R needs to modernize. S7 is in the right direction if it
| displaces all previous failures. I love tidyverse though.
| troymc wrote:
| I'm not familiar with S7, and Google wasn't much help. What is
| S7?
| t-kalinowski wrote:
| It's a new OOP system for R.
|
| repo: https://github.com/RConsortium/OOP-WG
|
| talk: https://www.youtube.com/watch?v=P3FxCvSueag&t=258s
| kgwgk wrote:
| https://adv-r.hadley.nz/oo.html
|
| "There are multiple OOP systems to choose from. In this book,
| I'll focus on the three that I believe are most important:
| S3, R6, and S4. S3 and S4 are provided by base R. R6 is
| provided by the R6 package, and is similar to the Reference
| Classes, or RC for short, from base R.
|
| "There is disagreement about the relative importance of the
| OOP systems. I think S3 is most important, followed by R6,
| then S4. Others believe that S4 is most important, followed
| by RC, and that S3 should be avoided. This means that
| different R communities use different systems."
|
| https://rconsortium.github.io/OOP-WG/
|
| "The S7 package is a new OOP system designed to be a
| successor to S3 and S4."
|
| (https://xkcd.com/927/)
| disgruntledphd2 wrote:
| Just like Python has 4 different methods for string formatting,
| R has 4 different OOP systems...
|
| S7 does look nice though, from the README overview at least.
| kkoncevicius wrote:
| > You'll notice one other small change to the tidyverse message:
| we now advertise the conflicted package.
|
| Not a fan of the nature in which tidyverse promotes itself. They
| used to be more aggressive - i.e. advertising their books in
| package load messages [1]. But these new changes still feel like
| an ad.
|
| To me, as an outsider, tidyverse seems to mainly be about
| "dplyr". The main thing holding all of it together otherwise is
| just the fact that majority of their functions return a "tibble"
| instead of a data.frame, forcing you to load all the jungle of
| the rest of their stuff whenever you use one of their packages.
|
| [1]: https://win-vector.com/2019/08/30/it-is-time-for-cran-to-
| ban...
| ProjectArcturis wrote:
| I wish someone would make a package with tidyverse syntax and
| data.table performance.
| mb5 wrote:
| Can't say I've used it, but isn't that what dtplyr is supposed
| to provide?
|
| https://dtplyr.tidyverse.org/
| Zeratoss wrote:
| https://markfairbanks.github.io/tidytable/
|
| maybe this?
| j7ake wrote:
| Has anybody switched from R (tidyverse and ggplot2) to python and
| found they can be equally or more productive in tabular data
| wrangling, data exploration and visualisations?
|
| Still using R because I find python cannot do exploratory data
| analysis of visualisation as well as R.
| epgui wrote:
| What R has going for it is ggplot2, which is heavily inspired
| by a formally validated "Grammar of Graphics" (see the L
| Wilkinson book). One of the main reasons why the tidyverse
| works so well is because it tends to follow the organizing
| principle of "tidy data" (see the H Wickham paper) with some
| combinator interoperability between packages. R itself is
| pretty insane ("bad") in terms of language design, even though
| it has some neat tricks and features.
|
| Python is a much saner language than R (I am not talking of
| packages here), but it's still super clunky by design (less so
| than R). It can most certainly do all the things that R does,
| although its plotting libraries are either lacking in their
| design (matplotlib) or maturity (the other GoG-inspired
| libraries).
|
| When you continue on your journey to learn more languages, at
| some point you realize that when you have a well-designed,
| simple and consistent language, you really don't need very many
| packages to get the job done fast. I had this a-ha moment with
| Clojure (although I find working with the JVM annoying).
| disgruntledphd2 wrote:
| For data manipulation, R/dplyr is still much, much superior.
|
| However, there's a ggplot clone called plotnine in Python which
| seems to be reasonably comprehensive.
| bart_spoon wrote:
| No, I'm in the same boat. If I'm only interested in data
| wrangling, visualization, or analysis, R blows Python out of
| the water. With all the time and effort that have gone into
| Python data libraries, it is basically on par with base R. With
| close to a decade of tidyverse development, it really isn't
| even close.
|
| But the second I'm doing anything outside of those specific
| tasks above, Python outclasses R to an equal degree.
| tpoacher wrote:
| My 2C/ on the tidyverse is that it is a bit misleading to call it
| "a collection of packages". I think it would be more accurate to
| call it a "framework". Though I suppose "opinionated collection
| of packages" is perhaps a fuzzy synonym for framework.
|
| The difference between "collection of packages" and "framework"
| is that "collection of packages" implies interoperable via API
| conventions, but otherwise standalone packages, which could
| easily be used in isolation alongside base R.
|
| Whereas "framework" generally means tightly coupled components,
| in a "do things my way or the highway" fashion.
|
| Technically speaking, it's not "impossible" to mix tidyverse with
| base R, but in general if you wanted to mix and match tidy vs
| base R syntax, you'd find yourself working "against" the tidy api
| rather than alongside it most of the time.
|
| R is not my main language, but every time I come across a task
| and I need to make a decision between R vs tidyR, the two feel
| like completely different languages.
|
| I also have some beef against some of the decisions/conventions
| used (e.g. abusing metaprogramming to pass variables and use the
| variable's label as a value, completely ignoring the variable's
| actual value, instead of passing it as a string or as an actual
| variable ... seriously confusing when you first come across it).
| RosanaAnaDana wrote:
| I think this is a great framing. The tidy verse framework is
| straight up antagonistic towards base R and any non Hadleyverse
| packages wwith minor exception.
|
| The other glaring rhino in the room is the relationship between
| RStudio and the Hadleyverse.
|
| I find the relationship and degree of influence the tidy verse
| and Hadleyite way of doing things to be a dark cloud over the R
| community.
| HFguy wrote:
| Why is it a dark cloud? Could you just choose to not use
| tidyverse libraries?
| jwilber wrote:
| I honestly don't believe R would have any sort of user-base
| at all if the tidyverse didn't exist
| kgwgk wrote:
| R had some sort of user-base when the tidyverse didn't
| exist.
| kuhewa wrote:
| For the most part that community is still around and
| still uses quite a bit of base R, you will see them
| overrepresented answering curly questions in help forums
| iaw wrote:
| My experience is that usually leveraging quosures I can make
| a wrapper function to get Hadlerverse behavior from any other
| packages.
|
| I believe that Hadley's work is the driving reason R remains
| a competitive alternative to Python for data analysis.
| kuhewa wrote:
| I find it entirely possible and it's my normal work flow to mix
| a little tidyverse in with base R without going against the
| grain, the worst it gets its probably converting data from long
| to wide format now and again but the functionality to do so is
| intuitive and simple. Bigger differences between graphics but
| there wouldn't be a need to mix ggplot grid graphics and base r
| graphics in the same plot (although panels in the same multi
| figure plot for example is trivial).
| civilized wrote:
| Eh, framework is a bit strong (if we're thinking about
| something like Ruby on Rails). You can easily mix dplyr and
| lubricate with base R things like for loops, subscripted
| assignments, low-level functions like grep, etc. I do it all
| the time.
|
| I agree that nonstandard evaluation can be tricky, but the nice
| notation you get is worth it IMHO, and at this point you can
| switch to standard evaluation in a number of convenient ways.
| My favorite way is to just use base R temporarily, since
| outside of modeling functions it mostly works by standard eval.
|
| And I think that's good advice in general. The best way to use
| the Tidyverse is to use it for what it's good at. And for
| things it's not so good at, use something else for a bit before
| coming back to it.
| kgwgk wrote:
| > Whereas "framework" generally means tightly coupled
| components, in a "do things my way or the highway" fashion.
|
| Using that definition how would lubridate be part of the
| tidyverse "framework"?
|
| > Technically speaking, it's not "impossible" to mix tidyverse
| with base R, but in general if you wanted to mix and match tidy
| vs base R syntax, you'd find yourself working "against" the
| tidy api rather than alongside it most of the time.
|
| I don't know, wrapping readxl::read_excel with a as.data.frame
| or as.data.table call works beautifully for me.
| RosanaAnaDana wrote:
| I think he's almost certainly speaking of the %>% group_by
| convention that is basically demanded of within the
| tidyworld.
|
| I load magrittr and ggplot and toss the rest out. Right to
| left reading order is all I really want from that world. Data
| tables are such a superior convention to tibbles that it's
| not even funny (I need a tibbles like I need another hole in
| the head).
| HFguy wrote:
| What is wrong with tibbles?
| Bootvis wrote:
| They are not data.tables and sometimes I prefer working
| with data.tables.
| kuhewa wrote:
| Their print method annoys me and I don't like the
| benevolent paternalism of attempting to prevent you from
| modifying indexes of the dataframe.
| kuhewa wrote:
| But base r already has split() which does the same thing,
| turns a dataframe into a list of dataframe divided by
| values of a column. With the base r pipe operator and base
| functions like subset, aggregate, transform, split, you can
| basically replicate that convention while loading exactly
| no tidyverse packages. You can also use group_by(data,
| column) without a pipe if wanting to smuggle in a specific
| dplyr function.
| [deleted]
| tecleandor wrote:
| There's a bit of circular logic in here. Can somebody make a
| short explanation on what's this is about?
| Overview The tidyverse is a set of packages that
| work in harmony because they share common data representations
| and API design. The tidyverse package is designed to make it easy
| to install and load core packages from the tidyverse in a single
| command.
| mattkrause wrote:
| Under the hood, the tidyverse is comprised of many packages,
| some very large and sprawling, like ggplot, and others that do
| one single thing (readr). Here's a list:
| https://www.tidyverse.org/packages/
|
| Individual packages can be used in isolation, if that's what
| you need. However, they complement each other nicely, in that
| they share data representations and similar conventions while
| handling different facets of data analysis. Thus, people often
| import the whole thing in one shot _as the tidyverse_ and go
| from there. For example, you might read the data in with readr,
| clean it up with tidyr or dplyr, analyze it with tidymodels,
| and plot the results with ggplot.
|
| The python equivalent would be a hypothetical meta-package that
| tightly integrates pandas, seaborn, and sklearn.
| jstx1 wrote:
| It looks circular because of term overloading.
|
| There is "the tidyverse" which is a collection of packages like
| dplyr and tidyr for data manipulation and ggplot2 for
| visualization.
|
| Then there's "the tidyverse package" which allows you to import
| them all at once with "library(tidyverse)". You don't have to
| use the tidyverse package, you can load individual libraries as
| you need them.
| epgui wrote:
| This!
|
| Also worth noting that one of the common organizing
| principles between all packages in the tidyverse is the
| concept of "tidy data" (see specifically H Wickham's paper),
| and the use of intercompatible combinators.
| kgwgk wrote:
| It's quite confusing.
|
| I don't think that lubridate has any relationship with anything
| else, for example, apart from being created by the same people.
| qsort wrote:
| It's a popular set of packages for data analysis in R. Similar
| to Python, it's an interpreted language with compiled libraries
| that do most of the heavy lifting.
| sowbug wrote:
| Thanks; that's helpful. "Data analysis in R" belongs
| somewhere in the quoted intro paragraph. Otherwise that
| paragraph could describe almost anything: game engine,
| graphics library, string processing, HTML rendering, embedded
| systems...
| [deleted]
| VeninVidiaVicii wrote:
| I have some concerns about the direction that certain packages
| within tidyverse, particularly dplyr, are heading.
|
| One issue I've noticed is the increasing frequency of warnings,
| such as when performing left or inner joins. These warnings can
| be a bit overwhelming at times.
|
| However, it's worth noting that you can suppress these warnings
| by using the "multiple='all'" parameter, which is easy to fumble
| when you're exploring data, why not simply "all=TRUE"? It seems
| like a lack of ergonomics and considering user experience.
| nojito wrote:
| joins are one of the biggest causes of data quality and
| integrity issues in the data space.
| naijaboiler wrote:
| I avoid joins in R like a plague. Been burned. If I am
| working with a finite number of colummns, I just use base R
| match and add column by column. yes it breaks DRY but I don't
| get weird surprises. and I can re-run any section of the code
| at any time from any point without breaking things or joining
| the same and creating unintended duplicate columns
| victorbohr wrote:
| Take a look at the powerjoin package. It let's you specify
| strict rules for your joins, for example requiring unique
| keys in the right-side table when doing a left join. It's
| got some other neat features as well.
| LaserPineapple wrote:
| What kind of weird surprises have you run into? Are they
| specific to R?
| VeninVidiaVicii wrote:
| Yeah maybe, but chaining a bunch of joins together when
| exploring some new data is a thing I do all the time. Then,
| 4-5 warnings push my results off the screen. It's
| infuriating.
| civilized wrote:
| I am an unapologetic Hadley fanboy. He is a genius. Never in my
| working life has anyone made so many good decisions that directly
| benefitted me. My R productivity is so incredibly much higher
| than it would have been without him.
|
| The worst thing I can say about tidyverse is that I find the
| stack traces increasingly gnarly. The information you need to
| debug is there, there's just a lot of filler. I have some faith
| that they know and they're working on it.
| qudat wrote:
| I've had to dip my toes into R recently to help my wife with data
| cleaning. `dplyr` is what I reach for when helping her.
|
| The pipe operator as well as treating csv tables as SQL was
| intuitive for me as a software engineer.
| peatmoss wrote:
| The Tidyverse is a true gem in modern programming that deals with
| data.
|
| This release feels like a nice incremental improvement rather
| than a revolution. Adding lubridate as a core package will save
| me a few keystrokes. I also did not know about the conflicts
| package, but will certainly be using it in the future.
|
| I'm a bit surprised that other "better" programming languages
| haven't done more to replicate the Tidyverse. The Tidyverse is a
| masterclass in metaprogramming, and has a pretty big following.
| While R has some warts as a language, I'm not sure I've seen
| anything comparable to Tidyverse in terms of scope and influence.
| I see Julia trying with the Dataframes ecosystem, but it's got a
| ways to go before that feels cohesive like Tidyverse.
|
| If I could wave a magic wand, I'd get Hadley and the RStudio crew
| really excited about Julia or Racket. R _does_ have warts, and I
| dream of the day when something better is as good.
| 0xDEF wrote:
| >I'm a bit surprised that other "better" programming languages
| haven't done more to replicate the Tidyverse
|
| Python has a very weak metaprogramming story so it is hard if
| not impossible to replicate something like Tidyverse in that.
|
| If Ruby had succeeded in data science instead of Python it
| would almost certainly be with Tidyverse-like abstractions.
| AuthorizedCust wrote:
| > _This release feels like a nice incremental improvement
| rather than a revolution._
|
| Look at the big picture, not simply the delta from the prior
| .x.y release.
|
| The Tidyverse feels like a product that evolves over time,
| rather than a traditional approach that jumps versions. Major
| changes sometimes come in seemingly minor releases. I feel that
| 2.0 simply means the evolution has gone on enough to put a
| retrospective marker in the sand.
|
| For something geared more to data science and exploration,
| rather than production use, I'm good with the evolutionary
| approach.
| peatmoss wrote:
| I agree 100% that this is a good thing. I had intended this
| as a compliment that Tidyverse had achieved this level of
| maturity, but failed in my under-caffeinated state. _Had_
| Tidyverse 2.0.0. been a revolution, I 'd have been worried.
| _Wintermute wrote:
| I don't know if it's impressive or worrying regarding how much
| Posit/RStudio as a company have completely taken over a
| programming language.
|
| I have colleagues who write R code all day, and they don't know
| the difference between RStudio the IDE and R the language. They
| can't perform basic tasks such as defining a function, or
| adding a column without the tidyverse.
| kuhewa wrote:
| I write in R for much of my work days, and perhaps because I
| learned when the tidyverse was in its infancy (or before) I
| don't use much at all of its packages besides ggplot2. With
| the base r pipe operator I use zero dplyr and find most of
| its functionality has long existed in base R with functions
| like transform(). lubridate has a few good convenience
| functions (that ultimately I could do without), and I do like
| plotting in ggplot2 - in fact the latter is the only place I
| reckon there has been too much of a takeover because most
| plotting development effort I imagine is based on the grid
| and ggplot2 framework now. I do like tidyr::pivot_longer(),
| although it wouldn't change much if I had to use the less
| intuitive base R versions for dataframe manipulation.
|
| Your colleagues are probably a symptom of how low the barrier
| to entry has been made by the tidyverse moreso than
| indicative of a total capture of the language.
| fbdab103 wrote:
| I see that same thing all the time. RStudio = R, and they
| have no concept of the language without the IDE. Which is
| fine, I guess? They are not professional programmers, but
| people with a job to do.
|
| It is just frustrating to me when they RShiny all the things,
| and I have to pickup the pieces because the spaghetti has
| collapsed under its own weight.
| peatmoss wrote:
| I see it a little bit more like a meritocracy in that space
| that RStudio has largely won. Microsoft has certainly made
| an attempt to become the face of R. Remember when they
| purchased Revolution Analytics and started baking it into
| everything? Or how about Oracle making a play on Graalvm
| with "FastR"?
|
| RStudio has somewhat indefatigably looked for pain points
| that R users had back in the olden days. And yes, I
| remember R back when lattice plots were the hottest game in
| town. Not only has the Tidyverse made R infinitely more
| pleasant to work with, the community around Tidyverse has
| made the R community infinitely less insufferable for
| newcomers. The obtuse responses to beginners on the old R
| mailing lists were _legendary_.
|
| My preferred R IDE is emacs, and I'm not upset with what
| RStudio has done. When I interact with devtools, I don't
| feel that I'm wanting for functionality. This is in spite
| of the fact that RStudio has heavily integrated devtools
| functionality into their IDE, and could arguably have made
| devtools less easy to work with using generic tooling.
|
| I understand the mediocrity of many RShiny applications in
| terms of how they are written. But, another way of looking
| at all those dashboards and interactive applications, is
| that they are AMAZING prototypes written by scientists,
| grad students, journalists, etc., who would have never been
| able to build them some other way. If we forced everyone to
| write these apps in Rust, of course the average quality
| would be higher, but we'd also have way fewer of them.
| Democratization of technology means that its users will be
| more... medium.
| fbdab103 wrote:
| >...who would have never been able to build them some
| other way...
|
| It has definitely become the best available MVP
| generator. I just wish I liked R more. Python still have
| nothing even close to how easily a novice can spin up a
| Shiny application.
| kgwgk wrote:
| I'm not sure how usable it is, but Shiny for Python
| exists: https://shiny.rstudio.com/py/
| fbdab103 wrote:
| I think that package is still pretty early days, but does
| show promise. I was speaking less to the code and more to
| the ecosystem.
|
| If I tell someone they should use Streamlit/Dash/Shiny
| for Python, how do they get started? The beginner
| onboarding experience is horrible for someone without an
| existing programming background. Download Python, maybe
| instruct them to setup a "virtual environment", use this
| "pip" thing to install these packages, point them at a
| way to edit code, and try to give them a way to launch
| it. God help them if want to share their work. Huge
| amount of existing resources assume Linux by default or
| that people have any idea how to use Docker.
|
| While I hate...basically everything about the R
| ecosystem, the beginner experience is leagues better.
| Download RStudio. It will setup R, install packages, and
| has a big button that will launch the Shiny application.
| Straightforward guidance on how to get it hosted on an
| existing provider.
| peatmoss wrote:
| Julia seems to be doing a better job catching up to R in
| this space than Python. I haven't used it personally, but
| the demos of Genie Framework are impressive:
| https://github.com/GenieFramework/Genie.jl /
| https://genieframework.com/
| leephillips wrote:
| "Tidier.jl is a 100% Julia implementation of the R tidyverse
| mini-language in Julia."
|
| https://github.com/TidierOrg/Tidier.jl
| peatmoss wrote:
| This illustrates the point perfectly. Julia is attempting
| this and has a beachhead with Dataframes.jl. Confusingly
| though, Tidier.jl isn't really analogous to R's Tidyverse.
| It's more like one of a handful of meta-packages around
| Dataframes.jl.
|
| Then there are Grammar of Graphics (ggplot was Tidyverse's
| first star) style plotting libraries that Julia has been
| building. I'm probably most excited about Algebra of Graphics
| (https://github.com/MakieOrg/AlgebraOfGraphics.jl/) as part
| of the Makie Plots ecosystem. It does still feel a bit like
| Julia community can't decide between following Matplotlib or
| R's Grid/Ggplot approach.
|
| The seeds of a Tidyverse for Julia are there, but it'll take
| some time to achieve the consistency and maturity of the
| original Tidyverse.
| martinsmit wrote:
| I think DF.jl works remarkably well as a "fits in RAM"
| dataframe backend, but I think it just lacks in usability
| and integratedness with the wider Julia ecosystem. Or
| rather, the wider ecosystem isn't as mature in key data
| analysis areas.
|
| In particular, as you mention, plotting is one of the
| evolving parts of the ecosystem. Plots.jl is _fine_ , Makie
| is powerful but very DIY, and AoG is slick but unwieldy.
| ggplot2 is far from perfect, but it works so well due to
| its maturity and integration with the rest of the
| Tidyverse.
|
| In my ideal world, there would be a DataFrames.jl wrapper
| to provide nice (not just _nicer_ like the two DFM.jl
| packages) syntax, and a powerful high-level plotting
| package (Makie is powerful but syntax is low level, Plots
| is mid on both) which is heavily integrated with the
| wrapper package.
|
| Admittedly, I'm not a data scientist (anymore) so I don't
| follow the new developments in the dataviz scene much. If
| something like this exists then I would love to find it.
|
| I wonder what my ideal syntax would look like anyway. Maybe
| something close to Tidyverse but with symbols as column
| names `:col_name` for ambiguity reasons.
| peatmoss wrote:
| I'd agree with all of this. It's not so much the presence
| of ecosystem, it's the maturity of the ecosystem and
| knowing how to navigate it. Lots of competing approaches
| in a relatively small community makes me a little
| nervous.
|
| If people are curious, DataFrames.jl has done a fair bit
| of work in consolidating a list of other packages that
| complement DataFrames.jl:
|
| https://dataframes.juliadata.org/latest/#DataFrames.jl-
| and-t...
|
| I do have to say that DataFrames.jl itself is a pretty
| impressive bit of work, and Bogumil Kaminski deserves no
| small amount of credit. If Julia ends up stealing
| significant mindshare from R / Python data science
| communities it'll be because of of this work.
|
| Also for the Julia curious, I'll say that I find the
| ergonomics and sensibilities of the DataFrames.jl
| ecosystem much more in line with R and the Tidyverse than
| Python/Pandas.
| martinsmit wrote:
| Bogumil is a truly outstanding member of the community
| and DataFrames.jl is an impressive, versatile package.
|
| From my perspective, however, DataFrames.jl's power is
| what makes it quite unergonomic for me. As an example,
| take the `args => transformations => result` syntax for
| doing pretty much anything in DataFrames. It versatile,
| but the lack of rank polymorphism in Julia i.e.
| broadcasting/mapping has to be explicit (which is usually
| a good thing given that type polymorphism is Julia's
| whole schtick) means that the transformation syntax feels
| cumbersome.
|
| It's not that I want everything rowwise by default, an
| option provided by DataFramesMacros.jl, it's that I want
| things to be rank polymorphic when it makes sense. Base R
| got this right, hell S got this right, and so the
| Tidyverse inherited it and it makes the package so much
| more ergonomic than it would otherwise be.
|
| I cannot overstate how impressive DataFrames.jl is, but I
| have to caveat this with "but I really try to avoid using
| it if possible". It's a shame, but I just think R's
| laissez-faire hackability, which in many cases results in
| spaghetti code, works really well in the tabular
| programming world where ergonomics are king and
| performance is easy.
| chaosfox wrote:
| I have been using R almost daily for years now and its no
| hyperbole when I say that if ggplot2/dplyr wasn't a thing I would
| have never bothered with R.
| trts wrote:
| Adding to that, it is what makes me miss R when using other
| languages. It just isn't very much fun to me to do discovery
| and building in python, but ultimately where things get
| operationalized.
___________________________________________________________________
(page generated 2023-04-09 23:01 UTC)