--- title: "Asymmetric Smoothed-Association Matrices" output: rmarkdown::html_vignette: toc: true fig_width: 7 fig_height: 7 mathjax: "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml.js" pandoc_args: ["--include-in-header=mathjax-config.html"] vignette: > %\VignetteIndexEntry{Asymmetric Smoothed-Association Matrices} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.alt = paste( "Asymmetric smoothed-association matrix drawn by janusplot:", "each off-diagonal cell shows the fitted mgcv spline of the row", "variable on the column variable, with a 95% confidence band", "over the raw points, a border coloured by non-linearity and a", "strip of annotations above the panel; the diagonal cells show", "the kernel density of each variable." ), fig.width = 7, fig.height = 7, dpi = 110, eval = TRUE ) library(ggplot2) set.seed(2026L) has_mass <- requireNamespace("MASS", quietly = TRUE) has_penguins <- requireNamespace("palmerpenguins", quietly = TRUE) has_kernr <- requireNamespace("kernR", quietly = TRUE) # One row of summary(fit): the cell `response ~ s(predictor)`. cell <- function(fit, response, predictor) { tab <- summary(fit) tab[tab$var_y == response & tab$var_x == predictor, ] } f1 <- function(v) sprintf("%.1f", v) f2 <- function(v) sprintf("%.2f", v) ``` ## Why Pearson is not enough A Pearson correlation matrix gives one scalar per pair of variables. Two things are discarded in that collapse: 1. The *shape* of the association -- linear, monotone non-linear, U-shaped, or irregular. 2. The *direction* -- whether $y$ is a smooth function of $x$ differs in general from whether $x$ is a smooth function of $y$, because leverage and noise are directional. janusplot fits both directions of every pair with `mgcv::gam()` and draws both fits in a matrix. Each smooth carries its effective degrees of freedom (EDF) and F-test, and `adjust` can add covariates and random effects. ## Quick start `janusplot_fit()` fits every ordered pair once. Printing the fit gives an overview, and `plot()` draws the matrix. ```{r quickstart, fig.cap = "Asymmetric association matrix for four mtcars traits. Each cell shows the row variable as a smooth function of the column variable; the border colour is the cell's non-linearity and the background its Pearson correlation.", fig.alt = "Four-by-four asymmetric smoothed-association matrix for mpg, hp, wt and qsec, with coloured cell borders and blue to orange cell backgrounds."} library(janusplot) fit <- janusplot_fit(mtcars[, c("mpg", "hp", "wt", "qsec")]) fit plot(fit) ``` Every view below draws from this one fit without refitting. Each off-diagonal cell `[r, c]` shows `vars[r] ~ s(vars[c])`: the row variable on the vertical axis, the column variable on the horizontal axis, as in `pairs()`. The key in the top-left corner states this with the first two variables. In every cell: - the raw points are grey, under the fitted spline and its 95% confidence band; - the border colour is the non-linearity of that smooth, EDF minus 1: dark purple for a straight line, through green to yellow for a strongly bent curve; - the background is the pair's signed Pearson correlation, white at 0, blue towards -1 and orange towards +1; - the strip above the panel carries the significance glyph of the smooth's F-test (`***` p < .001, `**` p < .01, `*` p < .05, `.` p < .1; read from the multiplicity-adjusted p-value when the fit uses `p_adjust`) and, in the upper triangle, the asymmetry index `A` of the pair (0 when both directions are equally complex; defined below), written once because it belongs to the pair. The diagonal shows each variable's kernel density with a rug of its values: a skewed, bimodal or clipped variable shows there before it bends a curve. With `group`, one density per level is drawn in the group colour, scaled so the levels sum to the grey pooled density. The default annotations, shape legend and corner key depend on the number of variables: from thirteen variables the strip carries the glyph alone, and A is read from `summary()`. The two cells of a pair differ. Here `qsec ~ s(hp)` is a straight line (EDF `r f2(cell(fit, "qsec", "hp")$edf)`), while `hp ~ s(qsec)` bends (EDF `r f2(cell(fit, "hp", "qsec")$edf)`), so the pair has the largest asymmetry index of the matrix, A = `r f2(cell(fit, "hp", "qsec")$asymmetry_index)`. ## The matrix ### What each cell encodes `colour_by` chooses the quantity on the border: `"nonlinearity"` (the default), `"asymmetry"`, a correlation (`"pearson"`, `"spearman"`, `"kendall"`), `"deviance_gap"` (deviance explained by the smooth minus that of the straight line) or `"none"`. `background_by` chooses the correlation behind the points, or `"none"`. `annotations` chooses what the strip says (shape codes and categories are explained in *Shape metrics explained* below), from `"A"`, `"edf"`, `"signif"`, `"code"` (the two-letter shape code), `"shape"`, `"k_warn"`, `"n"`, `"r"` and `"formula"`. A synthetic example with a quadratic and a sinusoidal pair: ```{r nonlinear, fig.height = 8.5, fig.cap = "Association matrix for a quadratic pair, a sinusoidal pair and an independent variable. The border colour picks out the non-linear cells; the strip carries the shape code, decoded by the legend below.", fig.alt = "Four-by-four smoothed-association matrix for x1, x2 (quadratic in x1), x3 (sinusoidal in x1) and x4 (independent noise), followed by a legend of shape categories."} n <- 300 x1 <- runif(n, -3, 3) d <- data.frame( x1 = x1, x2 = x1^2 + rnorm(n, sd = 0.6), # quadratic in x1 x3 = sin(x1) + rnorm(n, sd = 0.4), # sinusoidal in x1 x4 = rnorm(n) # independent ) fit_nl <- janusplot_fit(d) plot(fit_nl, annotations = c("code", "signif")) ``` `x2 ~ s(x1)` has EDF `r f1(cell(fit_nl, "x2", "x1")$edf)` and `x3 ~ s(x1)` has EDF `r f1(cell(fit_nl, "x3", "x1")$edf)`, and their borders are at the yellow end of the scale. The Pearson correlation of `x1` and `x2` is `r f2(cell(fit_nl, "x2", "x1")$cor_pearson)`, so their background is almost white: the relationship is strongly quadratic and nearly uncorrelated. With `"code"` among the annotations, the legend of the 27 shape categories is attached below the matrix (`janusplot_style(show_shape_legend = TRUE)` or `FALSE` sets it whatever the annotations). ### Asymmetry: a heteroscedastic example When the noise scale depends on a predictor, the two directional smooths diverge: $y \sim s(x)$ recovers the linear mean, while $x \sim s(y)$ estimates $E[x \mid y]$, which bends where the spread of $y$ is largest. ```{r asym, fig.cap = "Association matrix for a heteroscedastic x-y pair (noise variance grows with x) plus an independent z, coloured by the asymmetry index A of each pair.", fig.alt = "Three-by-three smoothed-association matrix for x, y (heteroscedastic in x) and z (independent noise); the two x-y cells carry visibly different fitted curves."} n <- 400 x <- runif(n, 0, 5) het <- data.frame( x = x, y = 0.5 * x + rnorm(n, sd = 0.3 + 0.4 * x), # variance grows with x z = rnorm(n) ) fit_het <- janusplot_fit(het) plot(fit_het, colour_by = "asymmetry", annotations = c("A", "edf")) ``` The asymmetry index of a pair compares the EDF of its two smooths, $A = |EDF_{y \sim s(x)} - EDF_{x \sim s(y)}| / (EDF_{y \sim s(x)} + EDF_{x \sim s(y)}) \in [0, 1]$, and is the same in both cells. Here $y \sim s(x)$ has EDF `r f2(cell(fit_het, "y", "x")$edf)` and $x \sim s(y)$ has EDF `r f2(cell(fit_het, "x", "y")$edf)`. The `x`-`y` pair has A = `r f2(cell(fit_het, "y", "x")$asymmetry_index)`; the pairs with the independent `z` sit near A = 0. A is estimated from the sample: at finite `n`, one direction of a REML-selected smooth can pick up a spurious wiggle between two unrelated variables and give their pair a clearly positive A. A also depends on the distribution of the predictor, not only on the relationship. When `y` is a straight line in `x` plus noise, the mean of `y` given `x` is linear, but the mean of `x` given `y` is linear only when `x` is Gaussian. A uniform or otherwise bounded `x` bends the reverse mean near the ends of the range. With 200 rows and a uniform `x`, a straight-line pair typically has an A between 0.5 and 0.7, where a Gaussian `x` gives an A near 0.05. Here `x` is uniform, so part of the `x`-`y` index would appear with constant noise as well. Read A beside both EDFs and the predictor's distribution, not as a measure of non-linearity on its own. Shape labels describe the fitted smooth. They are not evidence of an association: a smooth through pure noise with EDF near 1 still gets a monotone label. The p-value carries the evidence, so `print()` shows it beside each label, and the strip adds "n.s." to a shape code or glyph in a cell whose p-value is 0.05 or more. ### Colouring points by group `group` colours the raw points by a factor, character or logical column of the data. It is drawing only: the column never enters the model. ```{r group, eval = has_penguins, fig.cap = "Palmer penguins trait matrix with the raw points coloured by species. The smooths are fitted across all three species.", fig.alt = "Four-by-four association matrix for bill length, bill depth, flipper length and body mass, with points in three colours by species."} pen_vars <- c("bill_length_mm", "bill_depth_mm", "flipper_length_mm", "body_mass_g") fit_pen <- janusplot_fit(palmerpenguins::penguins, pen_vars) plot(fit_pen, group = "species") ``` The pooled curves of `bill_depth_mm` cut across the three species clusters, so each pooled shape mixes within-species and between-species trends. The section on adjustment below fits species in the model. ### Ordering the variables For a larger matrix, `order = "hclust"` reorders the axes by hierarchical clustering of the absolute Pearson correlations: ```{r hclust, eval = has_mass, fig.cap = "Boston housing matrix with axes reordered by hierarchical clustering on |correlation|: medv, lstat and rm group together, and age, indus, nox and dis form a second block.", fig.alt = "Seven-by-seven Boston housing association matrix with rows and columns reordered by hierarchical clustering on absolute correlation."} data(Boston, package = "MASS") fit_boston <- janusplot_fit(Boston[, c("medv", "lstat", "rm", "age", "indus", "nox", "dis")]) plot(fit_boston, order = "hclust") ``` The reordering places `lstat` (r = -0.74) and `rm` (r = 0.70), the two variables most correlated with `medv`, next to it, apart from the `age`-`indus`-`nox`-`dis` block, whose members all have `|r| >= 0.6` with each other. ### Appearance: `janusplot_style()` Every appearance setting lives in one `janusplot_style()` object passed as `style`: the palette, where the colour is painted (`colour_channel = "border"`, `"tile"` or `"fill"`), the points, the band, the text sizes, the legends and the key. `"border"` colours the cell outline, `"tile"` a small square in the bottom-right corner and `"fill"` the whole panel. Printing a style lists the settings that differ from the defaults. ```{r style, fig.cap = "The mtcars fit redrawn with the asymmetry index on a corner tile, the magma palette, and no background.", fig.alt = "Four-by-four association matrix for mtcars with a small coloured tile in the bottom-right corner of each cell and white backgrounds."} st <- janusplot_style(colour_channel = "tile", palette = "magma") st plot(fit, colour_by = "asymmetry", background_by = "none", style = st) ``` The palette applies to the `colour_by` scale. A correlation is symmetric around zero, so a correlation `colour_by` is drawn on a diverging palette (`RdBu` unless another diverging one is named); a sequential palette given with a correlation is replaced by `RdBu`. **Colourblind-safe choices:** - *Sequential:* `viridis` (default), `magma`, `inferno`, `plasma`, `cividis`, `mako`, `rocket`, `YlOrRd`, `YlGnBu`, `Blues`, `Greens`. - *Diverging:* `RdYlBu`, `RdBu`, `PuOr`. **High-contrast but not colourblind-safe:** `turbo`, `Spectral`. ### Width on the page The default sizes are set for a figure one page wide (170 mm). For one journal column (85 mm), draw at most four variables with a fixed text size and without the shape legend and the glossary (the caption that explains the symbols): ```{r narrow, fig.width = 85 / 25.4, fig.height = 90 / 25.4, fig.cap = "The mtcars matrix at one journal column, 85 mm wide, with 6-point text.", fig.alt = "Small four-by-four association matrix for mtcars sized for a single journal column."} narrow <- janusplot_style(text_size = 6, show_shape_legend = FALSE, show_glossary = FALSE) plot(fit, style = narrow) ``` ### One call: `janusplot()` `janusplot()` fits and draws in one call. It takes the view arguments of `plot()` and passes model arguments (`k`, `engine`, `na_action` and the others of `janusplot_fit()`) through `...`: ```{r oneshot, eval = FALSE} janusplot(mtcars[, c("mpg", "hp", "wt", "qsec")], order = "hclust") # is the same figure as plot(janusplot_fit(mtcars[, c("mpg", "hp", "wt", "qsec")]), order = "hclust") ``` Keep the fit when the same data will be drawn more than once. ## The pair The matrix is for finding pairs; `janusplot_pair()` shows one pair at full size. The motorcycle-crash data of `MASS::mcycle` record head acceleration (`accel`) against time after impact (`times`). ```{r pair, eval = has_mass, fig.width = 7, fig.height = 3.6, fig.cap = "The two directions of the mcycle pair: acceleration as a smooth function of time on the left, time as a smooth function of acceleration on the right.", fig.alt = "Two scatterplot panels side by side for the mcycle data, each with a fitted spline and confidence band; the left curve dips and recovers over time, the right curve is a much smoother fit of time on acceleration."} fit_mc <- janusplot_fit(MASS::mcycle) janusplot_pair(fit_mc, x = "times", y = "accel") ``` The left panel is `accel ~ s(times)`, the right `times ~ s(accel)`. Each carries the model formula as its title and a strip with the pair's statistics: `A`, the EDF of that direction, the Pearson correlation, the significance glyph and the number of rows used. The acceleration trace is a wave in time (EDF `r if (has_mass) f1(cell(fit_mc, "accel", "times")$edf)`); time is a far simpler function of acceleration (EDF `r if (has_mass) f1(cell(fit_mc, "times", "accel")$edf)`). `display = "both"` overlays the two curves in one panel. Both variables are standardised on the rows the pair used, and the reverse fit is drawn with its axes swapped, so both curves share one plane: ```{r pair-both, eval = has_mass, fig.width = 4.5, fig.height = 4.5, fig.cap = "Both directions of the mcycle pair in one standardised panel.", fig.alt = "One panel with standardised time on the horizontal axis and standardised acceleration on the vertical axis, with two fitted curves in different colours."} janusplot_pair(fit_mc, x = "times", y = "accel", display = "both") ``` Several pairs are drawn as blocks of two panels: the blocks stack top to bottom, the stacks sit side by side (`nrow` blocks per stack, `ncol` stacks), and the two panels of a pair stay close together. Four pairs of the mtcars fit in two stacks: ```{r pair-grid, fig.width = 7, fig.height = 5, fig.cap = "Four pairs of the mtcars fit, two blocks per stack in two stacks.", fig.alt = "Grid of four pair blocks, two per column, each block holding the two directional fits of one mtcars pair."} janusplot_pair(fit, x = c("hp", "wt", "qsec", "qsec"), y = c("mpg", "mpg", "hp", "wt"), ncol = 2L) ``` ## Marginal against adjusted: `janusplot_compare()` `adjust` adds covariates and random effects to every pairwise model, for example `adjust = ~ species` or `adjust = ~ s(site, bs = "re")`. `janusplot_compare()` sets two fits of the same variables side by side, cell by cell, and reports how each fitted shape moves. ```{r compare, eval = has_penguins, fig.cap = "Penguin traits fitted marginally (grey dashed curves) and adjusted for species (solid curves with bands), points coloured by species. A red border marks a cell whose trend changed: rising, falling or non-monotone.", fig.alt = "Four-by-four matrix for the penguin traits with two curves in every cell, a dashed grey marginal curve and a solid species-adjusted curve, and red borders on the cells whose trend changed."} fit_pen_adj <- janusplot_fit(palmerpenguins::penguins, pen_vars, adjust = ~ species) cmp <- janusplot_compare(fit_pen, fit_pen_adj, labels = c("marginal", "species-adjusted")) cmp plot(cmp, group = "species") ``` ```{r compare-numbers, include = FALSE, eval = has_penguins} n_changed <- sum(cmp$table$shape_changed, na.rm = TRUE) n_trend <- sum(cmp$table$trend_changed, na.rm = TRUE) pen_cor <- stats::cor(palmerpenguins::penguins[, pen_vars], use = "pairwise.complete.obs")["bill_depth_mm", ] ``` The shape label changes in `r if (has_penguins) n_changed` of the `r if (has_penguins) nrow(cmp$table)` directed cells once species is in the model, and its trend (rising, falling or non-monotone) in `r if (has_penguins) n_trend` of them, the cells with a red border. Pooled across species, `bill_depth_mm` correlates negatively with the other three traits (r from `r if (has_penguins) round(min(pen_cor), 2)` to `r if (has_penguins) round(max(pen_cor[pen_vars != "bill_depth_mm"]), 2)`), and Gentoo penguins are both the largest and the shallowest-billed. Adjusted for species, every curve involving `bill_depth_mm` rises. The comparison shows how sensitive each fitted shape is to the adjustment; why a shape moves is a question for the study design. A fit with `adjust` draws no correlation background in `plot()`, `janusplot()` or `janusplot_pair()`: its curves are adjusted, while the correlation is marginal. ## Direction: `janusplot_direction()` An asymmetry index says that the two directions of a pair are not interchangeable; it does not say which variable drives the other. `janusplot_direction()` tests whether one direction, and only one, fits an additive-noise model. Under the additive-noise model $y = f(x) + e$ with $e$ independent of $x$, the model holds in at most one direction for all but a small set of joint distributions. The function fits both directions and tests each set of residuals for independence from its predictor with a kernel (HSIC) permutation test. ```{r direction, eval = has_kernr && has_mass} janusplot_direction(fit_mc, x = "times", y = "accel", seed = 2026L) ``` A `"forward"` or `"reverse"` verdict says that the residuals are independent in that direction only, which is the condition the additive-noise model needs; it is not a proof of causation, since the model assumes no hidden common cause, no feedback and additive noise. Every other pattern returns `"undecided"`: residuals independent in both directions, in neither, or a linear relationship with Gaussian variables, where the model cannot tell the directions apart. For the motorcycle pair the verdict is `"undecided"`: the residual spread changes with time, so the residuals depend on the predictor in both directions. The test is run on the marginal pair; a conditional test under `adjust` is planned for 1.1. ## The table: `summary()` `summary()` returns one row per directed cell (one model, `y ~ s(x)` or `x ~ s(y)`), with every number the plots draw: EDF, raw F-test p-values and the multiplicity-adjusted ones (`janusplot_fit(p_adjust = "BH")` or `"holm"`; equal to the raw values by default), deviance explained, the asymmetry index and its signed version (`directional_complexity`, positive in the cell whose smooth is the more complex), the three correlations, the shape indices, counts and labels, the k-check columns and any error. ```{r table} tab <- summary(fit_het) tab[, c("var_y", "var_x", "edf", "association_p", "asymmetry_index", "shape_category")] ``` Each row is the model `var_y ~ s(var_x)`. The pair-level fields, the prediction grids and the derivative curves are in `fit$pairs`; the models themselves are kept with `janusplot_fit(keep_fits = TRUE)`. ## Handling missing data ```{r missing, fig.cap = "airquality matrix under pairwise NA handling: each cell is fitted from the rows complete for that pair, so the cells not touching Ozone or Solar.R use all 153 rows.", fig.alt = "Four-by-four association matrix for airquality's Ozone, Solar.R, Wind and Temp, fitted with pairwise-complete rows."} # airquality has missing values in Ozone and Solar.R fit_aq <- janusplot_fit(airquality[, c("Ozone", "Solar.R", "Wind", "Temp")], na_action = "pairwise") plot(fit_aq, annotations = c("n", "signif")) ``` `na_action = "pairwise"` (the default) uses all rows for which *that* pair is complete; `"complete"` restricts to rows complete across every variable (listwise deletion). Here the `Wind`-`Temp` cells are fitted on all 153 rows, while a cell involving `Ozone` or `Solar.R` drops only the rows missing on one of its two variables: more data per cell, at the cost of cells fitted on slightly different samples. The `"n"` annotation prints the rows each cell used. ## Shape metrics explained Every fitted smooth is summarised by two continuous indices and two discrete counts. These drive the 27-category classifier and appear as columns of `summary(fit)` and as fields of each entry of `fit$pairs`. Let `f(x)` be the fitted smooth on a grid of `n_grid` equally spaced points across the predictor range (200 by default; 100 for a fit made with `derivatives = integer()`), with `f'` and `f''` the numerical first and second derivatives. Let `w(x)` be the empirical density of the predictor on the same grid, normalised to `sum(w) = 1`. - **`monotonicity_index`** (paper symbol `M`): `M = sum(w * f') / sum(w * |f'|) in [-1, 1]` `+1` means strictly increasing, `-1` strictly decreasing, `0` a non-monotone curve (bowl, dome, wave). - **`convexity_index`** (paper symbol `C`): `C = sum(w * f'') / sum(w * |f''|) in [-1, 1]` `+1` means globally convex (bowl-up), `-1` globally concave (bowl-down), `0` inflection-dominated (S-curve, sine, flat). Both indices are density-weighted so they describe the smooth *where the data actually live*, not extrapolated tails, and are invariant under a **positive** affine rescaling of `y`: replacing `y` with `a * y + b` for `a > 0` leaves `M` and `C` unchanged. For `a < 0` both `f'` and `f''` flip sign, so `M` and `C` flip sign too -- a monotone-increasing relationship read off `y` reads as monotone-decreasing off `-y`, as it should. - **`n_turning_points`** -- count of interior extrema (sign changes of `f'`), robust to noise via lobe-mass weighting. - **`n_inflections`** -- count of interior curvature flips (sign changes of `f''`), same robust counting. Together the pair `(n_turning_points, n_inflections)` drives the primary `shape_category` dispatch; `(monotonicity_index, convexity_index)` disambiguate within the monotone `(0, 0)` cell. A curve with one turning point is split by the type of that extremum, read from the sign of `f'` on either side of it: a maximum gives `inverted_u`, `skewed_peak`, `broad_peak` or `rippled_peak` for 0, 1, 2 or 3+ inflections, and a minimum gives the reflected `u_shape`, `skewed_trough`, `broad_trough` or `rippled_trough`. Replacing `y` with `-y` therefore maps each single-extremum category onto its mirror, so the taxonomy is closed under reflection. The full taxonomy with 2-letter codes, archetypes, and thumbnail curves is available from `janusplot_shape_hierarchy()` and is drawn as a legend below the matrix whenever the cells carry the shape code. Tune the thresholds applied to these indices via `janusplot_shape_cutoffs()`. The `shape_borderline` column gives the distance from each label to the nearest cutoff that would change it; small values mark labels a slightly different smooth would move. See the [shape-recognition-sensitivity](shape-recognition-sensitivity.html) vignette for how faithfully the classifier recovers ground-truth shapes across sample-size and noise regimes. ## Derivative views: theoretical justification and applied use Each matrix shows one quantity. `display = "fit"` (default) shows the fitted smooth; `display = "d1"` shows $\hat f'(x)$; `display = "d2"` shows $\hat f''(x)$. A title above the matrix names the mode. `janusplot_fit()` computes both derivatives at fit time (`derivatives = 1:2`) with a pointwise 95% band (`derivative_ci = "pointwise"`), so the three views below are drawn from one fit. Orders beyond two are not offered -- see *Noise amplification* below. ```{r derivs-fit, fig.height = 4.5, fig.cap = "Fitted-smooth (level) view of x against a rippled-monotone y1 = x + sin(3x) and a convex y2 = 0.5x^2.", fig.alt = "Three-by-three fitted-smooth matrix for x, y1 (x plus a sine ripple) and y2 (a quadratic), all against x."} set.seed(2026L) n <- 300L xs <- runif(n, -pi, pi) df <- data.frame( x = xs, y1 = xs + sin(3 * xs) + rnorm(n, sd = 0.15), y2 = 0.5 * xs^2 + rnorm(n, sd = 0.8) ) fit_df <- janusplot_fit(df) plot(fit_df, display = "fit") ``` ```{r derivs-d1, fig.height = 4.5, fig.cap = "First-derivative view of the same three variables: the y1 panel oscillates around a positive baseline (a locally reversing gain despite the globally increasing fit above), while the y2 panel is the straight line f'(x) = x.", fig.alt = "First-derivative panels for x against y1 and y2, showing an oscillating curve for y1 and a straight increasing line for y2."} plot(fit_df, display = "d1") ``` ```{r derivs-d2, fig.height = 4.5, fig.cap = "Second-derivative view: y1's curvature oscillates through zero repeatedly (the sine term), while y2's curvature is flat and positive (a constant second derivative, as expected for a quadratic).", fig.alt = "Second-derivative panels for x against y1 and y2, showing an oscillating curve for y1 and a flat positive line for y2."} plot(fit_df, display = "d2") ``` Simultaneous bands use the Monte Carlo critical multiplier of Simpson (2018); they are chosen at fit time: ```{r derivs-sim, fig.height = 4.5, fig.cap = "First-derivative view with simultaneous 95% Monte Carlo confidence bands: the y1 panel's oscillation crosses zero with a band wide enough to show where a reversal is and is not distinguishable from noise.", fig.alt = "First-derivative panels with simultaneous confidence ribbons for x against y1 and y2."} fit_sim <- janusplot_fit(df, derivative_ci = "simultaneous", derivative_ci_nsim = 2000L) plot(fit_sim, display = "d1") ``` ### What derivatives reveal that the fit hides The fitted smooth $\hat f(x) = \mathbb{E}[y\mid x]$ is a level description. Its derivatives are different statistical objects with their own interpretations: - $\hat f'(x)$ -- the **local rate of change** of $y$ in $x$. Zero crossings localise the turning points of $\hat f$; the sign of $\hat f'$ gives the direction of monotonicity; the magnitude gives the sensitivity at the operating point $x$. In control engineering this is literally the process gain $K(x) = \partial y / \partial u$ that gain-scheduled controllers are built around (Rugh & Shamma, 2000; Leith & Leithead, 2000). In causal analysis of a continuous treatment it is the derivative of the dose--response curve $\mu'(t) = \partial \mathbb{E}[Y(t)] / \partial t$, which Zhang & Chen (2025) argue is often *the* treatment-effect object of interest, not the curve itself. - $\hat f''(x)$ -- the **local curvature**. Zero crossings localise the inflection points of $\hat f$; a persistently positive second derivative flags accelerating growth, persistently negative flags saturation (diminishing returns). $\hat f''$ is the input to the convexity index $C$ defined earlier in this vignette, so the derivative panel exposes the *local* signal behind that scalar summary. The asymmetric matrix layout sharpens this. `janusplot_fit()` fits both $\hat f_{y\mid x}(x)$ and $\hat f_{x\mid y}(y)$, so the derivative panels of the two cells of a pair answer different questions: the cell in row $y$, column $x$ is "how steeply does $y$ respond to a nudge in $x$ at this operating point" (forward gain); the cell in row $x$, column $y$ is "how steeply must $x$ change to induce a unit change in $y$" (inverse sensitivity). For an asymmetric process these do not transpose into each other, and the directional asymmetry is a diagnostic the symmetric correlation matrix cannot expose (Janzing & Schölkopf, 2010). ### Estimation -- the LP matrix Let $X_p = X_p(\mathbf{x}_g)$ denote the design (linear predictor) matrix of the fitted GAM evaluated on the plotting grid $\mathbf{x}_g$, obtained from `predict(gam_fit, newdata = ..., type = "lpmatrix")` (Wood, 2017, §7.2.4). With penalised posterior mean $\hat{\boldsymbol\beta} = \mathtt{coef(gam\_fit)}$ and posterior covariance $V_p = \mathtt{gam\_fit\$Vp}$, we construct a finite-difference operator $D^{(k)}$ on the *rows* of $X_p$ (central differences in the interior, second-order forward / backward stencils at the endpoints) and read off $$ \hat f^{(k)}(x_i) = \bigl[D^{(k)} \hat{\boldsymbol\beta}\bigr]_i, \qquad \widehat{\mathrm{Var}}\bigl(\hat f^{(k)}(x_i)\bigr) = \bigl[D^{(k)} V_p \bigl(D^{(k)}\bigr)^{\!\top}\bigr]_{ii}. $$ Pointwise $95\%$ intervals are $\hat f^{(k)}(x) \pm 1.96\,\sqrt{\cdot}$. This is the standard Wood (2017) construction, and is what `gratia::derivatives()` implements in its default mode (Simpson, 2014; Simpson, 2018). Columns of $X_p$ corresponding to `adjust` terms held at typical values contribute identical rows across the grid, so their finite differences are zero and they drop out of both $\hat f^{(k)}$ and its variance -- the derivative in the panel is therefore the derivative of the *partial smooth actually shown in the fit panel*, as expected. For simultaneous intervals over the full grid (a stricter question than pointwise, and what you want for formal feature localisation), `janusplot()` implements the Monte Carlo construction of Ruppert, Wand & Carroll (2003, §6.5), popularised for GAMs by Simpson (2018): draw $\tilde{\boldsymbol\beta}_b \sim \mathcal{N}(\hat{\boldsymbol\beta}, V_p)$ for $b = 1, \ldots, B$ and take the $(1-\alpha)$ quantile of $\max_i |D^{(k)}_i (\tilde{\boldsymbol\beta}_b - \hat{\boldsymbol\beta})| / \mathtt{se}_i$ across the plotting grid as a critical multiplier $c_\alpha$ on the pointwise SE, so the simultaneous band is $\hat f^{(k)}(x) \pm c_\alpha\,\mathtt{se}(x)$. Choose it with `janusplot_fit(derivative_ci = "simultaneous")`; the default is the pointwise band, and `derivative_ci = "none"` stores no band. The implementation uses $B = 1000$ (see `derivative_ci_nsim`); Simpson (2018) uses $10\,000$, which is affordable when a tighter quantile estimate is needed. ### Noise amplification and why we cap at $k = 2$ Finite differencing of raw data amplifies noise; penalised splines do not eliminate that amplification, they trade it against bias via the REML-selected smoothing parameter. `mgcv`'s default $m = 2$ thin-plate penalty is an L2 penalty on $\int (f'')^2$, which controls $\hat f'$ pointwise but leaves $\hat f^{(3)}$ unpenalised and bounded only by the basis rank $k$ (Wood, 2017, §5.3; Eilers & Marx, 1996) -- this is the actual mechanism behind the $k \ge 3$ refusal below, not a general claim that third derivatives are unusable. Consequently $\hat f^{(3)}$ is the derivative most exposed to noise amplification at ordinary sample sizes and moderate $k$, and so janusplot refuses $k \ge 3$ by design. For a domain-specific need of a higher-order derivative, specify a matching-order P-spline penalty explicitly (Eilers, Marx & Durbán, 2015) and differentiate the models kept by `janusplot_fit(keep_fits = TRUE)`. ### Applied use: gain estimation and dose--response Two strands in which the asymmetric derivative view is not a cosmetic add-on but the analytical primitive the practitioner actually wants. - **Process-gain scheduling.** In adaptive and gain-scheduled control, the controller is indexed by the local process gain $K(x) = \partial y / \partial u$ (Rugh & Shamma, 2000). For a steady-state input-output dataset, $\hat f_{y\mid u}'(u)$ is a direct data-driven estimate of $K(u)$, and its simultaneous CI tells the engineer whether the local gain is distinguishable from a reference gain over an operating envelope. The inverse panel $\hat f_{u\mid y}'(y)$ is the feedforward-linearisation sensitivity; a large divergence between the two panels flags that a naive inverse controller will under-perform (Korda & Mezić, 2018). The matrix view makes a fleet of such pairs inspectable at once. - **Derivative of the dose--response curve as the causal estimand.** For a continuous treatment $T$ with unconfoundedness, the dose--response $\mu(t) = \mathbb{E}[Y(t)]$ and its derivative $\mu'(t)$ are both estimable, and recent work (Zhang & Chen, 2025) argues $\mu'(t)$ is often the more directly interpretable quantity -- it answers "how much does the expected outcome change per unit shift in treatment at this dose?" This is structurally the same estimand as the process gain above; the asymmetric-matrix derivative panel delivers both forward and reverse-conditioned derivative curves in the same frame, which is a direct diagnostic for Simpson's-paradox-style conditioning reversals (visible, for example, in the penguins `bill_depth_mm` $\times$ `body_mass_g` pair once `species` is adjusted for). ### References cited in this section Eilers, P. H. C., & Marx, B. D. (1996). Flexible smoothing with B-splines and penalties. *Statistical Science*, **11**(2), 89--121. Eilers, P. H. C., Marx, B. D., & Durbán, M. (2015). Twenty years of P-splines. *SORT*, **39**(2), 149--186. Janzing, D., & Schölkopf, B. (2010). Causal inference using the algorithmic Markov condition. *IEEE Transactions on Information Theory*, **56**(10), 5168--5194. Korda, M., & Mezić, I. (2018). Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control. *Automatica*, **93**, 149--160. Leith, D. J., & Leithead, W. E. (2000). Survey of gain-scheduling analysis and design. *International Journal of Control*, **73**(11), 1001--1025. Rugh, W. J., & Shamma, J. S. (2000). Research on gain scheduling. *Automatica*, **36**(10), 1401--1425. Ruppert, D., Wand, M. P., & Carroll, R. J. (2003). *Semiparametric Regression*. Cambridge University Press. Simpson, G. L. (2014). Simultaneous confidence intervals for derivatives of smooth terms in a GAM. *From the Bottom of the Heap* (blog post). Simpson, G. L. (2018). Modelling palaeoecological time series using generalised additive models. *Frontiers in Ecology and Evolution*, **6**, 149. Wood, S. N. (2017). *Generalized Additive Models: An Introduction with R* (2nd ed.). Chapman and Hall/CRC. Zhang, Y., & Chen, Y.-C. (2025). Doubly robust inference on causal derivative effects for continuous treatments. arXiv preprint [2501.06969]. ## Limitations - Pairwise view: `adjust` puts the same covariates in every cell, but each cell is still one smooth of one variable on another. Complement it with a multivariate model. - EDF depends on basis dimension `k`; defaults are sensible but domain-specific tuning is encouraged. - The asymmetry index should not be interpreted causally without strong assumptions. - `monotonicity_index` and `convexity_index` are scale-invariant in `y` but sensitive to the predictor-density weighting -- they describe the smooth on the observed support of `x`, not outside it. - A matrix shows one quantity (fit, d1 or d2). To compare the fit with its derivatives, call `plot()` on the same fit once per `display`; nothing is refitted. - Derivative panels carry pointwise 95% bands by default, valid at each point separately. For feature localisation over the whole range, fit with `derivative_ci = "simultaneous"` (Simpson, 2018). - The shape metrics (`M`, `C`, turning and inflection counts) are computed on the `n_grid` grid, 200 points by default. A fit made with `derivatives = integer()` uses 100 points, which can shift these values slightly and move a label that sits near a cutoff. ## Citation ```{r cite} citation("janusplot") ``` .