--- title: "LiblineaR 2.50-1: New Features, Correctness, and Performance" author: "Thibault Helleputte" date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{LiblineaR 2.50-1: New Features, Correctness, and Performance} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>") ``` ```{r setup} library(LiblineaR) ``` This version of `LiblineaR` updates its bundled copy of the LIBLINEAR C/C++ library from v2.1 (2015) to v2.50 (2024) -- the first upgrade of the bundled library in a decade, since `LiblineaR` 2.10-0 in July 2016. This vignette covers two things: the new capabilities this brings at the R level, and evidence that the upgrade is safe to rely on, along with what it changes in practice. ## New in this release Three new capabilities are available, on top of everything `LiblineaR` already supported. ### One-class SVM: outlier and novelty detection `type=21` fits a one-class SVM. Unlike every other `type`, it is **unsupervised** -- it learns what "normal" data looks like from `data` alone. `target` is still required as an argument (its length must match the number of rows of `data`) but its *values* are ignored. `bias` must be `<= 0` (this solver has no bias-term concept), and `nu` (a number in (0, 1), default 0.5) is an upper bound on the fraction of training points that may be treated as outliers. `predict()` returns `1` for inliers and `-1` for outliers; `decisionValues=TRUE` additionally returns the raw signed distance to the decision boundary (positive means inlier -- useful if you want to rank points by how anomalous they are, rather than just a yes/no call). ```{r oneclass} x <- iris[, 1:4] y <- iris[, 5] m_oc <- LiblineaR(x, y, type = 21, bias = -1, nu = 0.1) p_oc <- predict(m_oc, newx = x, decisionValues = TRUE) table(p_oc$predictions) head(p_oc$decisionValues) ``` ### `regularizeBias`: excluding the intercept from regularization By default, `LiblineaR`'s bias term (the intercept) is penalized by the L1/L2 regularization exactly like every other coefficient. Setting `regularizeBias=FALSE` excludes it, so it is fit freely. This is only valid combined with `bias=1` exactly, and only for `type` 0, 2, 5, 6 or 11; other combinations raise a clear error. Most users should leave this at its default (`TRUE`) -- an unregularized intercept is a specialized choice, useful mainly when you have strong reasons to believe the regularization penalty should apply only to the feature weights, not to the overall offset. ```{r regularizebias} m_default <- LiblineaR(x, y, type = 0, bias = 1) m_unreg <- LiblineaR(x, y, type = 0, bias = 1, regularizeBias = FALSE) ``` ### `findC` now supports SVR (`type=11`) For perspective: `heuristicC(data)` and `findC=TRUE` both help pick a good `cost`, but they work very differently. `heuristicC()` is a fast, closed-form formula -- it inspects `data` once and returns a single suggested `cost`, with no training involved at all. `findC=TRUE` is an actual empirical search -- it repeatedly trains and cross-validates across a doubling sequence of `cost` values and returns whichever one empirically scored best. `heuristicC()` is a cheap starting guess; `findC` is expensive but data-driven tuning, and the two compose naturally: `useInitC=TRUE` (the default) uses the `cost` you pass in -- often `heuristicC()`'s own suggestion -- as `findC`'s starting point for the search. `findC=TRUE` previously only searched for a good `cost` with `type=0` and `type=2`. It now also supports `type=11` (L2-regularized L2-loss SVR), additionally searching `svr_eps` at the same time. For this `type` only, `findC` returns a named list `list(C=..., svr_eps=...)` instead of a plain number, since there are now two values to report; `type` 0 and 2 are unaffected and still return a plain number. ```{r findc-svr} xr <- as.matrix(iris[, 1:3]) yr <- iris[, 4] best <- LiblineaR(xr, yr, type = 11, findC = TRUE, cross = 5) best m_svr <- LiblineaR(xr, yr, type = 11, cost = best$C, svr_eps = best$svr_eps) ``` ## Is the upgrade safe to rely on? Beyond `LiblineaR`'s own regression test suite, this section reports an independent check: models were trained with the **pristine, unmodified LIBLINEAR command-line tools** (`train`/`predict`, compiled directly from upstream source, no R or `LiblineaR` code involved at all) and compared against `LiblineaR`'s own output, at both the old (v2.10) and new (v2.50) LIBLINEAR version. ### Method Four engines were compared: the LIBLINEAR command-line tools at v2.10 and v2.50, and the `LiblineaR` R package before and after this upgrade. All four were trained on **byte-identical data** (the same train/test split of the same matrix) with matching hyperparameters (`cost`, `bias`, `epsilon`, `svr_eps`). Four datasets were used: two synthetic classification problems (2-class and 3-class, 600 rows each), one regression problem (600 rows), and one larger problem (8000 rows, 300 columns) reserved for timing. **A methodological caveat, confirmed empirically before trusting any of the numbers below:** LIBLINEAR's dual coordinate-descent solvers (and the L1-regularized solvers) shuffle their working set using a random number generator. `LiblineaR` routes this through R's own RNG (`set.seed()`-controlled), while the plain command-line tools use C's own unseeded `rand()`. **For these solver types, coefficients are not expected to match between `LiblineaR` and the command-line tools, even at the identical LIBLINEAR version** -- they are two different, independent, equally valid solutions to the same problem. What *is* expected, and checked below, is that predictions agree almost exactly. Primal, non-randomized solvers (logistic regression and L2-loss SVM/SVR, `type` 0, 2, 11) use no randomness at all, so their coefficients are expected to match the command-line tools almost to floating-point precision -- and are the real fidelity check. ### Result 1: does `LiblineaR` still correctly wrap LIBLINEAR? Maximum absolute coefficient difference between `LiblineaR` and the pristine command-line tools, same LIBLINEAR version, across the 3 correctness datasets (28 train/predict comparisons per version): ```{r fidelity-table} fidelity <- data.frame( type = c(0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 11L, 12L, 13L), solver = c("L2R_LR (primal)", "L2R_L2LOSS_SVC_DUAL", "L2R_L2LOSS_SVC (primal)", "L2R_L1LOSS_SVC_DUAL", "MCSVM_CS", "L1R_L2LOSS_SVC", "L1R_LR", "L2R_LR_DUAL", "L2R_L2LOSS_SVR (primal)", "L2R_L2LOSS_SVR_DUAL", "L2R_L1LOSS_SVR_DUAL"), max_abs_diff_v210 = c(1.72e-10, 9.65e-03, 1.98e-10, 3.60e-02, 1.38e-02, 3.93e-01, 6.82e-01, 7.32e-03, 1.84e-10, 7.93e-02, 1.71e-01), max_abs_diff_v250 = c(1.75e-10, 9.65e-03, 1.93e-10, 3.60e-02, 1.38e-02, 3.93e-01, 6.82e-01, 7.32e-03, 1.59e-10, 7.93e-02, 1.71e-01) ) knitr::kable(fidelity, digits = 4, col.names = c("type", "solver", "max|diff| vs CLI (2.10)", "max|diff| vs CLI (2.50)")) ``` Primal types (0, 2, 11) match the pristine command-line tools to **1e-10**, at both versions -- direct evidence that `LiblineaR` correctly passes data through to the same underlying solver, unaffected by the upgrade. Every dual/L1 type shows a larger difference, exactly as the caveat above predicts -- and across all 54 such comparisons (both versions, all 3 datasets), **minimum prediction agreement was 98.75%, with most exactly 100%**, confirming these are two valid solutions to the same problem, not a correctness gap. ### Result 2: what changed for an ordinary user Maximum absolute coefficient difference between `LiblineaR` before and after the upgrade, same 3 datasets: ```{r upgrade-table} upgrade <- data.frame( type = c(0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 11L, 12L, 13L), solver = fidelity$solver, max_abs_diff = c(0.0789, 0, 0.2281, 0, 0, 0, 0, 0, 0.0021, 0, 0) ) knitr::kable(upgrade, digits = 4, col.names = c("type", "solver", "max|diff| (2.10-26 vs. 2.50-1)")) ``` Only the 3 primal Newton-solved types (0, 2, 11) show any drift at all: LIBLINEAR replaced its trust-region Newton solver with a faster-converging line-search variant partway through this version range, refined further by later releases. Every dual/L1 type was **bit-for-bit identical** on these ordinary, well-conditioned datasets -- LIBLINEAR 2.50's lower iteration cap on dual coordinate descent (see Result 3) was never actually reached on data this well-behaved. Predictive performance (accuracy for classification, mean squared error for regression) matched to 4 decimal places for every type on every dataset. ### Result 3: what actually changed under harder conditions LIBLINEAR 2.50 caps its dual coordinate-descent solvers at 300 iterations (down from 1000 in earlier versions) and, for three solver types, automatically switches to the primal solver instead of running out the clock if that cap is reached. Result 2 shows this is invisible on easy-to-fit data. It stops being invisible on a harder problem. On a larger, more weakly-separated dataset (8000 rows, 300 columns): ```{r large-table} large <- data.frame( type = c(0L, 1L, 7L, 11L, 12L), solver = c("L2R_LR (primal)", "L2R_L2LOSS_SVC_DUAL", "L2R_LR_DUAL", "L2R_L2LOSS_SVR (primal)", "L2R_L2LOSS_SVR_DUAL"), metric_old = c(0.9188, 0.9219, 0.9200, 1.0234, 1.0502), metric_new = c(0.9181, 0.9188, 0.9200, 1.0235, 1.0234), time_old_s = c(0.133, 0.632, 1.035, 0.075, 3.400), time_new_s = c(0.147, 0.288, 0.998, 0.086, 1.201), speedup = c(0.90, 2.19, 1.04, 0.87, 2.83) ) knitr::kable(large, digits = 4, col.names = c("type", "solver", "accuracy/MSE (2.10-26)", "accuracy/MSE (2.50-1)", "time 2.10-26 (s)", "time 2.50-1 (s)", "speedup")) ``` Two solver types where the iteration cap is actually reached on this dataset (`type` 1 and `type` 12, both dual) show a **2.2x and 2.8x speedup** respectively -- and `type` 12's test-set mean squared error also *improved* (1.050 → 1.023), because the previous solver spent its full 1000-iteration budget grinding toward a barely-better optimum on a genuinely slow-converging problem, while the current one recognizes the cap, switches solver, and reaches a comparably good (here, slightly better) answer in a fraction of the time. The primal Newton types (0, 11) are essentially a wash in either direction (0.87-0.90x) -- expected, since that solver's algorithm changed but not its overall cost. ## Summary - **New capabilities**: one-class SVM for outlier/novelty detection, optional unregularized intercept, and `findC` support for SVR. - **Correctness**: `LiblineaR` continues to faithfully wrap LIBLINEAR, confirmed against the pristine command-line tools directly, not just against its own prior output. Coefficient-level agreement for every non-randomized solver is at floating-point precision. - **Impact on existing code**: minimal. Only 3 of the 11 previously available solver types show any coefficient drift at all, and it does not measurably affect predictive performance in this study. - **Performance**: dual coordinate-descent solvers on harder-to-converge problems can now be more than twice as fast, sometimes with *better* held-out performance, because LIBLINEAR 2.50 recognizes slow convergence and switches strategy instead of running a fixed iteration budget to completion. **Reproducibility note.** These figures were measured on `r R.version.string`, Linux, using LIBLINEAR upstream source at tags `v210` and `v250` (github.com/cjlin1/liblinear), compiled standalone with no changes. `set.seed()` was fixed identically across `LiblineaR` calls for a given comparison; the command-line tools have no equivalent seed control (see the caveat above). Datasets are synthetic and fixed by seed; absolute numbers will differ on other data, but the qualitative pattern (primal solvers: no timing benefit, but a different algorithm; dual solvers: identical on easy data, faster and possibly more accurate on hard data) is expected to hold generally, since it follows directly from the documented behavior of the underlying LIBLINEAR change, not from anything data-specific. **Authorship note.** The scope and content of this report -- what to test, which comparisons matter, and how to interpret the results -- were designed and directed by the package maintainer. Drafting of this vignette and the accompanying comparison scripts was AI-assisted (Claude, Anthropic). Every figure was produced by actually running the real LIBLINEAR and LiblineaR software, not generated by the AI; all code, results, and conclusions were reviewed and validated by the maintainer before publication. .