[HN Gopher] Show HN: TabPFN v2 - A SOTA foundation model for sma...
___________________________________________________________________
Show HN: TabPFN v2 - A SOTA foundation model for small tabular data
I am excited to announce the release of TabPFN v2, a tabular
foundation model that delivers state-of-the-art predictions on
small datasets in just 2.8 seconds for classification and 4.8
seconds for regression compared to strong baselines tuned for 4
hours. Published in Nature, this model outperforms traditional
methods on datasets with up to 10,000 samples and 500 features.
The model is available under an open license: a derivative of the
Apache 2 license with a single modification, adding an enhanced
attribution requirement inspired by the Llama 3 license:
https://github.com/PriorLabs/tabpfn. You can also try it via API:
https://github.com/PriorLabs/tabpfn-client TabPFN v2 is trained on
130 million synthetic tabular prediction datasets to perform in-
context learning and output a predictive distribution for the test
data points. Each dataset acts as one meta-datapoint to train the
TabPFN weights with SGD. As a foundation model, TabPFN allows for
fine-tuning, density estimation and data generation. Compared to
TabPFN v1, v2 now natively supports categorical features and
missing values. TabPFN v2 performs just as well on datasets with or
without these. It also handles outliers and uninformative features
naturally, problems that often throw off standard neural nets.
TabPFN v2 performs as well with half the data as the next best
baseline (CatBoost) with all the data. We also compared TabPFN to
the SOTA AutoML system AutoGluon 1.0. Standard TabPFN already
outperforms AutoGluon on classification and ties on regression, but
ensembling multiple TabPFNs in TabPFN v2 (PHE) is even better.
There are some limitations: TabPFN v2 is very fast to train and
does not require hyperparameter tuning, but inference is slow. The
model is also only designed for datasets up to 10k data points and
500 features. While it may perform well on larger datasets, it
hasn't been our focus. We're actively working on removing these
limitations and intend to release new versions of TabPFN that can
handle larger datasets, have faster inference and perform in
additional predictive settings such as time-series and recommender
systems. We would love for you to try out TabPFN v2 and give us
your feedback!
Author : onasta
Score : 135 points
Date : 2025-01-09 16:38 UTC (1 days ago)
(HTM) web link (www.nature.com)
(TXT) w3m dump (www.nature.com)
| OutOfHere wrote:
| Related repo: https://github.com/liam-sbhoo/tabpfn-time-series
| dill_1 wrote:
| Wow! Didn't expect the models to do so well on time series as
| well, will try this out.
| instanceofme wrote:
| Related: CARTE-AI, which can also deal with multiple tables.
|
| https://soda-inria.github.io/carte/
| https://arxiv.org/pdf/2402.16785
|
| The paper includes a comparison to TabPFN v1 (among others),
| noting the lack of categorical & missing values handling which v2
| now seems to have. Would be curious to see an updated comparison.
| onasta wrote:
| TabPFN is better on numerical data since v1 (see figure 6 in
| the CARTE paper). CARTE's main strength in on text features,
| which are now also supported for TabPFN v2 API version
| (https://github.com/PriorLabs/tabpfn-client). We compared this
| to CARTE and found our model to be generally quite better, and
| much faster. CARTE multi-table approach is also very
| interesting, and we want to tackle this setting in the future.
| ggnore7452 wrote:
| anyone tried this? is this actually overall better than
| xgboost/catboost?
| westurner wrote:
| Benchmark of tabpfn<2 compared to xgboost, lightgbm, and
| catboost: https://x.com/FrankRHutter/status/1583410845307977733
| .. https://news.ycombinator.com/item?id=33486914
| enigmaa99 wrote:
| Yes it actually is but the limitations of rows and features
| could be a hindrance.
| bbstats wrote:
| looks amazing - finally, DL that beats a tuned catboost?
| _giorgio_ wrote:
| It's probably the same model with the same limitations, released
| nearly two years ago?
|
| https://arxiv.org/abs/2207.01848
| ersiees wrote:
| No, it is *much* stronger, a different architecture and scales
| to 10x the number of examples. It can also do regression now,
| and handle categorical features. Please, have a quick look at
| the abstract before making such claims.
| onasta wrote:
| There have been a ton of improvements! Much better performance
| overall, way larger data size limit (1K-->10K rows, 100-->500
| features), regression support, native categorical data and
| missing values handling, much better support for uninformative
| or outlier features etc.
| enigmaa99 wrote:
| I tried this on a few CARTE datasets and it works surprisingly
| better!! Woahhh
| peepeepoopoo99 wrote:
| How can you train a tabular foundation model when the tabular
| features themselves are inherently domain-specific? Is there some
| kind of preprocessing step beforehand to match the inference time
| features with their closest analogues in the training set?
| ersiees wrote:
| Yes, there are normalizations applied before the features are
| fed to the neural network. Additionally, the neural network is
| trained on a very diverse set of artificial datasets.
| hooloovoo_zoo wrote:
| Were your benchmark methods tuned per dataset or across datasets?
| ersiees wrote:
| Tuned per dataset
| noahho wrote:
| Up to 4 hrs of tuning per dataset / split (10-fold CV)
| gcr wrote:
| Thanks for such a cool project! It's immediately apparent how to
| use it and I appreciate the brief examples.
|
| Quick question: In the breast cancer example from the README,
| simple support vector machines from sklearn (the first thing i
| tried to compare baseline performance, incidentally) seem to
| outperform TabPFN. Is this expected? I know it's a baseline to
| demonstrate ease of use rather than SOTA performance, but I am
| curious. # (TabPFN) In [13]: print("ROC
| AUC:", roc_auc_score(y_test, prediction_probabilities[:, 1]))
| ROC AUC: 0.996299494264216 # (LinearSVC) In
| [27]: from sklearn.svm import LinearSVC In [28]:
| clf=LinearSVC(C=0.01).fit(X_train, y_train) In
| [29]: roc_auc_score(y_test, clf.decision_function(X_test))
| Out[29]: 0.997532996176144
| noahho wrote:
| Author here! The breast cancer dataset is simple and heavily
| saturated, so small differences between methods are expected.
| As you say, single-use examples can be noisy due to randomness
| in how the data is randomly split into training and testing
| sets especially for a saturated dataset like this one. Cross-
| validation reduces this variance by averaging over multiple
| splits. I just ran this below: TabPFN mean ROC
| AUC: 0.9973 SVM mean ROC AUC: 0.9903
| TabPFN per split: [0.99737963 0.99639699 0.99966931 0.99338624
| 0.99966465] SVM per split: [0.99312152 0.98788077
| 0.99603175 0.98313492 0.99128102] from
| sklearn.model_selection import cross_val_score from
| tabpfn import TabPFNClassifier from sklearn.datasets
| import load_breast_cancer from sklearn.svm import
| LinearSVC import numpy as np data =
| load_breast_cancer() X, y = data.data, data.target
| # TabPFN tabpfn_clf = TabPFNClassifier()
| tabpfn_scores = cross_val_score(tabpfn_clf, X, y, cv=5,
| scoring='roc_auc') print("TabPFN per split:",
| tabpfn_scores) print("TabPFN mean ROC AUC:",
| np.mean(tabpfn_scores)) # SVM svm_clf =
| LinearSVC(C=0.01) svm_scores = cross_val_score(svm_clf,
| X, y, cv=5, scoring='roc_auc') print("SVM per
| split:", svm_scores) print("SVM mean ROC AUC:",
| np.mean(svm_scores))
|
| It's hard to communicate this properly, we should probably make
| sure to have a favourable example ready, but just included the
| simplest one!
| gcr wrote:
| thanks, this is helpful!
|
| I certainly appreciate how the example in the README makes it
| instantly apparent how to use the code.
| nickpsecurity wrote:
| A while back, I was looking for a project amateurs could do for
| experimenting with Transformer alternatives and optimization
| algorithms. My concept was grabbing objective, test functions
| from the literature, making custom ones based on realistic data,
| and layering them together based on real-world depth. Then,
| training various approaches on them using consumer GPU's or spot
| instances of high-end GPU's.
|
| What I read in this paper blew that idea out the water! I mean,
| it's still doable but you've far exceeded it.
|
| I love that you covered many types of structures, used 8x
| consumer GPU's more like OSS folks do (widely-accessible
| pretraining), claim no copyright infringement for pretraining,
| and use enough techniques in ML that people can enjoy Googling
| stuff for days.
|
| I do have some questions about what I might have overlooked in
| the paper.
|
| 1. Is the training data and code available to reproduce the
| model? And iteratively improve its architectural decisions?
|
| 2. Most authors claiming their data was legal or open were
| actually committing copyright infringement. Your method might
| dodge that if users generate their own synthetic data using
| methods they can verify aren't themselves encumbered. Is that
| code available under open licensing? If not, would you offer it
| for a fee for companies or free for researchers?
|
| 3. What specific, common uses could amateurs try that would
| display the model's ability in a business setting? (Both to drive
| more research or build products on the model.)
|
| I thank you for your time.
| ersiees wrote:
| Author here!
|
| Thanks :)
|
| 1. Only for the first version, not for this version. I am
| sorry! 2. Yeah ours is guaranteed ok, as we wrote code to
| generate it basically just from plain torch ops. The code to
| run inference is available, just not the training code and data
| generation. 3. We have put it to work on time series data,
| which is very business relevant for example
| https://github.com/liam-sbhoo/tabpfn-time-series, and we have a
| table in the Appendix with all datasets we evaluate on in our
| main analysis to give you some ideas for possible datasets.
| tmostak wrote:
| This looks amazing!
|
| Just looking through the code a bit, it seems that the model both
| supports a (custom) attention mechanism between features and
| between rows (code uses the term items)? If so, does the
| attention between rows help improve accuracy significantly?
|
| Generally, for standard regression and classification use cases,
| rows (observations) are seen to be independent, but I'm guessing
| cross-row attention might help the model see the gestalt of the
| data in some way that improves accuracy even when the
| independence assumption holds?
| dist-epoch wrote:
| Speculating, cross-row might give you information where you are
| in that row distribution.
| ersiees wrote:
| Author here: The new introduction of attention between features
| did make a big impact compared to the first variant of TabPFN.
| The old model handled every feature like it was completely
| different to be feature 5 vs 15, but actually features are
| typically more-or-less permutation invariant. So the logic is
| similar to why a CNN is better for images than an MLP.
| patcon wrote:
| Neat! Might this even be useful to impute missing data for a
| sparse network of votes, for a system like this (pol.is) whose
| goal is to do dimensional reduction and visualise the opinion
| space of divisive social topics:
| https://gwern.net/doc/sociology/2021-small.pdf
|
| 200 voters on 50 statements would fall within the 10,000 sample
| threshold. This is well within the bounds of some existing
| conversations with open data, so it could be tested... Potential
| values on each statement are agree/disagree/pass (+1/-1/0)
|
| https://github.com/compdemocracy/openData/blob/master/brexit...
|
| https://github.com/compdemocracy/openData/blob/master/brexit...
| noahho wrote:
| Looks like a great use case! We have a method specifically for
| imputation in the tabpfn-extensions package
| (https://github.com/PriorLabs/tabpfn-
| extensions/blob/dbc3f5da...). It needs some cleaning up before
| I want to highlight in the notebooks and docs.
| storyweaver2 wrote:
| Did you compare the performance with o1 or Claude 3.5 Sonnet?
| noahho wrote:
| Author here! The fundamental challenge is that LLMs like O1 and
| Claude 3.5 simply aren't built for the unique structures of
| tabular data. When processing tables through LLMs, the
| inefficiencies quickly become apparent - tokenizing a 10,000 x
| 100 table as a sequence and numerical values as tokens creates
| massive inefficiencies.
|
| There's some interesting work on using LLMs for tabular data
| (TabLLM:
| https://proceedings.mlr.press/v206/hegselmann23a.html), but
| this only works for datasets with tens of samples rather than
| the thousands of rows needed in real-world applications.
|
| What o1 and other LLMs typically do is wrap around existing
| tabular tools like XGBoost or scikit-learn. While this works,
| they're ultimately constrained by these tools' limitations.
| We're taking a fundamentally different approach - building
| foundation models that natively understand tabular
| relationships and patterns. Our approach combines the benefits
| of foundation models with architectures specifically designed
| for tabular data structures.
| pplonski86 wrote:
| Amazing results! Beating AutoML with single model is not easy :)
|
| Could you please explain like I'm five what is doing a trick? You
| have model pre-trained on large set of small datasets and you
| leverage it to boost performance?
|
| Training is fast, few seconds, but what is time needed to compute
| predictions?
|
| How large is the model?
| ersiees wrote:
| To put it very simply, the trick is that while the others train
| a new model for each problem, TabPFN is pre-trained to handle
| any kind of problem on the fly.
|
| To draw a parallel to NLP: previously people trained a neural
| network for each kind of text classification they wanted to do,
| but then LLMs came around that pre-trained to learn to perform
| new tasks on the fly. Similarly, TabPFN learns to do new tasks
| on the fly just from the context (dataset) given.
|
| Training and prediction in these models is by default one and
| the same, similar to how the prediction of the next token in an
| LLM is not split into learning from context and then doing the
| actual prediction. There is a way to split this even up,
| though, then the predictions, I believe, take something like
| 1/10s for medium-sized datasets.
| Dowwie wrote:
| Congrats on your release. What is the best way to share feedback?
| I would like to share with you what I believe to be a challenging
| problem that this may help with.
| noahho wrote:
| thanks a ton! If it's public please share in the Discord
| https://discord.com/channels/1285598202732482621/ > #use-cases
| (just created!), if not, mail me at noah@priorlabs.ai
| mlepath wrote:
| Great work!
|
| Do you see any artifacts from having trained on synthetic data?
| Is there a natural benchmark dataset (real tables in the wild)?
|
| In my experience synthetic data can only take you so far, it has
| all the quirk the dataset creator can think of but the real value
| is usually in patterns they cannot. Vision took a huge leap
| forward with ImageNet dataset release
___________________________________________________________________
(page generated 2025-01-10 23:02 UTC)