← Publications / Tabular Foundation Models & Distillation
Exploring Fine-Tuning for Tabular Foundation Models
Aditya Tanna, Pratinav Seth, Mohamed Bouadi, Vinay Kumar Sankarapu
Abstract
Tabular Foundation Models (TFMs) have recently shown strong in-context learning capabilities on structured data, achieving zero-shot performance comparable to traditional machine learning methods. We find that zero-shot TFMs already achieve strong performance, while the benefits of fine-tuning are highly model and data-dependent. Meta-learning and PEFT provide moderate gains under specific conditions, whereas full supervised fine-tuning (SFT) often reduces accuracy or calibration quality. This work presents the first comprehensive study of fine-tuning in TFMs across benchmarks including TALENT, OpenML-CC18, and TabZilla. We compare Zero-Shot, Meta-Learning, Supervised (SFT), and parameter-efficient (PEFT) approaches, analyzing how dataset factors such as imbalance, size, and dimensionality affect outcomes. Our findings cover performance, calibration, and fairness, offering practical guidelines on when fine-tuning is most beneficial and its limitations.
Cite
@inproceedings{tanna2026exploring,
title = {{Exploring Fine-Tuning for Tabular Foundation Models}},
author = {Aditya Tanna and Pratinav Seth and Mohamed Bouadi and Vinay Kumar Sankarapu},
booktitle = {WWW 2026},
year = {2026},
eprint = {2601.09654},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2601.09654}
}
