When Does Parameter-Efficient Fine-Tuning Beat Full Fine-Tuning in Code-Change Learning?
A systematic study across four PEFT methods and seven code-language models — and PastaK, a probing-guided framework that trains a fraction of the parameters and still wins.
The first systematic comparison of PEFT (LoRA, adapters, prompt tuning) against full-model fine-tuning across seven code LMs on real code-change tasks — plus PastaK, a probing-guided framework that trains only a fraction of the parameters and still beats full fine-tuning.
Why it matters
We examine four prevalent PEFT methods and compare them with full-model fine-tuning across seven PLMs on code-change-related tasks.
Key findings
- On defect prediction, PEFT wins With expert features, all four PEFT methods beat FMFT — AT and LoRA raise F1 by 5.08% and 9.75% over state-of-the-art baselines.
- On commit-message generation, PEFT is at best on par PEFT is at best on par with FMFT; AT and LoRA remain the strongest efficient options. This asymmetry is the paper's central insight.
- Cross-lingual transfer favours PEFT When trained in one language and tested in another, AT and LoRA improve BLEU by up to 4.62% and 7.56% over FMFT.
- PastaK: probe once, tune better PastaK beats full FMFT by +1.48% BLEU, +3.21% METEOR, +1.87% ROUGE-L on commit-message generation while saving 26.26% of training time and 20.65% of memory.
PastaK
- Compares Adapter Tuning, LoRA, Prompt Tuning, and Prefix Tuning against full-model fine-tuning across seven code PLMs.
- Two code-change tasks: Just-In-Time Defect Prediction (JIT-DP) and Commit Message Generation (CMG).
- Layer-wise probing (including two new probing tasks) reveals where PEFT and FMFT each excel.
- PastaK assigns bottom K layers to full fine-tuning and the rest to a PEFT method, read directly from probe data.
What it means for practitioners
For classification-style code-change tasks and cross-lingual/low-resource settings, PEFT gives better accuracy at a fraction of the cost and should replace FMFT; for demanding generative tasks, use AT or LoRA when compute is limited, and consider PastaK to get full-model quality without the full-model bill.
Get the paper & cite it
Official citation: Shuo Liu, Jacky W. Keung, Zhi Jin, Zhen Yang, Fang Liu, Hao Zhang (2026). An Empirical Study of Parameter-Efficient Fine-Tuning in Code Change Learning and Beyond. IEEE Transactions on Software Engineering. DOI: 10.1109/TSE.2025.3637335.