Bio-LoRA and protein-model evaluation

The manuscript

I am a co-first author of Bio-LoRA: Biological Priors as Inductive Bias for Low-Rank Adaptation, an unpublished research manuscript. It asks whether biological information can provide a useful starting point for adapting a pretrained protein language model.

The manuscript’s method uses biological matrices and ESM-2 embeddings to construct an SVD-derived initialization for LoRA’s down-projection; its up-projection starts at zero. The manuscript covers mutation effects, protein stability, and signal-peptide prediction. I developed the signal-peptide training and evaluation code.

Manuscript record

D. S. Abubaker, M. Faheem, A. S. Mukherjee, S. Othman, S. Sifar, S. Sthapit, Y. Wang, M. Shao, and M. Shafique. Bio-LoRA: Biological Priors as Inductive Bias for Low-Rank Adaptation. Unpublished manuscript, 2026. Co-first author.

Signal-peptide evaluation

I extended the evaluation framework to signal-peptide classification and cleavage-site prediction, including task adapters, model heads, and metrics. I built experiment configurations and ran comparisons on NYU Abu Dhabi’s HPC cluster across ESM-2 150M, 650M, and 3B models.

Follow-up experiments projected residue-level biological features into the model’s hidden states alongside LoRA. This feature injection is separate from the manuscript’s LoRA initialization method.

I compared the methods with vanilla LoRA and full fine-tuning, investigated initialization and prior controls, and recorded speed and memory behavior under different training protocols. The follow-up included three-seed confirmation runs.

Results across model sizes

The saved SP5 benchmark comparison used ESM-2 650M, seeds 11, 22, and 33, and 1,231 ground-truth signal-peptide-positive examples. The metric is exact cleavage-site match on that subset.

At 650M, BLOSUM62 feature injection had the highest mean exact cleavage-site match in this comparison. Full fine-tuning had the highest mean at 150M and 3B. The figure below includes all six methods in the saved 650M comparison.

The follow-up also changed which configurations I prioritized. Removing weight decay or reducing the initial feature-injection scale looked better in the first seed. Across three seeds, their mean exact cleavage-site match fell below the original BLOSUM62 setting: 84.92% without weight decay and 85.30% with the smaller initial scale, compared with 86.08% for the original setting.