TinyBridge-TriFuse: Lightweight Multi-Expert Fusion for Frozen Vision-Language Models

Mukthikka V; Piyush; Gurpreet Singh

doi:10.55524/ijirem.2026.13.2.18

Abstract

Multimodal large language models (MLLMs) are currently dominated by visual instruction tuning (VIT), where frozen vision and language backbones are bridged by lightweight trainable modules. We curated and downloaded 30 recent papers (2022–2025) in this direction, with a cumulative 4,555 citations as of April 28, 2026, and analyzed architecture and training trends. The strongest recurring pattern is a shift from full-model tuning to parameter-efficient adaptation, token compression, and data selection. Motivated by this, we propose TinyBridge-TriFuse, a frozen-backbone connector family that combines linear pooled-feature alignment, a small MLP expert, and a token-aware bridge expert. We provide a full IEEE-style formulation, a data-efficient training recipe, and real measured experiments. On frozen OpenCLIP ViT-B/32 features for CIFAR-10, TinyBridge-TriFuse reaches 0.9390 test accuracy, exceeding LinearAlign (0.9350) by +0.0040 and improving over zero-shot CLIP by +0.0728. We also report TinyBridge-DynaFuse, a tiny-gate variant (1,035 extra parameters) that improves calibration to 0.0326 ECE.

Keywords

Multimodal Large Language Model, Visual Instruction Tuning, Frozen Backbone, Parameter-Efficient Training, Vision-Language Alignment

References

J.-B. Alayrac et al., “Flamingo: A visual language model for few-shot learning,” arXiv preprint arXiv:2204.14198, 2022. Available from: https://arxiv.org/abs/2204.14198
Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597, 2023. Available from: https://arxiv.org/abs/2301.12597
Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” arXiv preprint arXiv:2304.08485, 2023. Available from: https://arxiv.org/abs/2304.08485
W. Dai et al., “InstructBLIP: Towards general-purpose vision-language models with instruction tuning,” arXiv preprint arXiv:2305.06500, 2023. Available from: https://arxiv.org/abs/2305.06500
D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “MiniGPT-4: Enhancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592, 2023. Available from: https://arxiv.org/abs/2304.10592
P. Gao et al., “LLaMA-adapter V2: Parameter-efficient visual instruction model,” arXiv preprint arXiv:2304.15010, 2023. Available from: https://arxiv.org/abs/2304.15010
Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” arXiv preprint arXiv:2310.03744, 2023. Available from: https://arxiv.org/abs/2310.03744
Awadalla et al., “OpenFlamingo: An open-source framework for training large autoregressive vision-language models,” arXiv preprint arXiv:2308.01390, 2023. Available from: https://arxiv.org/abs/2308.01390
Bai et al., “Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond,” arXiv preprint arXiv:2308.12966, 2023. Available from: https://arxiv.org/abs/2308.12966
P. Wang et al., “Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191, 2024. Available from: https://arxiv.org/abs/2409.12191
Z. Peng et al., “Kosmos-2: Grounding multimodal large language models to the world,” arXiv preprint arXiv:2306.14824, 2023. Available from: https://arxiv.org/abs/2306.14824
Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao, “Shikra: Unleashing multimodal LLM’s referential dialogue magic,” arXiv preprint arXiv:2306.15195, 2023. Available from: https://arxiv.org/abs/2306.15195
Luo, Y. Zhou, T. Ren, S. Chen, X. Sun, and R. Ji, “Cheap and quick: Efficient vision-language instruction tuning for large language models,” arXiv preprint arXiv:2305.15023, 2023. Available from: https://arxiv.org/abs/2305.15023
B. Lin et al., “MoE-LLaVA: Mixture of experts for large vision-language models,” arXiv preprint arXiv:2401.15947, 2024. Available from: https://arxiv.org/abs/2401.15947
B. Zhou et al., “TinyLLaVA: A framework of small-scale large multimodal models,” arXiv preprint arXiv:2402.14289, 2024. Available from: https://arxiv.org/abs/2402.14289
S. Tong et al., “Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs,” arXiv preprint arXiv:2406.16860, 2024. Available from: https://arxiv.org/abs/2406.16860
Li et al., “LLaVA-NeXT-interleave: Tackling multi-image, video, and 3D in large multimodal models,” arXiv preprint arXiv:2407.07895, 2024. Available from: https://arxiv.org/abs/2407.07895
Lu et al., “DeepSeek-VL: Towards real-world vision-language understanding,” arXiv preprint arXiv:2403.05525, 2024. Available from: https://arxiv.org/abs/2403.05525
Bi et al., “LLaVA steering: Visual instruction tuning with 500x fewer parameters through modality linear representation-steering,” arXiv preprint arXiv:2412.12359, 2024. Available from: https://arxiv.org/abs/2412.12359
D. Wang, J. Cui, M. Li, W. Lin, B. Chen, and H. Zhang, “Instruction tuning-free visual token complement for multimodal LLMs,” arXiv preprint arXiv:2408.05019, 2024. Available from: https://arxiv.org/abs/2408.05019
P. Jiao, B. Zhu, J. Chen, C.-W. Ngo, and Y.-G. Jiang, “From holistic to localized: Local enhanced adapters for efficient visual instruction fine-tuning,” arXiv preprint arXiv:2411.12787, 2024. Available from: https://arxiv.org/abs/2411.12787
C. Oh, J. Li, S. Im, and S. Li, “Visual instruction bottleneck tuning,” arXiv preprint arXiv:2505.13946, 2025. Available from: https://arxiv.org/abs/2505.13946
B. Li, Z. Zhang, S. Liu, W. Yu, and X. Wang, “Top-down compression: Revisit efficient vision token projection for visual instruction tuning,” arXiv preprint arXiv:2505.11945, 2025. Available from: https://arxiv.org/abs/2505.11945
Z. Zhou et al., “Learning to instruct for visual instruction tuning,” arXiv preprint arXiv:2503.22215, 2025. Available from: https://arxiv.org/abs/2503.22215
M. Lee, M. Seo, T. Qu, T. Tuytelaars, and J. Choi, “OASIS: Online sample selection for continual visual instruction tuning,” arXiv preprint arXiv:2506.02011, 2025. Available from: https://arxiv.org/abs/2506.02011
Y. Ma et al., “MLLM-selector: Necessity and diversity-driven high-value data selection for enhanced visual instruction tuning,” arXiv preprint arXiv:2503.20502, 2025. Available from: https://arxiv.org/abs/2503.20502
B. McKinzie et al., “MM1: Methods, analysis & insights from multimodal LLM pre-training,” arXiv preprint arXiv:2403.09611, 2024. Available from: https://arxiv.org/abs/2403.09611
H. Zhang et al., “MM1.5: Methods, analysis & insights from multimodal LLM fine-tuning,” arXiv preprint arXiv:2409.20566, 2024. Available from: https://arxiv.org/abs/2409.20566
S. Li, R. Lin, and S. Pei, “Multi-modal preference alignment remedies degradation of visual instruction tuning on language models,” arXiv preprint arXiv:2402.10884, 2024. Available from: https://arxiv.org/abs/2402.10884
Y. Zhang, H. Fan, and Y. Yang, “Prompt-aware adapter: Towards learning adaptive visual tokens for multimodal large language models,” arXiv preprint arXiv:2405.15684, 2024. Available from: https://arxiv.org/abs/2405.15684
Singh, L. Qamar, N. V. Volta, A. Velamuri, and A. Khanyile, “Vision–Language Foundation Models and Multimodal Large Language Models: A Comprehensive Survey of Architectures, Benchmarks, and Open Challenges,” Preprints, 2026. Available from: https://doi.org/10.20944/preprints202602.0467.v2
Singh, “A Review of Multimodal Vision–Language Models: Foundations, Applications, and Future Directions,” Preprints, 2025. Available from: https://doi.org/10.20944/preprints202510.2511.v1

Cites this article as

M. V, Piyush, G. Singh, "TinyBridge-TriFuse: Lightweight Multi-Expert Fusion for Frozen Vision-Language Models", International Journal of Innovative Research in Engineering and Management (IJIREM), Vol-13, Issue-2, Page No-135-142, 2026. Available from: https://doi.org/10.55524/ijirem.2026.13.2.18