Zilong He | ViLab

Enhancing Visual Question Answering Via Clustered In-Context Sequence Configuration

Sun, 14 Sep 2025 00:00:00 +0000

本文针对多模态大语言模型在多模态上下文学习中的演示序列配置问题，提出一种基于聚类的上下文配置方法。该方法自适应地对候选数据进行分组，并从每个簇中选取演示样本，以增强序列内多样性并保持语义一致性，从而减少高相似演示带来的归纳偏置，使模型更关注演示的主要意图。在OK-VQA、VQAv2、VizWiz和TextVQA四个视觉问答基准上的实验验证了其有效性。

Ee-mllm: A data-efficient and compute-efficient multimodal large language model

Wed, 21 Aug 2024 00:00:00 +0000

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated satisfactory performance across various vision-language tasks. Current approaches for vision and language interaction fall into two categories: self-attention-based and cross-attention-based methods. However, both approaches present inherent limitations, forcing a trade-off between data and computational efficiency. To address this issue, we introduce the Data-$\textbf{E}$fficient and Compute-$\textbf{E}$fficient $\textbf{MLLM}$ ($\textbf{EE-MLLM}$). Specifically, we modify the original self-attention mechanism in MLLM to a composite attention mechanism. This mechanism has two key characteristics: 1) eliminating the computational overhead of self-attention among visual tokens to achieve $\textbf{compute efficiency}$, and 2) reusing the weights from each layer of LLM to facilitate effective vision-language modality alignment for $\textbf{data efficiency}$. As a result, EE-MLLM significantly outperforms Flamingo with limited training data, and reduces the prefilling time to 79 ms on an H800 GPU, compared to LLaVA’s 277 ms. To further investigate the efficiency of EE-MLLM, we present a training-free variant named EE-MLLM-F, which reduces the computation cost of self-attention-based method without additional training. Experimental results demonstrate the effectiveness of EE-MLLM across a range of benchmarks, including general-purpose datasets like MMBench and SeedBench, as well as fine-grained tasks such as TextVQA and DocVQA.

Task navigator: Decomposing complex tasks for multimodal large language models

Mon, 17 Jun 2024 00:00:00 +0000

本文提出一种名为 Task Navigator 的框架，利用大语言模型作为导航器，将复杂多模态任务逐步分解为更易处理的子问题，并引导多模态大语言模型按步骤求解。该方法无需重新训练模型，而是系统化地调用 MLLM 已有的多种能力，如 OCR、识别、推理等，从而提升复杂任务的处理效果。作者还构建了包含数学推理、嵌入式文本问答和视觉规划等任务的基准，验证了框架的有效性。