Yizhou Zhou | ViLab

Visual perception by large language model’s weights

Mon, 16 Dec 2024 00:00:00 +0000

Existing Multimodal Large Language Models (MLLMs) follow the paradigm that perceives visual information by aligning visual features with the input space of Large Language Models (LLMs) and concatenating visual tokens with text tokens to form a unified sequence input for LLMs. These methods demonstrate promising results on various vision-language tasks but are limited by the high computational effort due to the extended input sequence resulting from the involvement of visual tokens. In this paper, instead of input space alignment, we propose a novel parameter space alignment paradigm that represents visual information as model weights. For each input image, we use a vision encoder to extract visual features, convert features into perceptual weights, and merge the perceptual weights with LLM’s weights. In this way, the input of LLM does not require visual tokens, which reduces the length of the input sequence and greatly improves efficiency. Following this paradigm, we propose VLoRA with the perceptual weights generator. The perceptual weights generator is designed to convert visual features to perceptual weights with low-rank property, exhibiting a form similar to LoRA. The experimental results show that our VLoRA achieves comparable performance on various benchmarks for MLLMs, while significantly reducing the computational costs for both training and inference. Code and models are released at\url {https://github. com/FeipengMa6/VLoRA}.

Ee-mllm: A data-efficient and compute-efficient multimodal large language model

Wed, 21 Aug 2024 00:00:00 +0000

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated satisfactory performance across various vision-language tasks. Current approaches for vision and language interaction fall into two categories: self-attention-based and cross-attention-based methods. However, both approaches present inherent limitations, forcing a trade-off between data and computational efficiency. To address this issue, we introduce the Data-$\textbf{E}$fficient and Compute-$\textbf{E}$fficient $\textbf{MLLM}$ ($\textbf{EE-MLLM}$). Specifically, we modify the original self-attention mechanism in MLLM to a composite attention mechanism. This mechanism has two key characteristics: 1) eliminating the computational overhead of self-attention among visual tokens to achieve $\textbf{compute efficiency}$, and 2) reusing the weights from each layer of LLM to facilitate effective vision-language modality alignment for $\textbf{data efficiency}$. As a result, EE-MLLM significantly outperforms Flamingo with limited training data, and reduces the prefilling time to 79 ms on an H800 GPU, compared to LLaVA’s 277 ms. To further investigate the efficiency of EE-MLLM, we present a training-free variant named EE-MLLM-F, which reduces the computation cost of self-attention-based method without additional training. Experimental results demonstrate the effectiveness of EE-MLLM across a range of benchmarks, including general-purpose datasets like MMBench and SeedBench, as well as fine-grained tasks such as TextVQA and DocVQA.

Task navigator: Decomposing complex tasks for multimodal large language models

Mon, 17 Jun 2024 00:00:00 +0000

本文提出一种名为 Task Navigator 的框架，利用大语言模型作为导航器，将复杂多模态任务逐步分解为更易处理的子问题，并引导多模态大语言模型按步骤求解。该方法无需重新训练模型，而是系统化地调用 MLLM 已有的多种能力，如 OCR、识别、推理等，从而提升复杂任务的处理效果。作者还构建了包含数学推理、嵌入式文本问答和视觉规划等任务的基准，验证了框架的有效性。

Multi-modal generative embedding model

Wed, 29 May 2024 00:00:00 +0000

本文提出多模态生成嵌入模型MM-GEM，将生成与嵌入两种目标统一于单个大语言模型中，实现每个模态仅需一个模型。通过引入PoolAggregator提升效率并支持细粒度嵌入与生成。实验表明，生成与嵌入目标并不显著冲突，模型在跨模态检索、零样本分类和图像描述等任务上表现优异，同时具备区域级描述生成与检索能力，并在长文本图像检索中取得显著提升。

Image captioning with multi-context synthetic data

Sun, 24 Mar 2024 00:00:00 +0000

本文提出一种基于多上下文合成数据的图像描述生成方法（ICSD）。针对现有合成数据仅能生成单一视角、上下文有限的问题，利用大语言模型从语料中提取同一场景的多视角描述并压缩为多上下文句子，再通过扩散模型生成复杂图像，仅用合成图像-文本对训练描述模型。在MSCOCO、Flickr30k和NoCaps等数据集上取得领先性能。

Text-Only Image Captioning with Multi-Context Data Generation.

Sun, 01 Jan 2023 00:00:00 +0000

本文针对仅使用文本数据训练图像描述模型的任务，提出一种多上下文数据生成方法。通过构造多样化的文本上下文，生成合成图像-描述训练对，使模型学习跨模态对齐与语义描述能力，减少对真实图像标注的依赖。实验表明该方法在多个图像描述基准上取得有效性能，为数据稀缺场景下的多模态学习提供了新思路。