跨模态检索 | ViLab https://vilab.team/tag/%E8%B7%A8%E6%A8%A1%E6%80%81%E6%A3%80%E7%B4%A2/ 跨模态检索 Hugo Blox Builder (https://hugoblox.com)en-usWed, 29 May 2024 00:00:00 +0000 https://vilab.team/media/icon_hu2896232876136423579.png 跨模态检索 https://vilab.team/tag/%E8%B7%A8%E6%A8%A1%E6%80%81%E6%A3%80%E7%B4%A2/ Multi-modal generative embedding model https://vilab.team/publication/multi-modal-generative-embedding-model/ Wed, 29 May 2024 00:00:00 +0000 https://vilab.team/publication/multi-modal-generative-embedding-model/ <p>本文提出多模态生成嵌入模型MM-GEM，将生成与嵌入两种目标统一于单个大语言模型中，实现每个模态仅需一个模型。通过引入PoolAggregator提升效率并支持细粒度嵌入与生成。实验表明，生成与嵌入目标并不显著冲突，模型在跨模态检索、零样本分类和图像描述等任务上表现优异，同时具备区域级描述生成与检索能力，并在长文本图像检索中取得显著提升。</p>