Yueyi Zhang | ViLab

Seeing the unseen: Zooming in the dark with event cameras

Sat, 14 Mar 2026 00:00:00 +0000

本文提出RetinexEVSR，首个事件驱动的低光视频超分辨率框架。该框架利用高对比度事件信号与Retinex先验，通过双向跨模态融合策略，有效整合噪声事件数据与退化RGB帧中的有用信息。其中，照明引导事件增强模块利用Retinex模型导出的光照图逐步细化事件特征，抑制低光伪影并保留高对比度细节；事件引导反射率增强模块则通过多尺度融合机制动态恢复反射率细节。实验表明，该方法在三个数据集上达到最优性能，在SDSD基准上相比先前事件方法提升2.95 dB，并减少65%运行时间。

EvTexture++: Event-Driven Texture Enhancement for Video Super-Resolution

Mon, 02 Feb 2026 00:00:00 +0000

本文提出EvTexture++，一种事件驱动的视频超分辨率纹理增强框架。与以往将事件用于运动估计不同，该方法利用事件的高频时空细节显式恢复纹理，通过定制纹理增强分支和迭代纹理增强模块，逐步挖掘高时间分辨率事件信息，实现纹理区域的渐进细化，从而生成更精确、细节更丰富的高分辨率视频。该框架还可作为即插即用模块提升现有VSR模型性能。

Create anything anywhere: Layout-controllable personalized diffusion model for multiple subjects

Mon, 30 Jun 2025 00:00:00 +0000

Diffusion models have significantly advanced text-to-image generation, laying the foundation for the development of personalized generative frameworks. However, existing methods lack precise layout controllability and overlook the potential of dynamic features of reference subjects in improving fidelity. In this work, we propose Layout-Controllable Personalized Diffusion (LCP-Diffusion) model, a novel framework that integrates subject identity preservation with flexible layout guidance in a tuning-free approach. Our model employs a Dynamic-Static Complementary Visual Refining module to comprehensively capture the intricate details of reference subjects, and introduces a Dual Layout Control mechanism to enforce robust spatial control across both training and inference stages. Extensive experiments validate that LCP-Diffusion excels in both identity preservation and layout controllability. To the best of our …

Incomplete multi-modal brain tumor segmentation via learnable sorting state space model

Tue, 10 Jun 2025 00:00:00 +0000

本文提出一种可学习排序状态空间模型（LS3M），用于不完整多模态脑肿瘤分割。该方法基于Mamba架构高效建模长距离依赖，并引入可微置换矩阵，根据模态特定特征对输入序列进行动态重排序，从而保留3D脑MRI中关键的空间归纳偏置与长程语义相关性。LS3M能够充分利用可用模态信息，提升分割性能。

D-FINE: Redefine regression task of DETRs as fine-grained distribution refinement

Thu, 01 May 2025 00:00:00 +0000

D-FINE是一种实时目标检测器，通过重新定义DETR中的边界框回归任务实现高精度定位。其核心包含细粒度分布细化（FDR）和全局最优定位自蒸馏（GO-LSD）。FDR将回归从预测固定坐标改为迭代细化概率分布，提供细粒度中间表示；GO-LSD通过自蒸馏将定位知识从最终层传递到浅层，并简化深层残差预测。在COCO上达到54.0%/55.8% AP，124/78 FPS，预训练后达57.1%/59.3% AP，超越现有实时检测器，并显著提升多种DETR模型性能。

Efficient event-based semantic segmentation via exploiting frame-event fusion: A hybrid neural network approach

Fri, 11 Apr 2025 00:00:00 +0000

本文提出一种高效的混合神经网络框架，用于事件相机语义分割。该框架包含处理事件流的脉冲神经网络（SNN）分支和处理帧图像的人工神经网络（ANN）分支，并设计了自适应时间加权（ATW）注入器、事件驱动稀疏（EDS）注入器和通道选择融合（CSF）模块，以充分融合帧与事件的互补时空信息。在DDD17-Seg、DSEC-Semantic和M3ED-Semantic数据集上取得了最先进精度，并在DSEC-Semantic上降低63%能耗。

Event-enhanced blurry video super-resolution

Fri, 11 Apr 2025 00:00:00 +0000

In this paper, we tackle the task of blurry video super-resolution (BVSR), aiming to generate high-resolution (HR) videos from low-resolution (LR) and blurry inputs. Current BVSR methods often fail to restore sharp details at high resolutions, resulting in noticeable artifacts and jitter due to insufficient motion information for deconvolution and the lack of high-frequency details in LR frames. To address these challenges, we introduce event signals into BVSR and propose a novel event-enhanced network, Ev-DeblurVSR. To effectively fuse information from frames and events for feature deblurring, we introduce a reciprocal feature deblurring module that leverages motion information from intra-frame events to deblur frame features while reciprocally using global scene context from the frames to enhance event features. Furthermore, to enhance temporal consistency, we propose a hybrid deformable alignment module that fully exploits the complementary motion information from inter-frame events and optical flow to improve motion estimation in the deformable alignment process. Extensive evaluations demonstrate that Ev-DeblurVSR establishes a new state-of-the-art performance on both synthetic and real-world datasets. Notably, on real data, our method is 2.59 dB more accurate and 7.28× faster than the recent best BVSR baseline FMA-Net.

Spiking point transformer for point cloud classification

Fri, 11 Apr 2025 00:00:00 +0000

本文提出Spiking Point Transformer（SPT），首个基于Transformer的脉冲神经网络框架，用于三维点云分类。SPT设计队列驱动采样直接编码，在降低计算成本的同时保留关键支撑点；并引入混合动力学积分发放神经元（HD-IF），模拟选择性神经元激活，减少对特定人工神经元的过度依赖。在多个真实与合成点云基准上取得领先结果，理论能耗较ANN对应模型降低至少6.4倍。

Hierarchical Task-aware Temporal Modeling and Matching for few-shot action recognition

Tue, 01 Apr 2025 00:00:00 +0000

本文针对少样本动作识别中训练样本稀缺且视频结构复杂的问题，提出分层任务感知时间建模与匹配方法（HTTMM）。该方法通过分层结构充分建模时空特征，并利用任务感知机制增强对关键运动模式的感知，从而提升查询样本与支持样本之间的匹配效果。在多个基准数据集上的实验验证了其有效性，尤其适用于需要局部运动感知的细粒度动作分类任务。

Semantic-aware late-stage supervised contrastive learning for fine-grained action recognition

Wed, 08 Jan 2025 00:00:00 +0000

本文针对细粒度动作识别中类间差异小、类内差异大的挑战，提出了一种语义感知的后期监督对比学习方法。该方法通过后期监督对比学习策略，有效减少了对比学习所需的训练轮次，降低了计算成本；同时引入语义距离建模，在调整特征表示时显式考虑细粒度动作之间的语义关系，从而提升判别能力。实验表明该方法在多个细粒度动作识别基准上取得了优越性能。

Enhancing Visual Tracking by Leveraging High-frequency Information within Event Signals

Wed, 01 Jan 2025 00:00:00 +0000

Traditional object trackers struggle in degraded scenarios, lacking sufficient appearance details of moving targets for precise tracking. Recent trackers have integrated highfrequency event signals to assist tracking. However, they neglect the high-temporalresolution motion information inherent in events, limiting their performance especially in occlusion and background clutter. To address these challenges, we propose HFTrack, a novel tracker designed to fully leverage the spatio-temporal high-frequency information within event signals, thereby enhancing the tracking performance. Specifically, we introduce a frequency-based feature enhancement module, which enriches the frame feature with high-frequency components from events in frequency space, capturing detailed appearance information of moving targets. Additionally, we propose a spatio-temporal information decoder with an auto-regressive temporal query, integrating both historical motion cues from events and enhanced spatial features for robust target localization. Experimental results demonstrate that our HFTrack significantly outperforms existing trackers, showcasing its strong ability to track the target under challenging conditions.

Visual perception by large language model’s weights

Mon, 16 Dec 2024 00:00:00 +0000

Existing Multimodal Large Language Models (MLLMs) follow the paradigm that perceives visual information by aligning visual features with the input space of Large Language Models (LLMs) and concatenating visual tokens with text tokens to form a unified sequence input for LLMs. These methods demonstrate promising results on various vision-language tasks but are limited by the high computational effort due to the extended input sequence resulting from the involvement of visual tokens. In this paper, instead of input space alignment, we propose a novel parameter space alignment paradigm that represents visual information as model weights. For each input image, we use a vision encoder to extract visual features, convert features into perceptual weights, and merge the perceptual weights with LLM’s weights. In this way, the input of LLM does not require visual tokens, which reduces the length of the input sequence and greatly improves efficiency. Following this paradigm, we propose VLoRA with the perceptual weights generator. The perceptual weights generator is designed to convert visual features to perceptual weights with low-rank property, exhibiting a form similar to LoRA. The experimental results show that our VLoRA achieves comparable performance on various benchmarks for MLLMs, while significantly reducing the computational costs for both training and inference. Code and models are released at\url {https://github. com/FeipengMa6/VLoRA}.

Multi-modal diffusion network with controllable variability for medical image segmentation

Tue, 03 Dec 2024 00:00:00 +0000

本文提出一种具有可控变异性的多模态扩散分割网络（MMDSN），用于医学图像分割。该方法通过医学文本注释实现多模态条件控制，增强视觉语义表示的一致性，并建立视觉与语言之间的对应关系。同时，MMDSN 在潜在高斯空间中对多个时间步的不确定性分布进行约束，从而控制每个去噪时间步的变异性，减少扩散模型随机采样带来的分割偏差。在 Qata-Covid19 等数据集上的实验验证了其有效性。

Asymmetric event-guided video super-resolution

Mon, 28 Oct 2024 00:00:00 +0000

本文首次提出非对称事件引导的视频超分辨率任务，针对事件相机与RGB相机难以严格标定的实际场景，构建了非对称事件引导视频超分辨率网络（AsEVSRN）。该网络通过专门设计的事件特征利用与跨模态融合机制，充分发挥事件相机高时间分辨率优势，有效提升视频超分辨率性能，拓展了事件相机在双摄手机、无人机等新兴高分辨率设备上的应用潜力。

Semantic-enhanced point-box joint prompting for video object segmentation

Sun, 27 Oct 2024 00:00:00 +0000

本文提出基于SAM的语义增强点框联合提示框架SAM-SPB，用于视频对象分割。该框架通过点跟踪分支维持对象局部结构信息，并利用语义感知的基于记忆的框跟踪分支跨帧传播对象语义一致性，从而结合局部与全局线索实现鲁棒分割。在主流VOS基准上取得了领先性能，验证了点框联合提示相比仅用点提示的优势。

Event-adapted video super-resolution

Sun, 29 Sep 2024 00:00:00 +0000

Introducing event cameras into video super-resolution (VSR) shows great promise. In practice, however, integrating event data as a new modality necessitates a laborious model architecture design. This not only consumes substantial time and effort but also disregards valuable insights from successful existing VSR models. Furthermore, the resource-intensive process of retraining these newly designed models exacerbates the challenge. In this paper, inspired by the recent success of parameter-efficient tuning in reducing the number of trainable parameters of a pre-trained model for downstream tasks, we introduce the Event AdapTER (EATER) for VSR. EATER efficiently utilizes knowledge of VSR models at the feature level through two lightweight and trainable components: the event-adapted alignment (EAA) unit and the event-adapted fusion (EAF) unit. The EAA unit aligns multiple frames based on the event …

Event-based head pose estimation: Benchmark and method

Sun, 29 Sep 2024 00:00:00 +0000

本文针对传统RGB方法在剧烈运动和极端光照下头部姿态估计困难的问题，引入事件相机的高时间分辨率与高动态范围优势。作者构建了两个大规模事件头部姿态数据集，包含282个序列，覆盖不同分辨率与场景；并提出事件头部姿态估计网络EV-HPE，设计了事件时空融合模块和事件运动感知注意力模块，有效结合事件流时空信息，提升姿态估计精度与鲁棒性。

Ee-mllm: A data-efficient and compute-efficient multimodal large language model

Wed, 21 Aug 2024 00:00:00 +0000

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated satisfactory performance across various vision-language tasks. Current approaches for vision and language interaction fall into two categories: self-attention-based and cross-attention-based methods. However, both approaches present inherent limitations, forcing a trade-off between data and computational efficiency. To address this issue, we introduce the Data-$\textbf{E}$fficient and Compute-$\textbf{E}$fficient $\textbf{MLLM}$ ($\textbf{EE-MLLM}$). Specifically, we modify the original self-attention mechanism in MLLM to a composite attention mechanism. This mechanism has two key characteristics: 1) eliminating the computational overhead of self-attention among visual tokens to achieve $\textbf{compute efficiency}$, and 2) reusing the weights from each layer of LLM to facilitate effective vision-language modality alignment for $\textbf{data efficiency}$. As a result, EE-MLLM significantly outperforms Flamingo with limited training data, and reduces the prefilling time to 79 ms on an H800 GPU, compared to LLaVA’s 277 ms. To further investigate the efficiency of EE-MLLM, we present a training-free variant named EE-MLLM-F, which reduces the computation cost of self-attention-based method without additional training. Experimental results demonstrate the effectiveness of EE-MLLM across a range of benchmarks, including general-purpose datasets like MMBench and SeedBench, as well as fine-grained tasks such as TextVQA and DocVQA.

A micro-expression recognition system with event cameras

Mon, 15 Jul 2024 00:00:00 +0000

本文提出了一种基于事件相机的微表情识别系统。针对微表情持续时间短、幅度微弱、难以用传统相机捕捉的问题，系统利用事件相机的高时间分辨率特性，设计了事件增强运动提取器（EEME）以放大细微运动，并引入事件引导注意力（EGA）聚焦关键面部区域，从而提升微表情识别的准确性与鲁棒性。该系统为情感计算领域提供了有效工具。

Estme: Event-driven spatio-temporal motion enhancement for micro-expression recognition

Mon, 15 Jul 2024 00:00:00 +0000

本文针对微表情识别中动作幅度小、持续时间短、难以捕捉的问题，提出了一种事件驱动的时空运动增强网络。该方法引入事件相机捕获的高时间分辨率事件信号，设计事件增强运动提取模块以增强细微运动细节，并利用事件引导注意力模块聚焦特定区域的微小变化，从而获取更精确的空间特征。在合成和真实数据集上的实验结果表明，该方法在微表情识别任务上具有优越性能。

Task navigator: Decomposing complex tasks for multimodal large language models

Mon, 17 Jun 2024 00:00:00 +0000

本文提出一种名为 Task Navigator 的框架，利用大语言模型作为导航器，将复杂多模态任务逐步分解为更易处理的子问题，并引导多模态大语言模型按步骤求解。该方法无需重新训练模型，而是系统化地调用 MLLM 已有的多种能力，如 OCR、识别、推理等，从而提升复杂任务的处理效果。作者还构建了包含数学推理、嵌入式文本问答和视觉规划等任务的基准，验证了框架的有效性。

Event-assisted low-light video object segmentation

Sun, 16 Jun 2024 00:00:00 +0000

本文针对低光照条件下视频目标分割（VOS）性能严重下降的问题，提出一种利用事件相机数据辅助分割的新框架。该方法包含两个关键模块：自适应跨模态融合（ACMF）模块，用于提取并融合图像与事件模态特征以抑制噪声干扰；事件引导记忆匹配（EGMM）模块，用于修正低光下查询帧与记忆帧之间的相似度计算误差。实验表明，该方法在合成和真实低光数据集上均能显著提升分割精度，生成更准确的目标掩码。

Scene adaptive sparse transformer for event-based object detection

Sun, 16 Jun 2024 00:00:00 +0000

本文针对事件相机目标检测中Transformer计算成本过高的问题，提出场景自适应稀疏Transformer（SAST）。该方法通过窗口-令牌协同稀疏化与场景特定稀疏优化，在保持低计算量的同时实现高检测性能，并能根据场景复杂度自适应调整稀疏程度，有效平衡了检测精度与效率。

Deep multi-threshold spiking-UNet for image processing

Fri, 14 Jun 2024 00:00:00 +0000

本文提出Spiking-UNet，将脉冲神经网络与U-Net架构相结合用于图像处理任务。针对脉冲传播导致的信息损失问题，设计多阈值脉冲神经元以增强信息传递能力；同时采用基于预训练U-Net的转换与微调训练策略，有效解决了训练难题。在图像分割和去噪等任务上验证了所提方法的有效性。

Multi-modal generative embedding model

Wed, 29 May 2024 00:00:00 +0000

本文提出多模态生成嵌入模型MM-GEM，将生成与嵌入两种目标统一于单个大语言模型中，实现每个模态仅需一个模型。通过引入PoolAggregator提升效率并支持细粒度嵌入与生成。实验表明，生成与嵌入目标并不显著冲突，模型在跨模态检索、零样本分类和图像描述等任务上表现优异，同时具备区域级描述生成与检索能力，并在长文本图像检索中取得显著提升。

Semi-supervised medical image segmentation via dynamic pseudo-label refinement

Mon, 27 May 2024 00:00:00 +0000

本文提出一种基于动态伪标签优化的半监督医学图像分割框架。针对双视角方法易丢失重要数据且伪标签不准确的问题，设计分层伪标签生成（HPLG）与动态伪标签校正（DPLC）两个互补模块，按可靠性生成分层像素级伪标签，并利用双视角的一致性与差异进行动态修正，从而提升分割性能与标签质量。

Understanding of facial features in face perception: insights from deep convolutional neural networks

Tue, 09 Apr 2024 00:00:00 +0000

Introduction

Image captioning with multi-context synthetic data

Sun, 24 Mar 2024 00:00:00 +0000

本文提出一种基于多上下文合成数据的图像描述生成方法（ICSD）。针对现有合成数据仅能生成单一视角、上下文有限的问题，利用大语言模型从语料中提取同一场景的多视角描述并压缩为多上下文句子，再通过扩散模型生成复杂图像，仅用合成图像-文本对训练描述模型。在MSCOCO、Flickr30k和NoCaps等数据集上取得领先性能。

Tmformer: Token merging transformer for brain tumor segmentation with missing modalities

Sun, 24 Mar 2024 00:00:00 +0000

本文提出 TMFormer，一种用于缺失模态脑肿瘤分割的 Token 合并 Transformer。该方法通过提取并合并可用模态为更紧凑的 token 序列，解决现有方法以零图填充缺失模态带来的特征偏差与冗余计算问题。其核心包括单模态 Token 合并块（UMB）和多模态 Token 合并块（MMB），分别增强单模态表示并缓解多模态融合偏差。在 BraTS 2018 和 2020 数据集上的实验表明，TMFormer 在缺失模态场景下优于现有方法。

Anatomical consistency distillation and inconsistency synthesis for brain tumor segmentation with missing modalities

Mon, 01 Jan 2024 00:00:00 +0000

本文提出ACDIS框架，用于解决脑肿瘤分割中MRI模态缺失的问题。通过解剖一致性蒸馏将多模态图像中的共享解剖结构迁移至单模态表示，并利用模态特征合成块生成模态特定特征，从而增强单模态图像在特定区域的组织表现。该方法有效缓解了模态缺失带来的性能下降，提升了分割精度。

Attention-guided contrastive masked image modeling for transformer-based self-supervised learning

Sun, 08 Oct 2023 00:00:00 +0000

本文提出注意力引导的对比掩码图像建模方法（ACoMIM），融合对比学习与掩码图像建模两种自监督范式，并利用视觉Transformer的注意力机制提升表征能力。该方法包含两个预训练任务：一是根据注意力引导预测掩码区域的特征，二是比较掩码图像与未掩码图像的全局特征。两个任务相互补充，有效缓解了图像信息稀疏与分布不均的问题，在多种下游任务上验证了方法的有效性。

Video super-resolution via event-driven temporal alignment

Sun, 08 Oct 2023 00:00:00 +0000

本文提出一种事件驱动的双向视频超分辨率框架（EBVSR），利用事件相机的高时间分辨率特性捕捉非线性运动，并设计事件辅助的时间对齐模块，以补充光流法在快速光照变化下的不足。同时构建基于事件的帧合成模块，通过双向跨模态融合增强网络对光照变化的鲁棒性。在合成和真实数据上的实验验证了该方法在视频超分辨率任务中的有效性。

Eoformer: Edge-oriented transformer for brain tumor segmentation

Sun, 01 Oct 2023 00:00:00 +0000

本文提出边缘导向Transformer（EoFormer），用于脑肿瘤MRI图像分割。该方法采用CNN-Transformer混合编码器，CNN提取局部低级特征，Transformer建模长距离依赖以生成全局高级特征；解码器集成边缘导向Sobel与Laplacian锐化模块，增强边缘信息。同时引入高效注意力与重参数化技术，提升特征表示能力与分割精度。

Get: Group event transformer for event-based vision

Sun, 01 Oct 2023 00:00:00 +0000

本文提出一种基于分组的事件视觉Transformer骨干网络GET，用于事件相机视觉任务。GET将事件按时间戳和极性分组为Group Token，并在特征提取过程中解耦时空信息与极性信息。通过事件双自注意力模块和分组Token聚合模块，实现空间与时间-极性信息的有效通信与整合，充分利用事件数据特性，提升事件视觉任务性能。

Learned rate-distortion cost prediction for ultrafast screen content intra coding

Tue, 18 Jul 2023 00:00:00 +0000

本文面向实时屏幕内容通信中的超快编码需求，提出基于学习的率失真代价预测方法。该方法不再通过实际编码计算RD代价，而是构建神经网络分别预测帧内预测、调色板及普通IBC模式的RD代价，并对IBC merge模式结合运动补偿与线性回归进行预测。利用预测结果生成分区-模式映射集，从而显著降低H.265/HEVC SCC扩展的编码复杂度，在保持编码效率的同时实现超快屏幕内容帧内编码。

Multimodal sentiment analysis with preferential fusion and distance-aware contrastive learning

Mon, 10 Jul 2023 00:00:00 +0000

本文针对多模态情感分析中文本模态与情感标签之间存在的虚假关联问题，提出了一种名为PriSA的新框架。该框架首先通过优先跨模态融合方法，利用文本模态引导计算跨模态相关性；随后引入距离感知对比学习，利用情感标签之间的距离信息进一步计算混合模态相关性；最终基于混合模态相关性和判别性类内特征识别情感信息。实验表明该方法能有效缓解文本虚假关联带来的影响，提升多模态情感分析性能。

Better and faster: Adaptive event conversion for event-based object detection

Mon, 26 Jun 2023 00:00:00 +0000

本文针对事件相机目标检测任务，提出高效事件表示Hyper Histogram，充分保留事件极性与时间信息；设计自适应事件转换模块AEC，基于事件密度通过自适应队列将事件流转换为超直方图，并适配现有帧基检测器；还提出事件增强方法Shadow Mosaic，提升样本多样性与泛化能力。在YOLOv5、Deformable-DETR和RetinaNet上验证，在1Mpx、Gen1和MVSEC-NIGHTL21数据集上取得显著优势，且推理速度快。

Deep spiking-unet for image processing

Sun, 01 Jan 2023 00:00:00 +0000

本文提出一种深度脉冲U-Net架构，将脉冲神经网络的生物合理性与U-Net的多尺度特征提取能力相结合，用于图像处理任务。通过脉冲神经元替代传统激活函数，在保持图像处理性能的同时显著降低计算能耗，为低功耗边缘端图像处理提供了新思路。

Text-Only Image Captioning with Multi-Context Data Generation.

Sun, 01 Jan 2023 00:00:00 +0000

本文针对仅使用文本数据训练图像描述模型的任务，提出一种多上下文数据生成方法。通过构造多样化的文本上下文，生成合成图像-描述训练对，使模型学习跨模态对齐与语义描述能力，减少对真实图像标注的依赖。实验表明该方法在多个图像描述基准上取得有效性能，为数据稀缺场景下的多模态学习提供了新思路。