Zhiwei Xiong | ViLab

Event-enhanced blurry video super-resolution

Fri, 11 Apr 2025 00:00:00 +0000

In this paper, we tackle the task of blurry video super-resolution (BVSR), aiming to generate high-resolution (HR) videos from low-resolution (LR) and blurry inputs. Current BVSR methods often fail to restore sharp details at high resolutions, resulting in noticeable artifacts and jitter due to insufficient motion information for deconvolution and the lack of high-frequency details in LR frames. To address these challenges, we introduce event signals into BVSR and propose a novel event-enhanced network, Ev-DeblurVSR. To effectively fuse information from frames and events for feature deblurring, we introduce a reciprocal feature deblurring module that leverages motion information from intra-frame events to deblur frame features while reciprocally using global scene context from the frames to enhance event features. Furthermore, to enhance temporal consistency, we propose a hybrid deformable alignment module that fully exploits the complementary motion information from inter-frame events and optical flow to improve motion estimation in the deformable alignment process. Extensive evaluations demonstrate that Ev-DeblurVSR establishes a new state-of-the-art performance on both synthetic and real-world datasets. Notably, on real data, our method is 2.59 dB more accurate and 7.28× faster than the recent best BVSR baseline FMA-Net.

Asymmetric event-guided video super-resolution

Mon, 28 Oct 2024 00:00:00 +0000

本文首次提出非对称事件引导的视频超分辨率任务，针对事件相机与RGB相机难以严格标定的实际场景，构建了非对称事件引导视频超分辨率网络（AsEVSRN）。该网络通过专门设计的事件特征利用与跨模态融合机制，充分发挥事件相机高时间分辨率优势，有效提升视频超分辨率性能，拓展了事件相机在双摄手机、无人机等新兴高分辨率设备上的应用潜力。

Event-adapted video super-resolution

Sun, 29 Sep 2024 00:00:00 +0000

Introducing event cameras into video super-resolution (VSR) shows great promise. In practice, however, integrating event data as a new modality necessitates a laborious model architecture design. This not only consumes substantial time and effort but also disregards valuable insights from successful existing VSR models. Furthermore, the resource-intensive process of retraining these newly designed models exacerbates the challenge. In this paper, inspired by the recent success of parameter-efficient tuning in reducing the number of trainable parameters of a pre-trained model for downstream tasks, we introduce the Event AdapTER (EATER) for VSR. EATER efficiently utilizes knowledge of VSR models at the feature level through two lightweight and trainable components: the event-adapted alignment (EAA) unit and the event-adapted fusion (EAF) unit. The EAA unit aligns multiple frames based on the event …

Event-assisted low-light video object segmentation

Sun, 16 Jun 2024 00:00:00 +0000

本文针对低光照条件下视频目标分割（VOS）性能严重下降的问题，提出一种利用事件相机数据辅助分割的新框架。该方法包含两个关键模块：自适应跨模态融合（ACMF）模块，用于提取并融合图像与事件模态特征以抑制噪声干扰；事件引导记忆匹配（EGMM）模块，用于修正低光下查询帧与记忆帧之间的相似度计算误差。实验表明，该方法在合成和真实低光数据集上均能显著提升分割精度，生成更准确的目标掩码。

Deep multi-threshold spiking-UNet for image processing

Fri, 14 Jun 2024 00:00:00 +0000

本文提出Spiking-UNet，将脉冲神经网络与U-Net架构相结合用于图像处理任务。针对脉冲传播导致的信息损失问题，设计多阈值脉冲神经元以增强信息传递能力；同时采用基于预训练U-Net的转换与微调训练策略，有效解决了训练难题。在图像分割和去噪等任务上验证了所提方法的有效性。

Get: Group event transformer for event-based vision

Sun, 01 Oct 2023 00:00:00 +0000

本文提出一种基于分组的事件视觉Transformer骨干网络GET，用于事件相机视觉任务。GET将事件按时间戳和极性分组为Group Token，并在特征提取过程中解耦时空信息与极性信息。通过事件双自注意力模块和分组Token聚合模块，实现空间与时间-极性信息的有效通信与整合，充分利用事件数据特性，提升事件视觉任务性能。

Deep spiking-unet for image processing

Sun, 01 Jan 2023 00:00:00 +0000

本文提出一种深度脉冲U-Net架构，将脉冲神经网络的生物合理性与U-Net的多尺度特征提取能力相结合，用于图像处理任务。通过脉冲神经元替代传统激活函数，在保持图像处理性能的同时显著降低计算能耗，为低功耗边缘端图像处理提供了新思路。