Jiahui Yuan | ViLab

Holo-World: Unified Camera, Object and Weather Control for Video World Model

Thu, 18 Jun 2026 00:00:00 +0000

本文提出Holo-World，一种统一的视频世界模型，可从单张图像出发，联合控制相机运动、物体动态和天气状态。作者构建了HoloStateData数据集，将多样视频转换为统一控制样本；并提出统一场景适配器，将世界保持与天气迁移分解到不同参数子空间，利用渲染背景、几何缓冲和物体控制维持场景结构，同时建模天气相关外观与粒子效果。场景-天气分解CFG进一步分别引导场景和天气残差，增强目标天气效果。实验表明，Holo-World在保持精确控制的同时，实现了优于视频到视频基线的天气状态生成。

Efficient event-based semantic segmentation via exploiting frame-event fusion: A hybrid neural network approach

Fri, 11 Apr 2025 00:00:00 +0000

本文提出一种高效的混合神经网络框架，用于事件相机语义分割。该框架包含处理事件流的脉冲神经网络（SNN）分支和处理帧图像的人工神经网络（ANN）分支，并设计了自适应时间加权（ATW）注入器、事件驱动稀疏（EDS）注入器和通道选择融合（CSF）模块，以充分融合帧与事件的互补时空信息。在DDD17-Seg、DSEC-Semantic和M3ED-Semantic数据集上取得了最先进精度，并在DSEC-Semantic上降低63%能耗。

Enhancing Visual Tracking by Leveraging High-frequency Information within Event Signals

Wed, 01 Jan 2025 00:00:00 +0000

Traditional object trackers struggle in degraded scenarios, lacking sufficient appearance details of moving targets for precise tracking. Recent trackers have integrated highfrequency event signals to assist tracking. However, they neglect the high-temporalresolution motion information inherent in events, limiting their performance especially in occlusion and background clutter. To address these challenges, we propose HFTrack, a novel tracker designed to fully leverage the spatio-temporal high-frequency information within event signals, thereby enhancing the tracking performance. Specifically, we introduce a frequency-based feature enhancement module, which enriches the frame feature with high-frequency components from events in frequency space, capturing detailed appearance information of moving targets. Additionally, we propose a spatio-temporal information decoder with an auto-regressive temporal query, integrating both historical motion cues from events and enhanced spatial features for robust target localization. Experimental results demonstrate that our HFTrack significantly outperforms existing trackers, showcasing its strong ability to track the target under challenging conditions.

Event-based head pose estimation: Benchmark and method

Sun, 29 Sep 2024 00:00:00 +0000

本文针对传统RGB方法在剧烈运动和极端光照下头部姿态估计困难的问题，引入事件相机的高时间分辨率与高动态范围优势。作者构建了两个大规模事件头部姿态数据集，包含282个序列，覆盖不同分辨率与场景；并提出事件头部姿态估计网络EV-HPE，设计了事件时空融合模块和事件运动感知注意力模块，有效结合事件流时空信息，提升姿态估计精度与鲁棒性。

Event-assisted low-light video object segmentation

Sun, 16 Jun 2024 00:00:00 +0000

本文针对低光照条件下视频目标分割（VOS）性能严重下降的问题，提出一种利用事件相机数据辅助分割的新框架。该方法包含两个关键模块：自适应跨模态融合（ACMF）模块，用于提取并融合图像与事件模态特征以抑制噪声干扰；事件引导记忆匹配（EGMM）模块，用于修正低光下查询帧与记忆帧之间的相似度计算误差。实验表明，该方法在合成和真实低光数据集上均能显著提升分割精度，生成更准确的目标掩码。