Self-supervised learning with vision transformer (ViT) has gained much attention recently. Most existing methods rely on either contrastive learning or masked image modeling. The former is suitable for global feature extraction but underperforms in fine-grained tasks. The later explores the internal structure of images but ignores the high information sparsity and unbalanced information distribution. In this paper, we propose a new approach called Attention-guided Contrastive Masked Image Modeling (ACoMIM), which integrates the merits of both paradigms and leverages the attention mechanism of ViT for effective representation. Specifically, it has two pretext tasks, predicting the features of masked regions guided by attention and comparing the global features of masked and unmasked images. We show that these two pretext tasks complement each other and improve our method’s performance. The experiments …
本文提出注意力引导的对比掩码图像建模方法（ACoMIM），融合对比学习与掩码图像建模两种自监督范式，并利用视觉Transformer的注意力机制提升表征能力。该方法包含两个预训练任务：一是根据注意力引导预测掩码区域的特征，二是比较掩码图像与未掩码图像的全局特征。两个任务相互补充，有效缓解了图像信息稀疏与分布不均的问题，在多种下游任务上验证了方法的有效性。