当前位置：首页 > news >正文

[ICLR 2022]How Much Can CLIP Benefit Vision-and-Language Tasks?

news 2025/6/12 15:58:05

论文网址：pdf

英文是纯手打的！论文原文的summarizing and paraphrasing。可能会出现难以避免的拼写错误和语法错误，若有发现欢迎评论指正！文章偏向于笔记，谨慎食用

1. 心得

2. 论文逐段精读

2.1. Abstract

2.2. Introduction

2.3. Background and Motivation

2.3.1. Motivation

2.4. CLIP-ViL

2.4.1. Visual Question Aswering

2.4.2. Image Captioning

2.4.3. Vision-and-Language Navigation

2.5. Vision-and-Language Pre-training

2.5.1. CLIP-VIL_p

2.5.2. Experiments

2.6. Analysis

2.7. Conclusions

1. 心得

（1）？非常简单的一篇文章，感觉在测试CLIP？

2. 论文逐段精读

2.1. Abstract

①Model pre-trained on large number of data brings better performance

②Scenarios suitable for CLIP: plug and fine-tune, or combining with V&L

2.2. Introduction

①Bottleneck of vision-and-language (V&L) tasks: visual representation and scarce labled data

②Most V&L tasks require complex reasoning, which can not use visual model directly

③They define two scenarios:

CLIP_ViL	CLIP in direct task-specific fine-tuning
CLIP_ViL_p	integrate CLIP with V&L pre-training on image-text pairs and transfer to downstream tasks

④Tasks: Visual Question Answering, Image Captioning, and Vision-and-Language Navigation

2.3. Background and Motivation

①Training stage:

visual encoder pretrianing, alignment (opt), downstream task

②Different types of model:

region based, network based, and CLIP (contrastive)

2.3.1. Motivation

①就是说直接把CLIP用在不同复杂视觉任务上性能一般般所以要小改一下

2.4. CLIP-ViL

2.4.1. Visual Question Aswering

①Performance of models on VQA v2.0 dataset:

2.4.2. Image Captioning

①Image captioning comparison table on COCO dataset:

2.4.3. Vision-and-Language Navigation

①The model performance on Room-to-Room (R2R) dataset:

②Changing ResNet to CLIP, the performance table:

2.5. Vision-and-Language Pre-training

2.5.1. CLIP-VIL_p

①For text segment $T$ , tokenize it into subwords $\{w_{1},w_{2},...,w_{k}\}$ and further embedded as the sum of its token, position and segment embeddings $\{\textbf{w}_{1},\textbf{w}_{2},...,\textbf{w}_{k}\}$