Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

发表于 2025-05-05 更新于 2025-05-12 分类于多模态/Multimodal 阅读次数：

本文字数： 2.3k 阅读时长 ≈ 4 分钟

摘要

The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). We have released our codes, models, and demos in https://github.com/OFA-Sys/Chinese-CLIP.

CLIP（Radford等人，2021）的巨大成功促进了对比学习在视觉-语言预训练领域的研究和应用。在本文中，我们构建了一个大规模中文图像-文本对数据集，其中大部分数据是从公开数据集检索得到，我们在这个新数据集上预训练了中文CLIP模型。我们开发了5个不同尺寸的中文CLIP模型，参数量从7700万到9.58亿不等。另外，我们提出一种两阶段预训练方法，首先是冻结图像编码器进行训练，然后进一步优化模型所有参数，这种方式能够有效提高模型性能。实验表明，中文CLIP预训练模型以零样本或者微调方式迁移到下游任务，在MUGE、Flickr30K-CN和COCO-CN上达到SOTA，在ELEVATER（Li et al.，2022）零样本图像分类测试也得到有竞争力的性能。代码、模型和DEMO已开源：https://github.com/OFA-Sys/Chinese-CLIP。