Contrastive Learning

Topics

This post covers the eigth lecture in the course: “Contrastive Learning.”

Contrastive learning has revolutionized deep learning and many of the applications covered later in this course will use it. This lecture covers contrastive loss, its potential problems, and what makes it work.

Lecture Video

Lecture notes

References Cited in Lecture 8: Contrastive Learning

Contrastive Loss

• Wang, Feng, and Huaping Liu. “Understanding the behaviour of contrastive loss.” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2495-2504. 2021.

Varities of Contrastive Learning

• Khosla, Prannay, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. “Supervised contrastive learning.” Advances in Neural Information Processing Systems 33 (2020): 18661-18673.

• Hermans, Alexander, Lucas Beyer, and Bastian Leibe. “In defense of the triplet loss for person re-identification.” arXiv preprint arXiv:1703.07737 (2017).

• Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. “Representation learning with contrastive predictive coding.” arXiv preprint arXiv:1807.03748 (2018).

• Sablayrolles, Alexandre, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. “Spreading vectors for similarity search.” arXiv preprint arXiv:1806.03198 (2018).

• Graf, Florian, Christoph Hofer, Marc Niethammer, and Roland Kwitt. “Dissecting supervised contrastive learning.” In International Conference on Machine Learning, pp. 3821-3830. PMLR, 2021.

• Jing, Li, Pascal Vincent, Yann LeCun, and Yuandong Tian. “Understanding dimensional collapse in contrastive self-supervised learning.” arXiv preprint arXiv:2110.09348 (2021).

Siamese Representation Learning

Chen, Xinlei, and Kaiming He. “Exploring simple siamese representation learning.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758. 2021.

Other Resources

• Contrastive Representation Learning: https://lilianweng.github.io/posts/2021-05-31-contrastive/

Image Source: https://crfm.stanford.edu/2022/04/14/contrastive-learning.html