Prompt and Prefix Tuning

February 24, 2023 - 1 minute read - Category: Intro - Tags: Deep learning


This post covers the ninth lecture in the course: “Prompt and Prefix Tuning.”

Prompt tuning, the idea of framing everything as a text prediction task using an enormous, frozen LLM (i.e. GPT3) has gained a lot of popularity in recent years. This lecture covers both discrete prompt tuning and continuous prompt tuning, also called prefix tuning.

Lecture Video

Watch the video

Lecture notes

References Cited in Lecture 8: Contrastive Learning

Prompting and Prefix Tuning

Liu, Pengfei, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.” ACM Computing Surveys 55, no. 9 (2023): 1-35.

Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. “Chain of thought prompting elicits reasoning in large language models.” arXiv preprint arXiv:2201.11903 (2022).

Khattab, Omar, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. “Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP.” arXiv preprint arXiv:2212.14024 (2022).

Li, Xiang Lisa, and Percy Liang. “Prefix-tuning: Optimizing continuous prompts for generation.” arXiv preprint arXiv:2101.00190 (2021).

Lester, Brian, Rami Al-Rfou, and Noah Constant. “The power of scale for parameter-efficient prompt tuning.” arXiv preprint arXiv:2104.08691 (2021).

Other Resources

Image Source: