Prompt and Prefix Tuning
This post covers the ninth lecture in the course: “Prompt and Prefix Tuning.”
Prompt tuning, the idea of framing everything as a text prediction task using an enormous, frozen LLM (i.e. GPT3) has gained a lot of popularity in recent years. This lecture covers both discrete prompt tuning and continuous prompt tuning, also called prefix tuning.
References Cited in Lecture 8: Contrastive Learning
Prompting and Prefix Tuning
Liu, Pengfei, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.” ACM Computing Surveys 55, no. 9 (2023): 1-35.
Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. “Chain of thought prompting elicits reasoning in large language models.” arXiv preprint arXiv:2201.11903 (2022).
Khattab, Omar, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. “Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP.” arXiv preprint arXiv:2212.14024 (2022).
Li, Xiang Lisa, and Percy Liang. “Prefix-tuning: Optimizing continuous prompts for generation.” arXiv preprint arXiv:2101.00190 (2021).
Lester, Brian, Rami Al-Rfou, and Noah Constant. “The power of scale for parameter-efficient prompt tuning.” arXiv preprint arXiv:2104.08691 (2021).
Image Source: https://thumtblog.github.io/2022/04/05/robust-prefix-tuning/