Prompt and Prefix Tuning

Topics

This post covers the ninth lecture in the course: “Prompt and Prefix Tuning.”

Prompt tuning, the idea of framing everything as a text prediction task using an enormous, frozen LLM (i.e. GPT3) has gained a lot of popularity in recent years. This lecture covers both discrete prompt tuning and continuous prompt tuning, also called prefix tuning.

Lecture Video

Lecture notes

References Cited in Lecture 8: Contrastive Learning

Prompting and Prefix Tuning

Liu, Pengfei, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.” ACM Computing Surveys 55, no. 9 (2023): 1-35.

Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. “Chain of thought prompting elicits reasoning in large language models.” arXiv preprint arXiv:2201.11903 (2022).

Khattab, Omar, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. “Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP.” arXiv preprint arXiv:2212.14024 (2022).

Li, Xiang Lisa, and Percy Liang. “Prefix-tuning: Optimizing continuous prompts for generation.” arXiv preprint arXiv:2101.00190 (2021).

Lester, Brian, Rami Al-Rfou, and Noah Constant. “The power of scale for parameter-efficient prompt tuning.” arXiv preprint arXiv:2104.08691 (2021).

Other Resources

Image Source: https://thumtblog.github.io/2022/04/05/robust-prefix-tuning/