CSCI-499 Fall 2023: Language Models in NLP
🍂 Fall 2023 ⏰ Mon / Wed 10:00 am - 11:50 am 📍 ZHS 360
Teaching Assistant: Avi Thawani
Office Hours: Wednesdays 4-5pm; Leavey Library Lower Level
Announcements
Nov 1: Week 12
- Quiz 6 was conducted in class.
Oct 30: Week 11
- Progress Report Feedback and grades are out.
- Quiz 6 on Wednesday, 11/1.
Oct 16: Week 9
- Graded Quiz 4 sheets were distributed in class.
- Please sign up for Final Paper Presentations (see Piazza post).
- Quiz 5 on Monday, 10/23.
Sep 27: Week 6
- Graded Quiz 3 sheets were distributed in class.
- Swabha is OOT next week, but we will have guest / TA lectures + Quiz 4 next week.
Sep 20: Week 5
- Quiz 3 was completed in class; grades will be shared next Wednesday. Future quiz dates will be announced in class (one class ahead).
- Project Proposals were graded and feedback shared via email.
Sep 11: Week 4
- Quiz 2 was completed in class; grades will be distributed next Wednesday.
- HW2 released on Friday, 9/15 and due by 9/29.
Sep 6: Week 3
- HW1 deadline is now extended by 24hrs; due on 9/7 by 11:59PM PT.
Aug 30: Week 2
- Quiz 1 was completed in class; grades will be distributed next Wednesday.
- No class / office hours next Monday (Labor Day).
Aug 21: Week 1
- Homework 1 has been released on Piazza as of Friday, Aug 25.
- We also have a new TA, Mozhdeh!
- Please start making teams of 3 for the class project.
Summary
Language models have been the talk of the town ever since OpenAI’s ChatGPT became available to all! However, language models have been studied in Natural Language Processing for decades now, even though NLP has been recently revolutionized by the advancement of large-scale language models achieving state-of-the-art performance across a wide variety of tasks. But what is truly behind this seemingly fantastical technology? This course will cover the fundamentals of modern language modeling, and how they have grown to be the behemoth they are today. Students will gain familiarity with how LMs are constructed, model architectures underlying them as well as get hands-on experience with building and evaluating small-scale LMs. The class will also explore details and variants of the real-world consequences of deploying large-scale LMs, such as the ethics and harms associated with them.
Pre-Requisites
Students are required to have taken CSCI-270 Introduction to Algorithms and Theory of Computing (4.0 units) as well as one of (CSCI-360 Introduction to AI, CSCI-467 Introduction to Machine Learning or equivalent experience). Fluency with python programming is recommended. Please email the instructor for special circumstances or specific clarifications.
Calendar + Syllabus
Also see the Detailed Calendar with slides and readings for all lectures. Equations to follow along in class are shared here.
Introduction to Language Models
Early Neural Language Models
- Aug 30
Sep 4- No Class Labor Day
- Sep 6
- Sep 11
- Sep 13
- Sep 18
- Sep 20
Modern Neural Language Models
- Sep 25
- Sep 27
- Oct 2
-
- Language Grounding : Jesse Thomason Slides
- No Additional Readings
- Oct 4
Oct 9- No Class Indigenous Peoples Day
- Oct 11
-
- Project Discussions
- Progress Report
- Due 10/13; HW3 Released
- Oct 16
Large Language Models (LLMs)
- Oct 18
- Oct 23
- Oct 25
- Oct 30
- Nov 1
-
- Prompting LLMs: Qinyuan Ye Slides
- Readings
- Nov 6
- Nov 8
-
- RLHF: Justin Cho Slides
- Readings
- Nov 13
-
- Project Discussions
- HW4 Due
Outro and Project Presentations
- Nov 15
-
- Putting it all together Slides
- No Additional Readings
- Nov 20
- Project Presentations
Nov 22- No Class Thanksgiving
- Nov 27
- Project Presentations
- Nov 29
- Project Presentations
Dec 4- No Class Study Week
Dec 6- No Class
- Dec 11
-
- Project Final Report
- Due Latest By 10 AM
Calendar is subject to change. More details, e.g. reading materials and additional resources will be added as the semester continues. All work (except the project final report) is due on the specified date by 11:59 PM PT.
Assignments
There will be three components to course grades, see more details.
- Homeworks (40%).
- Quizzes + Class Participation (20%).
- Class Project (40%).
Students are allowed a maximum of 6 late days total for all assignments (NO LATE DAYS ALLOWED FOR quizzes).
Note: Please familiarize yourself with the academic policies and read the note about student well-being.