Link Search Menu Expand Document

CSCI-499 Fall 2023: Language Models in NLP

🍂 Fall 2023     ⏰ Mon / Wed 10:00 am - 11:50 am     📍 ZHS 360

Instructor: Swabha Swayamdipta

Office Hours: Mondays 1-2pm; SAL 238

Teaching Assistant: Avi Thawani

Office Hours: Wednesdays 4-5pm; Leavey Library Lower Level

Teaching Assistant: Mozhdeh Gheini

Office Hours: Fridays 2-3pm; SAL Common Area


Nov 1: Week 12

  • Quiz 6 was conducted in class.

Oct 30: Week 11

  • Progress Report Feedback and grades are out.
  • Quiz 6 on Wednesday, 11/1.

Oct 16: Week 9

  • Graded Quiz 4 sheets were distributed in class.
  • Please sign up for Final Paper Presentations (see Piazza post).
  • Quiz 5 on Monday, 10/23.

Sep 27: Week 6

  • Graded Quiz 3 sheets were distributed in class.
  • Swabha is OOT next week, but we will have guest / TA lectures + Quiz 4 next week.

Sep 20: Week 5

  • Quiz 3 was completed in class; grades will be shared next Wednesday. Future quiz dates will be announced in class (one class ahead).
  • Project Proposals were graded and feedback shared via email.

Sep 11: Week 4

  • Quiz 2 was completed in class; grades will be distributed next Wednesday.
  • HW2 released on Friday, 9/15 and due by 9/29.

Sep 6: Week 3

  • HW1 deadline is now extended by 24hrs; due on 9/7 by 11:59PM PT.

Aug 30: Week 2

  • Quiz 1 was completed in class; grades will be distributed next Wednesday.
  • No class / office hours next Monday (Labor Day).

Aug 21: Week 1

  • Homework 1 has been released on Piazza as of Friday, Aug 25.
  • We also have a new TA, Mozhdeh!
  • Please start making teams of 3 for the class project.


Language models have been the talk of the town ever since OpenAI’s ChatGPT became available to all! However, language models have been studied in Natural Language Processing for decades now, even though NLP has been recently revolutionized by the advancement of large-scale language models achieving state-of-the-art performance across a wide variety of tasks. But what is truly behind this seemingly fantastical technology? This course will cover the fundamentals of modern language modeling, and how they have grown to be the behemoth they are today. Students will gain familiarity with how LMs are constructed, model architectures underlying them as well as get hands-on experience with building and evaluating small-scale LMs. The class will also explore details and variants of the real-world consequences of deploying large-scale LMs, such as the ethics and harms associated with them.


Students are required to have taken CSCI-270 Introduction to Algorithms and Theory of Computing (4.0 units) as well as one of (CSCI-360 Introduction to AI, CSCI-467 Introduction to Machine Learning or equivalent experience). Fluency with python programming is recommended. Please email the instructor for special circumstances or specific clarifications.

Calendar + Syllabus

Also see the Detailed Calendar with slides and readings for all lectures. Equations to follow along in class are shared here.

Introduction to Language Models

Aug 21
Introduction and Course Overview   Slides
No Additional Readings
Aug 23
n-gram Language Models I   Slides
HW1 Released 8/25
Aug 28
n-gram Language Models II   Slides

Early Neural Language Models

Aug 30
Word Embeddings I   Slides
Sep 4
No Class   Labor Day
Sep 6
Word Embeddings II   Slides
HW1 Due 9/7
Sep 11
Logistic Regression   Slides
Sep 13
Logistic Regression II   Slides   Proposal
HW2 Released 9/15
Sep 18
Feedforward Neural Nets and Backprop   Slides
Sep 20
Recurrent Neural Network LMs   Slides

Modern Neural Language Models

Sep 25
Sequence-To-Sequence and Attention   Slides
Sep 27
Transformers - Building Blocks   Slides
HW2 Due 9/29
Oct 2
Language Grounding : Jesse Thomason   Slides
No Additional Readings
Oct 4
PyTorch for Transformers [TA Lecture]   Slides
Oct 9
No Class   Indigenous Peoples Day
Oct 11
Project Discussions
Progress Report
Due 10/13; HW3 Released
Oct 16
Transformers - Building Blocks II   Slides

Large Language Models (LLMs)

Oct 18
Pre-training Transformers   Slides
Oct 23
Pre-training Transformers II   Slides
Oct 25
Generating from Language Models   Slides
HW3 Due 10/27
Oct 30
Generating from Language Models II   Slides
HW4 Released
Nov 1
Prompting LLMs: Qinyuan Ye   Slides
Nov 6
LLMs: Limitations and Harms   Slides
Nov 8
RLHF: Justin Cho   Slides
Nov 13
Project Discussions
HW4 Due

Outro and Project Presentations

Nov 15
Putting it all together   Slides
No Additional Readings
Nov 20
Project Presentations
Nov 22
No Class   Thanksgiving
Nov 27
Project Presentations
Nov 29
Project Presentations
Dec 4
No Class   Study Week
Dec 6
No Class  
Project Final Report

Calendar is subject to change. More details, e.g. reading materials and additional resources will be added as the semester continues. All work is due on the specified date by 11:59 PM PT.


There will be three components to course grades, see more details.

Students are allowed a maximum of 6 late days total for all assignments (NO LATE DAYS ALLOWED FOR quizzes).

Note: Please familiarize yourself with the academic policies and read the note about student well-being.