Unit 1: Tokenisation and embeddings
This unit covers tokenisation and embeddings, two fundamental concepts of modern NLP. Tokenisers split text into smaller units such as words, subwords, or characters. Embeddings are fixed-size vector representations of tokens that can be learned from data and optimised for different tasks.
Lectures
The lectures start with traditional word-based tokenisation and then present the Byte Pair Encoding (BPE) algorithm, which is used by most current language models. In the second half of the lectures, you will learn about embeddings and different methods for how they can be learned.
| Section | Title | Video | Slides | Quiz |
|---|---|---|---|---|
| 1.1 | Introduction to tokenisation | video | slides | quiz |
| 1.2 | The Byte Pair Encoding algorithm | video | slides | quiz |
| 1.3 | Tokenisation fairness | video | slides | quiz |
| 1.4 | Introduction to embeddings | video | slides | quiz |
| 1.5 | Word embeddings | video | slides | quiz |
| 1.6 | The skip-gram model | video | slides | quiz |
To earn a wildcard for this unit, you must complete the quizzes on the day before the online meeting.
Online meeting
In the online meeting for this unit, we will discuss the issue of bias in word embeddings. We will examine how biases arise, how to detect and measure them, and what mitigation strategies exist – along with their trade-offs and limits. These issues have broad consequences for real-world uses of NLP systems.
The meeting took take place on 2026-02-04.
Additional materials
Lab
In lab 1, you will build an understanding of how text can be transformed into representations that computers can process and learn from. Specifically, you will code and analyse a tokeniser based on the Byte Pair Encoding (BPE) algorithm, and then explore embeddings in the context of a simple text classifier architecture.