Word representations
To process words using neural networks, we need to represent them as vectors. In this unit, you will see different methods for learning these representations from data. The unit also introduces the idea of subword representations, including character-level representations.
Lectures
At the beginning of this unit, we introduce the idea of word representations: the basic building block that we use to build deep learning models that process language. Specifically, we look at word embeddings and how they can be learned from data using matrix factorisation methods and neural networks. We also discuss some challenges in representing words that can be solved by working with subword units. In the last lecture, we introduce the concept of contextualised embeddings, which recognise the varying meanings of words across different contexts.
| Section | Title | Video | Slides | Quiz |
|---|---|---|---|---|
| 2.1 | Introduction to word representations | video | slides | quiz |
| 2.2 | Word embeddings via matrix factorisation | video | slides | quiz |
| 2.3 | Word embeddings via neural networks | video | slides | quiz |
| 2.4 | The skip-gram model | video | slides | quiz |
| 2.5 | Subword models | video | slides | quiz |
| 2.6 | Contextualised word embeddings | video | slides | quiz |
Lab
In this lab you will implement the skip-gram model with negative sampling (SGNS) from Lecture 2.4, and use it to train word embeddings on the text of the Simple English Wikipedia. In the exploratory part of the lab, you will visually inspect the trained embeddings to learn more about what similarity means in embedding space.