Unit 3: Developing LLMs
In this unit, you will get an overview of different issues related to the development of large language models. In particular, the unit covers training strategies, the issue of data, emergent abilities of LLMs, and LLM alignment. It also discusses scaling laws, an important empirical finding about the relationship between the size and performance of large language models.
| Section | Title | Video | Slides | Quiz |
|---|---|---|---|---|
| 3.1 | Introduction to LLM development | video | slides | quiz |
| 3.2 | Training LLMs | video | slides | quiz |
| 3.3 | Data for LLM pretraining | video | slides | quiz |
| 3.4 | Scaling laws | video | slides | quiz |
| 3.5 | Emergent abilities of LLMs | video | slides | quiz |
| 3.6 | LLM alignment | video | slides | quiz |
Lab
Lab 3 is about pretraining large language models. You will work through the full pretraining process for a GPT model, explore different settings, and implement optimisations that make training more efficient. You will also reflect on the impact of data curation on the quality of the pretrained model. By the end of the lab, you will have a solid understanding of how large language models are trained from scratch.