Week 4
Dear all,
We are almost done with Unit 3, and it is time for my newsletter.
Do you have any questions about the lectures, lab, or project? Feel free to book a short meeting with me! Schedule a 15-min chat
Best, Marco
This week: Pretraining
Unit 3 has provided an overview of key issues in the development of large language models, with a focus on the pretraining stage. In particular, the unit covers data, scaling laws, and the impact of LLMs on the environment. In the lab, you work through the full pretraining process for your GPT model, explore different settings, and implement optimisations that make training more efficient. I hope the lab has given you a solid understanding of how large language models are trained from scratch.
Project ideas and group formation
As we approach the end of the lab phase, it is time to begin planning your project. If you still need inspiration, revisit the project ideas on the Project page – or ask your favourite AI assistant. Once you have a project idea, do not hesitate to book 15 minutes with me to get some feedback (booking link). If you still need a group to work with, we will have a third and final group-matching session during Tuesday’s seminar.
Next week: Alignment and current research
In the final unit of the course, you will learn more about the alignment stage of LLM training. You will also see several examples of current research in this and related areas. As most of you will already have noticed, the material for this unit is a bit different: it features both lectures (as in the previous units), lecture-style reviews of recent developments, and videos from research presentations.