Project
The second half of the NLP course is dedicated to a small research project. The main purpose of this is to give you the opportunity to identify, assess, and use NLP research literature (learning outcome 4). You will also deepen the knowledge you have acquired in the other parts of the course.
Basic information about the project
Schedule
Here is the schedule for the main activities during the project part of the course:
Project ideas
Here is a list of project ideas. For each idea, I also identify some of the challenges I would expect for the project in question. You can modify any project idea to your liking or propose an entirely new project.
Fine-tuning a pretrained language model
- Goal and Ideas
- Fine-tune an open-source LLM like BERT or GPT (e.g., from Hugging Face) for a specific downstream task, such as sentiment analysis on a specialised domain, text summarisation for a niche dataset (e.g., scientific papers), or named entity recognition for a custom dataset.
- Challenges
- Finding, curating and cleaning the dataset. Handling class imbalance. Avoiding overfitting during fine-tuning.
Retrieval-augmented generation
- Goal and Ideas
- Build and evaluate a retrieval-augmented generation system using libraries such as Haystack or LangChain. Connect data sources such as PDF collections. Experiment with different prompting schemes to improve the quality of the retrieved content.
- Challenges
- Finding appropriate libraries. Finding a way to evaluate the system.
Bias and fairness in NLP models
- Goal and Ideas
- Analyse biases in pretrained LLMs and suggest ways to mitigate them. For example, you could investigate gender or racial bias in text classification or generation task. There are established metrics (e.g., WEAT), and you could evaluate interventions like counterfactual data augmentation.
- Challenges
- Finding datasets to test biases and ensuring fair comparisons of mitigation strategies.
Tweaking the GPT implementation
- Goal and Ideas
- Experiment with tweaks to our implementation of GPT to search for the fastest way to train the model to a target loss value on the FineWeb-EDU validation set. You can take inspiration from Modded-NanoGPT. If you take on this project, you get three runs on an 8xA100 node on BerzeLiUs.
- Challenges
- Debugging the implementation and scaling to a GPU cluster.
Data augmentation for low-resource NLP
- Goal and Ideas
- Investigate the effectiveness of data augmentation techniques in low-resource settings. For example, you could implement and evaluate the impact of back-translation, paraphrasing, or synonym replacement to augment training data for a classification task.
- Challenges
- Implementing augmentation pipelines and ensuring they produce meaningful improvements.
Explainability for LLMs
- Goal and Ideas
- Implement and evaluate explainability techniques for predictions from LLMs. For example, you could use attention visualisation or saliency maps to explain a model’s predictions for text classification and compare post-hoc explanation methods (e.g., LIME) with inherent interpretability methods (e.g., Transformer Lens).
- Challenges
- Making explanations clear and evaluating the quality of explanations quantitatively or qualitatively.
Text-to-image or image-to-text tasks
- Goal and Ideas
- Combine NLP with computer vision for multi-modal tasks. For example, you could fine-tune CLIP and combine it with a language model for domain-specific image–text retrieval or investigate caption generation for domain-specific datasets (e.g., medical images).
- Challenges
- Handling multi-modal datasets and evaluating results effectively.
Projects from previous years
Take a look at the topics that have been explored in the course in previous years:
Deliverables
The main deliverables you will produce for the project are the group presentation at the course conference and the individual post-project paper.
For examples of group presentations, see the slides linked from the programme of the Course conference 2025.
For examples of post-project papers, click here.