We develop and analyse computational models of human language. Our work ranges from basic research on algorithms and machine learning to applied research in language technology and computational social science.
Research
Our current focus is on analysing and enhancing neural language models. Specifically, we are working on methods for improving model efficiency, trustworthiness, and usefulness for lesser-resourced languages. We also have a long-standing interest in work on the intersection of natural language processing and theoretical computer science.
We are participating in several national and international research collaborations, including the Wallenberg AI, Autonomous Systems and Software Program (WASP), the EU-funded project TrustLLM – Democratize Trustworthy and Efficient Large Language Model Technology for Europe, and the Swedish Excellence Centre for Computational Social Science (SweCSS).
Current Projects
Efficient Modularisation and Adaptation of Large Language Models
The latest generation of large language models (LLMs) is extremely resource-hungry, to the point where training and even deploying own models has become infeasible for most enterprises and public actors. We propose to tackle this problem by dissecting LLMs into modules that encapsulate functional capabilities such as languages and knowledge retrieval, and can be trained and updated independently of other model parts. We will develop a framework for the targeted adaptation of these modules and evaluate its effectiveness in settings where training data is relatively scarce, focusing on NLP for medium-resource languages such as Swedish.
Funded by WASP
TrustLLM: Democratize Trustworthy and Efficient Large Language Model Technology for Europe
TrustLLM is a European project involving 11 partner organizations. Its main objective is to develop open, trustworthy, and sustainable large language models (LLMs) initially targeting the Germanic languages. This will create the foundation for an advanced open ecosystem for next generation modular and extensible European trustworthy, sustainable, and democratized large language models. Within this project, LiU NLP leads the work package on multilingual training and transfer learning for low-resource languages. Here, we are researching methods to improve LLMs for low-resource languages through e.g. training efficient language adapters, better handling of tokenization, or exploiting typological relatedness for cross-lingual transfer.
Funded by the European Union | Project website
STING: Synthesis and analysis with Transducers and Invertible Neural Generators
Human communication is multimodal and occurs through speech, language, gesture, facial expression, and similar signals. To enable natural interactions with human beings, artificial agents must be capable of analysing and producing these rich and interdependent signals and connecting them to their semantic implications. STING aims to unify synthesis and analysis through transducers and invertible neural models, connecting concrete, continuous-valued sensory data such as images, sound, and motion, with high-level, predominantly discrete representations of meaning. The project has the potential to endow synthesis output with human-understandable high-level explanations while simultaneously improving the ability to attach probabilities to semantic representations.
Funded by WASP
A Practical Theory of Computation for Modern Neural Network Architectures
Today, neural networks are the most important computational paradigm for AI and machine learning. While their computational power has been studied for decades, the theoretical development has not been able to keep up with the recent upshot of new neural architectures, such as the Transformer. Also, existing theoretical results are of limited practical value because they often apply to monolithic architectures, whereas applications typically use networks composed of re-usable components. Moreover, theoretical results tend to make idealising assumptions such as infinite precision or unlimited training time rather than acknowledging that such commodities are in short supply and should thus be treated as parameters of expressive power. In this project, we want to address these shortcomings and develop a practical theory of computation for modern neural network architectures. The project combines methods from theoretical computer science – especially the theory of computation and formal languages – with empirical validation in natural language processing (NLP).
Funded by WASP
Interpreting and Grounding Pre-Trained Representations for Natural Language Processing
Building computers that understand human language is one of the central goals of artificial intelligence. A recent breakthrough on the way towards this goal is the development of neural models that learn deep contextualized representations of language. However, while these models have substantially advanced the state of the art in natural language processing (NLP), our understanding of the learned representations and our repertoire of techniques for integrating them with other knowledge representations and reasoning facilities remain severely limited. To address these gaps, we will develop new methods for interpreting, grounding, and integrating deep contextualized representations of language and evaluate the usefulness of these methods in the context of threat intelligence applications together with our industrial partner.
Funded by WASP
Selected Recent Publications
- Ehsan Doostmohammadi, Oskar Holmström, and Marco Kuhlmann. 2024. How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?. arXiv:2402.10770.
- Niklas Wretblad, Fredrik Gordh Riseby, Rahul Biswas, Amin Ahmadi, and Oskar Holmström. 2024. Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark. arXiv:2402.12243.
- Jenny Kunz and Oskar Holmström. 2024. The Impact of Language Adapters in Cross-Lingual Transfer for NLU. In Raúl Vázquez, Timothee Mickus, Jörg Tiedemann, Ivan Vulić, and Ahmet Üstün, editors, Proceedings of the 1st Workshop on Modular and Open Multilingual NLP (MOOMIN 2024), pages 24–43, St Julians, Malta. Association for Computational Linguistics.
- Ehsan Doostmohammadi, Tobias Norlund, Marco Kuhlmann, and Richard Johansson. 2023. Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 521–529, Toronto, Canada. Association for Computational Linguistics.
Teaching
Our teaching portfolio comprises courses and degree projects in natural language processing and text mining at the basic, advanced, and doctoral levels.
Current Courses
AI for Natural Language (Bachelor)
Writing computer programs that understand and produce human language is a central goal of artificial intelligence. This course aims to provide an understanding of how far we are from this goal: what is and what is not possible today. Students will learn about current methods in natural language processing, about what resources are required to build language technology systems, and about how one can assess the quality of these methods and systems.
Language & Computers (Bachelor)
This course aims to provide an introduction to basic linguistic concepts and to teach techniques for collecting, analysing and validating linguistic data sets using computers. The focus of the course is on linguistic data in text form.
Language Technology (Bachelor)
Language technology – technology for the analysis and interpretation of natural language – forms a key component of smart search engines, personal digital assistants, and many other innovative applications. The goal of this course is to give an introduction to language technology as an application area, as well as to its basic methods. The course focuses on methods that process text.
Natural Language Processing (Master)
Natural Language Processing (NLP) develops methods for making human language accessible to computers. This course aims to provide students with a theoretical understanding of and practical experience with the advanced algorithms that power modern NLP. The course focuses on methods based on deep neural networks.
Text Mining (Master)
Text Mining is about distilling actionable insights from text. The overall aim of this course is to provide students with practical experience of the main steps of text mining: information retrieval, processing of text data, modelling, analysis of experimental results. The course ends with an individual project where students work on a self-defined problem.
Deep Learning for NLP (PhD)
Natural Language Processing (NLP) develops methods for making human language accessible to computers. The goal of this course is to provide students with a theoretical understanding of and practical experience with the advanced algorithms that power modern NLP. The course focuses on methods based on deep neural networks.
This course is given in the context of the WASP Graduate School.
Open Thesis Proposals
Language Adaption of LLMs
Parameter-efficient fine-tuning techniques are increasingly popular as LLMs become larger and more powerful. They not only make the fine-tuning more cost-efficient and climate-friendly but also help preserve the models’ general capabilities and avoid catastrophic forgetting. In this project, you implement and compare different approaches to adapting a model to specific languages and evaluate them on multilingual datasets for different tasks and domains. You will join an ongoing project in our group and help us to improve our results and/or get new insights on where such methods are (not) useful.
You are welcome to contact us to discuss specific contribution. It could be along these lines:
- Finding the optimal adaption setup for a specific language or task.
- An in-depth evaluation of a language adaption method, including testing modifications from the literature or proposed by us or you.
- Developing a dataset in a lower-resourced language that tests for factual or commonsense knowledge associated with an associated culture.
Contact: Jenny Kunz
Fine-grained Language Identification
Identifying which language a text is written in sounds like a solved problem: most translation tools, for example, can automatically detect the language of the source text. However, language identification methods only tend to work reliably on longer inputs, such as documents or full sentences. There are many situations in which we might want to identify languages on a more fine-grained level, for example:
- In code-switching, bilingual speakers mix expressions from two languages within a single utterance.
- In datasets gathered from websites like Wikipedia, sentences often contain titles or names in other languages.
In this project, you will apply modern algorithms and machine learning techniques to try to improve language identification on a word or phrase level in mixed-language data. There are several ways you can approach this project:
- You could focus on disambiguating a specific pair of languages in code-switched text.
- You could focus on identifying spans in a common, high-resource language (such as English or Spanish) within texts of other, lower-resourced languages (such as Jamaican Creole or Northern Sámi).
- You could focus on disambiguating closely related languages (such as Danish and Norwegian) or dialects, which is particularly challenging as they will also tend to look very similar.
Contact: Marcel Bollmann
Team
Professor, Group Lead
Associate Professor
Postdoc
PhD Student
PhD Student
PhD Student
PhD Student
PhD Student
PhD Student
PhD Student
Visiting Student