Q&A from the last lecture

Regarding the project

How in-depth should the project plan go?

The project plan is for me to check that what you are planning to do is appropriate for the course project, and to give you feedback in case I see any problems with your plan, so that there are no bad surprises later on. For that, you should ideally list concrete models, datasets, etc. that you plan to use. However, you don’t have to write it like a report – it’s not part of the grading, it’s just to make sure your plan is reasonable and appropriate!

Standard datasets and benchmarks have already been tested on many models, so what’s the purpose of this project?

The purpose is not to simply find out “which models perform best”, as this has indeed most likely been tested already! The purpose is to understand more about how these benchmarks work, for example by:

• Taking a closer look at the datasets – Do they look like you would have expected? Is there anything surprising about them? How difficult do you find the task as a human, based on what you see in the dataset?
• Understanding how to run the benchmark yourself – What do you need to implement? What software packages can you use? What models make sense to test on this benchmark test?
• Taking a closer look at the evaluation – Which metric is being used to evaluate this benchmark? How good is this metric at capturing “performance” on the task? Where do the models still make errors and why could that be?

Do we need to run an experiment if we do the default project?

Yes, you should! If you look at the project instructions, Section 1.2 says “you should do your own implementation of the chosen task and evaluate its results”. See also the previous question about the purpose of the project – running your own (small-scale) experiments is part of this.

When will you publish the presentation schedule?

I plan to do this early next week (week 9) after I have reviewed your project plans.

Regarding the exam

How will the exam be structured?

I added some information on the new exam page now.

Are there any old or mock exams to practice on?

Unfortunately no, since the course contents changed so much this year, and I haven’t finalized the exam yet. However, to give you something to practice on, I have compiled some practice questions already now that you can find on the exam page. These are questions that will be similar to those in Part A of the exam.

Can everything from the lectures be on the exam? Which parts are being tested?

Yes, exam questions can be about any part of the lectures. I recommend paying particular attention to the concepts listed on the “important concepts” slides of the lectures – if you know and can explain (in basic terms) what these mean, you should have no problems in the exam. For practical math tasks that might be on the exam, the practice questions will give you a good idea of what to expect.

How many questions are there in total? Multiple-choice vs. analytical questions?

I haven’t finalized the exact number of questions yet. Part A may include multiple-choice questions and other, simpler tasks (e.g. math-related), whereas analytical questions will mostly be in Part B (see the exam page).