WG3 12th Meeting, Budapest

Agenda

Report on the shared task on morphosyntactic parsing

Report on the multilinguality survey

Brainstorming about an evaluation campaign in 2026 - Discussions on the two shared task proposals

Coming Shared task on Morphosyntactic Parsing

Task leaders presented the current status of the organization slides

The event page is available from 1st UniDive Shared Task on Morphosyntactic Parsing

Report on the subtask T3.3 Conceptions of multilinguality

Task leaders presented the current status of Task 3.3 slides

Evaluation Campaign Ideas for 2026

Opening the brainstorming session slides

Discussion on the two shared task proposals presented on the previous online meeting and on how to execute them:

PARSEME 2.0

AdmiRE subtask 1 (MWEs vs. static images with/without captions)

AdmiRE subtask 2 (MWEs vs. image sequences with/without captions)

MWE understanding (literal vs. idiomatic disambiguation

Text2MWE generation (given a description, generate a MWE which best describes it)

MWE2text generation (given a MWE in sentence, generate a scenario which explains it)

4 breakout groups to discuss the Shared Tasks (ST):

How to join the two STs?

Unordered List ItemWhich subtasks? How to organize a joint UniDive shared task? How to select or/and merge different tasks? Do you have other ideas of MWE/syntax-oriented tasks which answer the wishes above? How to optimally organize the community effort for these tasks?}}

Unordered List ItemHow to organize the STs in a sequential way?

Unordered List ItemSame questions, but for 2 separate shared tasks

How to \alert{evaluate the models generation}?

How to evaluate generation-oriented tasks? What is the necessary quantity of data? How to avoid contamination of models by the publicly available test data?

How to make the STs more trendy

Unordered List ItemHow to use the shared tasks to shed light on the universality of LLMs? How to make the tasks more interesting for the LLM community? Or rare language phenomena in general?