Discussion on the two shared task proposals presented on the previous online meeting and on how to execute them:
PARSEME 2.0
AdmiRE subtask 1 (MWEs vs. static images with/without captions)
AdmiRE subtask 2 (MWEs vs. image sequences with/without captions)
MWE understanding (literal vs. idiomatic disambiguation
Text2MWE generation (given a description, generate a MWE which best describes it)
MWE2text generation (given a MWE in sentence, generate a scenario which explains it)
4 breakout groups to discuss the Shared Tasks (ST):
How to join the two STs?
Unordered List ItemWhich subtasks? How to organize a joint UniDive shared task? How to select or/and merge different tasks? Do you have other ideas of MWE/syntax-oriented tasks which answer the wishes above? How to optimally organize the community effort for these tasks?}}
Unordered List ItemHow to organize the STs in a sequential way?
Unordered List ItemSame questions, but for 2 separate shared tasks
How to \alert{evaluate the models generation}?
How to evaluate generation-oriented tasks? What is the necessary quantity of data? How to avoid contamination of models by the publicly available test data?
How to make the STs more trendy
Unordered List ItemHow to use the shared tasks to shed light on the universality of LLMs? How to make the tasks more interesting for the LLM community? Or rare language phenomena in general?
Closure
WG3 will continue its activities and provide support for both of the proposed evaluation campaigns. This approach ensures that all relevant aspects of the project are addressed while leveraging the group's expertise effectively. Moving forward, WG3 will coordinate efforts to implement these tasks efficiently and will provide updates on progress in the upcoming meetings.