Universality, diversity and idiosyncrasy
in language technology
CA21167 COST Action

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
==== WG3 12th Meeting, Budapest  ====

===Agenda===
  * Report on the shared task on morphosyntactic parsing
  * Report on the multilinguality survey
  * Brainstorming about an evaluation campaign in 2026 -  Discussions on the two shared task proposals

===Coming Shared task on Morphosyntactic Parsing===
  * Task leaders presented the current status of the organization {{:wg3:T3.2budapest.pdf|slides}}
  * The event page is available from [[other-events:msp|1st UniDive Shared Task on Morphosyntactic Parsing]]

===Report on the subtask T3.3 Conceptions of multilinguality===
  * Task leaders presented the current status of Task 3.3 {{:wg3:unidive_t3.3_budapest_20250130.pdf|slides}}
===Evaluation Campaign Ideas for 2026=== 
  * Opening the brainstorming session {{:wg3:secondevalcampdiscussion.pdf|slides}}

== Discussion on the two shared task proposals presented on the previous online meeting and on how to execute them: ==

  * PARSEME 2.0
  * AdmiRE subtask 1 (MWEs vs. static images with/without captions)
  * AdmiRE subtask 2 (MWEs vs. image sequences with/without captions)
  * MWE understanding (literal vs. idiomatic disambiguation
  * Text2MWE generation (given a description, generate a MWE which best describes it)
  * MWE2text generation (given a MWE in sentence, generate a scenario which explains it)


==4 breakout groups to discuss the Shared Tasks (ST):==

  * How to join the two STs?
      * Unordered List ItemWhich subtasks? How to organize a joint UniDive shared task? How to select or/and merge different tasks? Do you have other ideas of MWE/syntax-oriented tasks which answer the wishes above? How to optimally organize the community effort for these tasks?}}
  * Unordered List ItemHow to organize the STs in a sequential way?
      * Unordered List ItemSame questions, but for 2 separate shared tasks
  * How to \alert{evaluate the models generation}?
      * How to evaluate generation-oriented tasks? What is the necessary quantity of data? How to avoid contamination of models by the publicly available test data?
  * How to make the STs more trendy
      * Unordered List ItemHow to use the shared tasks to shed light on the universality of LLMs? How to make the tasks more interesting for the LLM community? Or rare language phenomena in general?
    

==Closure==
  * WG3 will continue its activities and provide support for both of the proposed evaluation campaigns. This approach ensures that all relevant aspects of the project are addressed while leveraging the group's expertise effectively. Moving forward, WG3 will coordinate efforts to implement these tasks efficiently and will provide updates on progress in the upcoming meetings.