Universality, diversity and idiosyncrasy
in language technology
CA21167 COST Action

• Task 3.1: Documentation of tools/resources (Noémi Ligeti-Nagy)Slides
• Task 3.2: Evaluation campaign on morphosyntactic parsing (Omer Goldman) Guidelines
• Task 3.3: Conceptions of multilinguality (Adriana Pagano) Slides

We invite proposals for evaluation campaigns to be run as part of EU Unidive Cost Action. Unidive workgroup WG3 “MULTILINGUAL AND CROSS-LINGUAL LANGUAGE TECHNOLOGY” is dedicated to coordinating the development of tools leveraging universality and promoting diversity. According to the memorandum of understanding (MOU), WG3 should coordinate these activities notably by organizing multilingual evaluation campaigns on parsing and MWE identification. However, given that 1) the MOU was written quite some time ago, 2) the first task already put into action in WG3 focuses on morphosyntactic parsing, and 3) the field is advancing rapidly today, we believe that incorporating current technologies, such as large language models (LLMs), into the scope of the proposals is desirable. Unified modeling helps solve NLP tasks with higher accuracy and better awareness of diversity. For these reasons, in this call for evaluation campaigns, we welcome any type of task that aims at different types of idiosyncrasies and diversities, including but not limited to MWE identification, LLMs, idiom identification. We encourage the entire Unidive community (not limited to WG3 members) to come up with task ideas that could be interesting for our network. While it's not a strict requirement, we highly encourage proposals that are based on pilot studies with preliminary data, evaluation measures, and baselines already established. Please feel free to reach out to the WG3 leaders to discuss your idea at your convenience.

The proposers will have the opportunity during the online WG meetings to discuss their ideas with our members. The final written proposals, refined after the online discussion sessions (during WG3 online meetings), will be shared with the WG members. The proposals will be presented in person by the proposers at the 3rd general meeting in Budapest (29-30 January 2025), where the final decision on the selected proposals will be made by the community. The number of evaluation campaigns selected does not have to be limited to one.

The proposers are expected to lead the organization of their proposed evaluation campaign, discuss the possible venues (such as Semeval, SyntaxFest, etc.) for implementing the task, make necessary applications and announcements. To ensure that tasks have sufficient support, we require a minimum of two organizers at the time of the final presentation, with one of them being the proposer. The organizers are expected to provide a draft timetable at the final presentation and possible event venues together with a concrete plan.

• Initial announcement and discussion: 1st of October 2024 (online WG3 meeting)
• Preliminary task proposals due: 12th of November 2024
• Initial discussion with the community: 19th of November 2024 (online WG3 meeting)
• Final presentation: 29th of January 2025 (general meeting, Budapest)
• Task selection notification: February 2025

Next meeting (online): November 19, 2024, 12.00 CET

			Rescheduled: November 27, 12.00 CET

• Joakim Nivre (chair)
• Gülşen Eryiğit (co-chair)
• Amanda Kann
• Adriana Pagano
• Manon Scholivet
• Agata Savary
• Natasha Ringbom
• Csilla Horváth
• Tanja Samardzic
• Louis Estève
• Noémi Ligeti-Nagy
• Federica Iurescia
• Claudia Corbetta
• Alina Wróblewska
• Miryam de Lhoneux
• Dawit J.
• Inguna Skadiņa
• Radovan Garabík
• Ivelina Stoyanova
• Rusudan Makhachashvili
• Federica Gamba
• Omer Goldman
• Tunga Güngör
• Daniel Zeman
• Faruk Mardan
• Martin Benjamin
• Wessel Poelman
• Lucía Amorós-Poveda