We invite proposals for evaluation campaigns to be run as part of EU Unidive Cost Action. Unidive workgroup WG3 “MULTILINGUAL AND CROSS-LINGUAL LANGUAGE TECHNOLOGY” is dedicated to coordinating the development of tools leveraging universality and promoting diversity. According to the memorandum of understanding (MOU), WG3 should coordinate these activities notably by organizing multilingual evaluation campaigns on parsing and MWE identification. However, given that 1) the MOU was written quite some time ago, 2) the first task already put into action in WG3 focuses on morphosyntactic parsing, and 3) the field is advancing rapidly today, we believe that incorporating current technologies, such as large language models (LLMs), into the scope of the proposals is desirable. Unified modeling helps solve NLP tasks with higher accuracy and better awareness of diversity. For these reasons, in this call for evaluation campaigns, we welcome any type of task that aims at different types of idiosyncrasies and diversities, including but not limited to MWE identification, LLMs, idiom identification. We encourage the entire Unidive community (not limited to WG3 members) to come up with task ideas that could be interesting for our network. While it's not a strict requirement, we highly encourage proposals that are based on pilot studies with preliminary data, evaluation measures, and baselines already established. Please feel free to reach out to the WG3 leaders to discuss your idea at your convenience.
The proposers will have the opportunity during the online WG meetings to discuss their ideas with our members. The final written proposals, refined after the online discussion sessions (during WG3 online meetings), will be shared with the WG members. The proposals will be presented in person by the proposers at the 3rd general meeting in Budapest (29-30 January 2025), where the final decision on the selected proposals will be made by the community. The number of evaluation campaigns selected does not have to be limited to one.
The proposers are expected to lead the organization of their proposed evaluation campaign, discuss the possible venues (such as Semeval, SyntaxFest, etc.) for implementing the task, make necessary applications and announcements. To ensure that tasks have sufficient support, we require a minimum of two organizers at the time of the final presentation, with one of them being the proposer. The organizers are expected to provide a draft timetable at the final presentation and possible event venues together with a concrete plan.
Next meeting (online): November 19, 2024, 12.00 CET
Rescheduled: November 27, 12.00 CET