Minutes from the tenth WG 3 meeting (online 2024-10-01 12:00 CET)

Reports from Subtask Groups

Task 3.1: Documentation of tools/resources (Noémi Ligeti-Nagy)Slides

Task 3.2: Evaluation campaign on morphosyntactic parsing (Omer Goldman) Guidelines

Task 3.3: Conceptions of multilinguality (Adriana Pagano) Slides

Call for evaluation campaign proposals (Gülşen Eryiğit):

We invite proposals for evaluation campaigns to be run as part of EU Unidive Cost Action. Unidive workgroup WG3 “MULTILINGUAL AND CROSS-LINGUAL LANGUAGE TECHNOLOGY” is dedicated to coordinating the development of tools leveraging universality and promoting diversity. According to the memorandum of understanding (MOU), WG3 should coordinate these activities notably by organizing multilingual evaluation campaigns on parsing and MWE identification. However, given that 1) the MOU was written quite some time ago, 2) the first task already put into action in WG3 focuses on morphosyntactic parsing, and 3) the field is advancing rapidly today, we believe that incorporating current technologies, such as large language models (LLMs), into the scope of the proposals is desirable. Unified modeling helps solve NLP tasks with higher accuracy and better awareness of diversity. For these reasons, in this call for evaluation campaigns, we welcome any type of task that aims at different types of idiosyncrasies and diversities, including but not limited to MWE identification, LLMs, idiom identification. We encourage the entire Unidive community (not limited to WG3 members) to come up with task ideas that could be interesting for our network. While it's not a strict requirement, we highly encourage proposals that are based on pilot studies with preliminary data, evaluation measures, and baselines already established. Please feel free to reach out to the WG3 leaders to discuss your idea at your convenience.

Task Selection

The proposers will have the opportunity during the online WG meetings to discuss their ideas with our members. The final written proposals, refined after the online discussion sessions (during WG3 online meetings), will be shared with the WG members. The proposals will be presented in person by the proposers at the 3rd general meeting in Budapest (29-30 January 2025), where the final decision on the selected proposals will be made by the community. The number of evaluation campaigns selected does not have to be limited to one.

Task Organization

The proposers are expected to lead the organization of their proposed evaluation campaign, discuss the possible venues (such as Semeval, SyntaxFest, etc.) for implementing the task, make necessary applications and announcements. To ensure that tasks have sufficient support, we require a minimum of two organizers at the time of the final presentation, with one of them being the proposer. The organizers are expected to provide a draft timetable at the final presentation and possible event venues together with a concrete plan.

Important dates

Initial announcement and discussion: 1st of October 2024 (online WG3 meeting)

Preliminary task proposals due: 12th of November 2024

Initial discussion with the community: 19th of November 2024 (online WG3 meeting)

Final presentation: 29th of January 2025 (general meeting, Budapest)

Task selection notification: February 2025

Next meeting (online): November 19, 2024, 12.00 CET

Rescheduled: November 27, 12.00 CET

List of Participants

Joakim Nivre (chair)

Gülşen Eryiğit (co-chair)

Amanda Kann

Adriana Pagano

Manon Scholivet

Agata Savary

Natasha Ringbom

Csilla Horváth

Tanja Samardzic

Louis Estève

Noémi Ligeti-Nagy

Federica Iurescia

Claudia Corbetta

Alina Wróblewska

Miryam de Lhoneux

Dawit J.

Inguna Skadiņa

Radovan Garabík

Ivelina Stoyanova

Rusudan Makhachashvili

Federica Gamba

Omer Goldman

Tunga Güngör

Daniel Zeman

Faruk Mardan

Martin Benjamin

Wessel Poelman

Lucía Amorós-Poveda