Differences

This shows you the differences between two versions of the page.

--- wg3:wg3_meeting_2023-09-08 [2023/09/20 16:44] – [Documentation of Multilingual Language Technology Tools] joakim.nivre
+++ wg3:wg3_meeting_2023-09-08 [2023/09/26 10:22] (current) – [Multilingual Evaluation Campaigns] joakim.nivre
@@ Line 1: / Line 1: @@
 ===== WG3 2nd Meeting Minutes – 2023-09-08 =====
-The second WG meeting was mainly devoted to the two tasks/deliverables outlined in the MoU, documentation of multilingual language tools and multilingual evaluation campaigns. In addition, we discussed ways of working in the WG.
+The second WG meeting was mainly devoted to the two tasks/deliverables outlined in the MoU, documentation of multilingual language tools and organization of multilingual evaluation campaigns. In addition, we discussed ways of working in the WG.
 ==== Documentation of Multilingual Language Technology Tools ====
@@ Line 15: / Line 15: @@
 There was a lively discussion covering many topics but focusing on the choice of documentation platform and the scope of the documentation task. Here is a summary of the main points and some of the issues raised in relation to them:
-   * It makes sense to use on an existing platform because (a) we don't have the resources to develop our own platform (or maintain it after the end of the action) and (b) we don't need to start from scratch because many tools will already be documented.
+   * It makes sense to use on an existing platform for several reasons.
-   * Of the available platforms, ELG seems to be the best fit for what we want to do, although all platforms have pros and cons. For example, while ELG harvests information other platforms and sources, it does not provide direct access to tools and resources in the same way that CLARIN does.
+      * We don't have the resources to develop our own platform (or maintain it after the end of the action).
+      * We don't need to start from scratch because many tools will already be documented.
+   * Of the available platforms, ELG seems to be the best fit for what we want to do, although all platforms have pros and cons. For example:
+      * ELG harvests information from other platforms and sources, but it does not provide direct access to tools and resources in the same way that CLARIN does.
+      * It is not clear how ELG will be maintained in the future.
    * We need to define (and possibly narrow) the scope of our documentation task along several dimensions, including the following:
-      * What is meant by multilinguality? Do we include only strictly multilingual tools, or do we want to document monolingual tools for many languages?
+      * What is meant by multilinguality?
-      * What languages do we include? ELG focuses on languages in Europe, but could easily be extended. In practice, we may be limited by the expertise available within the action. How do we deal with dialects and language variants?
+         * Do we include only strictly multilingual tools, or do we want to document monolingual tools for many languages?
+      * What languages do we include?
+         * ELG focuses on languages in Europe, but can easily be extended.
+         * In practice, we may be limited by the expertise available within the action.
+         * How do we deal with dialects and language variants?
       * How do we take users into account?
       * Do we restrict ourselves to tools or do we also include (other) resources?
       * Should we try to integrate typology resources? How?
+The WG leaders will call a meeting with the task leaders to discuss next steps. Anyone else who would like to be involved in this task is welcome to contact the task leaders. The goal is to select a platform and define the scope of the task before the first general meeting in 2024.
+==== Multilingual Evaluation Campaigns ====
+The afternoon session was organized by the WG3 leaders and featured short invited presentations from WG members who had submitted ideas for evaluation campaigns:
+   * D. Zeman: UD Shared Task Inside Out ({{ :wg3:EvalCampaign_Dan.pdf|abstract}})
+   * A. Wróblewska: NLPre Benchmarking Platform ({{ :wg3:EvalCampaign_Alina.pdf|abstract}})
+   * O. Goldman, L. Weissweiler and R. Tsarfaty: Morpho-Syntactic Analyso-Parsing ({{ :wg3:EvalCampaign_Omer.pdf|abstract}}, {{ :wg3:EvalCampaign_Omer_Slides.pdf|slides}})
+   * T. Samardzic: Subword Pooling ({{ :wg3:EvalCampaign_Tanja.pdf|slides}})
+The proposals were discussed in breakout groups, followed by a general plenary discussion where several other ideas for evaluation campaigns were proposed as well. The WG leaders will call a meeting with the proposers to discuss next steps. Anyone else who would like to be involved in this task is welcome to contact the WG leaders. The goal is to organize at least one evaluation campaign during 2024 and a different one during 2025.
+==== Ways of Working ====
+In a short session after lunch, we discussed how to organize the work going forward. Two decisions were made:
+   * Following other WGs, we will organize regular online meetings for exchange of information, where task groups can report on their progress.
+   * The importance of information flow between different WGs was emphasised, and the WG leaders were tasked with proposing a system of WG liaisons to the other WG leaders. The role of WG liaisons is to report on work in other WGs at our meetings (and vice versa).