Differences

This shows you the differences between two versions of the page.

--- wg3:wg3_meeting_2025-03-12_edit [2025/03/12 11:39] – gulsen.eryigit
+++ wg3:wg3_meeting_2025-03-12_edit [2025/03/12 11:40] (current) – gulsen.eryigit
@@ Line 55: / Line 55: @@
 ====== PARSEME shared task (Manon, Agata) ======
-subtask 1 (PARSEME 2.0)
+  * subtask 1 (PARSEME 2.0)
-quite established framework
+  * quite established framework
-novelty: non-verbal MWEs, diversity measures
+  * novelty: non-verbal MWEs, diversity measures
-subtask 2 (MWE generation)
+  * subtask 2 (MWE generation)
-given a context with eliminated MWEs, restore this MWE
+  * given a context with eliminated MWEs, restore this MWE
-Problems: how to evaluate the system
+  * Problems: how to evaluate the system
-[ALINE] Consider taking into account the level of difficulty of the items? For example, some items will be more ambiguous and more difficult to determine
+  * [ALINE] Consider taking into account the level of difficulty of the items? For example, some items will be more ambiguous and more difficult to determine
-[JOAKIM] It is unclear which capacity of models we test
+  * [JOAKIM] It is unclear which capacity of models we test
-[TOM] Very difficult to evaluate, even manually.
+  * [TOM] Very difficult to evaluate, even manually.
-subtask 3 (MWE comprehension/disambiguation)
+  * subtask 3 (MWE comprehension/disambiguation)
-Given a sentence and a span of a potential idiomatic expressions, classify it as idiomatic, literal or coincidental
+  * Given a sentence and a span of a potential idiomatic expressions, classify it as idiomatic, literal or coincidental
-[GULSEN] There are some datasets for this task. Maybe the 3rd category complicates the things.
+  * [GULSEN] There are some datasets for this task. Maybe the 3rd category complicates the things.
-[JOAKIM]
+  * [JOAKIM]
-[TOM] The same as SemEval 2022 (EN, PT, Galician). There are artefact issues (the models don’t really pay attention to the context).
+  * [TOM] The same as SemEval 2022 (EN, PT, Galician). There are artefact issues (the models don’t really pay attention to the context).
-subtask 4 (paraphrasing)
+  * subtask 4 (paraphrasing)
-Given a sentence, rephrase it so that there are no MWEs
+  * Given a sentence, rephrase it so that there are no MWEs
-[AGATA] The input should be raw text, without a span. Objective: simplification of a text.
+  * [AGATA] The input should be raw text, without a span. Objective: simplification of a text.
-[JOAKIM] The most natural tasks among (2, 3 and 4). Close to what people do with LLMs.
+  * [JOAKIM] The most natural tasks among (2, 3 and 4). Close to what people do with LLMs.
-Can we avoid doing manual evaluation? (LLM as judge)
+  * Can we avoid doing manual evaluation? (LLM as judge)
-[TOM] His favorite
+  * [TOM] His favorite
-[ALINE] They work with questionnaires for humans for this problem. There is a synonym dataset. Another task: collect sentences with synonyms of MWEs.
+  * [ALINE] They work with questionnaires for humans for this problem. There is a synonym dataset. Another task: collect sentences with synonyms of MWEs.
-[ALINE] Sometimes the simplest way to express a meaning is with a MWE.
+  * [ALINE] Sometimes the simplest way to express a meaning is with a MWE.
-Questions:
+  * Questions:
-Which subtasks to choose?
+  * Which subtasks to choose?
-How to evaluate them?
+  * How to evaluate them?
 ====== AdMIRe extension ======
-  * Tom’s [[]][[https://docs.google.com/presentation/d/1PLeZfHiZeU7NY8BS6AmnEunnsPsk_MOwucOSzYusBD8/edit?usp=sharing|slides]]
+  * Tom’s [[https://docs.google.com/presentation/d/1PLeZfHiZeU7NY8BS6AmnEunnsPsk_MOwucOSzYusBD8/edit?usp=sharing|slides]]
-  * [[]][[https://semeval2025-task1.github.io/|Task website]]
+  * [[https://semeval2025-task1.github.io/|Task website]]
-  *  [[guidelines & notes]][[https://docs.google.com/document/d/1Suor8arKN5Npg9I4LEqpCma6p_k9vo3ZilioPltXtdA/edit?tab=t.0#heading=h.109xvas7yti|Data curation]]
+  *  [[https://docs.google.com/document/d/1Suor8arKN5Npg9I4LEqpCma6p_k9vo3ZilioPltXtdA/edit?tab=t.0#heading=h.109xvas7yti|Data curation guidelines & notes]]