publications

2024

1. arXiv

   Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals

   João A. Leite, Olesya Razuvayevskaya, Kalina Bontcheva, and 1 more author

   Oct 2024

   HTML PDF

   
2. CIKM

   EUvsDisinfo: A Dataset for Multilingual Detection of Pro-Kremlin Disinformation in News Articles

   João A. Leite, Olesya Razuvayevskaya, Kalina Bontcheva, and 1 more author

   In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, Boise, ID, USA, Oct 2024

   Abs DOI PDF

   

   This work introduces EUvsDisinfo, a multilingual dataset of disinformation articles originating from pro-Kremlin outlets, along with trustworthy articles from credible / less biased sources. It is sourced directly from the debunk articles written by experts leading the EUvsDisinfo project. Our dataset is the largest to-date resource in terms of the overall number of articles and distinct languages. It also provides the largest topical and temporal coverage. Using this dataset, we investigate the dissemination of pro-Kremlin disinformation across different languages, uncovering language-specific patterns targeting certain disinformation topics. We further analyse the evolution of topic distribution over an eight-year period, noting a significant surge in disinformation content before the full-scale invasion of Ukraine in 2022. Lastly, we demonstrate the dataset’s applicability in training models to effectively distinguish between disinformation and trustworthy content in multilingual settings.

3. WWW

   A Cross-Domain Study of the Use of Persuasion Techniques in Online Disinformation

   João A Leite, Olesya Razuvayevskaya, Carolina Scarton, and 1 more author

   Dec 2024

   PDF

   
4. PLOS ONE

   Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification

   Olesya Razuvayevskaya, Ben Wu, João A. Leite, and 5 more authors

   PLOS ONE, May 2024

   Abs DOI PDF

   

   Adapters and Low-Rank Adaptation (LoRA) are parameter-efficient fine-tuning techniques designed to make the training of language models more efficient. Previous results demonstrated that these methods can even improve performance on some classification tasks. This paper complements existing research by investigating how these techniques influence classification performance and computation costs compared to full fine-tuning. We focus specifically on multilingual text classification tasks (genre, framing, and persuasion techniques detection; with different input lengths, number of predicted classes and classification difficulty), some of which have limited training data. In addition, we conduct in-depth analyses of their efficacy across different training scenarios (training on the original multilingual data; on the translations into English; and on a subset of English-only data) and different languages. Our findings provide valuable insights into the applicability of parameter-efficient fine-tuning techniques, particularly for multilabel classification and non-parallel multilingual tasks which are aimed at analysing input texts of varying length.

5. arXiv

   A Survey on Automatic Credibility Assessment of Textual Credibility Signals in the Era of Large Language Models

   Ivan Srba, Olesya Razuvayevskaya, João A Leite, and 8 more authors

   Oct 2024

   

2023

1. ACL-SemEval

   SheffieldVeraAI at SemEval-2023 Task 3: Mono and Multilingual Approaches for News Genre, Topic and Persuasion Technique Classification

   Ben Wu, Olesya Razuvayevskaya, Freddy Heppell, and 4 more authors

   In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), Jul 2023

   Abs DOI PDF

   

   This paper describes our approach for SemEval- 2023 Task 3: Detecting the category, the fram- ing, and the persuasion techniques in online news in a multilingual setup. For Subtask 1 (News Genre), we propose an ensemble of fully trained and adapter mBERT models which was ranked joint-first for German, and had the high- est mean rank of multi-language teams. For Subtask 2 (Framing), we achieved first place in 3 languages, and the best average rank across all the languages, by using two separate ensem- bles: a monolingual RoBERTa-MUPPETLARGE and an ensemble of XLM-RoBERTaLARGE with adapters and task adaptive pretraining. For Sub- task 3 (Persuasion Techniques), we trained a monolingual RoBERTa-Base model for English and a multilingual mBERT model for the re- maining languages, which achieved top 10 for all languages, including 2nd for English. For each subtask, we compared monolingual and multilingual approaches, and considered class imbalance techniques.

2. RANLP

   Noisy Self-Training with Data Augmentations for Offensive and Hate Speech Detection Tasks

   João Leite, Carolina Scarton, and Diego Silva

   In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, Sep 2023

   Abs PDF

   

   Online social media is rife with offensive and hateful comments, prompting the need for their automatic detection given the sheer amount of posts created every second. Creating high-quality human-labelled datasets for this task is difficult and costly, especially because non-offensive posts are significantly more frequent than offensive ones. However, unlabelled data is abundant, easier, and cheaper to obtain. In this scenario, self-training methods, using weakly-labelled examples to increase the amount of training data, can be employed. Recent “noisy” self-training approaches incorporate data augmentation techniques to ensure prediction consistency and increase robustness against noisy data and adversarial attacks. In this paper, we experiment with default and noisy self-training using three different textual data augmentation techniques across five different pre-trained BERT architectures varying in size. We evaluate our experiments on two offensive/hate-speech datasets and demonstrate that (i) self-training consistently improves performance regardless of model size, resulting in up to +1.5% F1-macro on both datasets, and (ii) noisy self-training with textual data augmentations, despite being successfully applied in similar settings, decreases performance on offensive and hate-speech domains when compared to the default method, even with state-of-the-art augmentations such as backtranslation.

2020

1. AACL

   Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysis

   João A. Leite, Diego Silva, Kalina Bontcheva, and 1 more author

   In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, Dec 2020

   Abs HTML PDF

   

   Hate speech and toxic comments are a common concern of social media platform users. Although these comments are, fortunately, the minority in these platforms, they are still capable of causing harm. Therefore, identifying these comments is an important task for studying and preventing the proliferation of toxicity in social media. Previous work in automatically detecting toxic comments focus mainly in English, with very few work in languages like Brazilian Portuguese. In this paper, we propose a new large-scale dataset for Brazilian Portuguese with tweets annotated as either toxic or non-toxic or in different types of toxicity. We present our dataset collection and annotation process, where we aimed to select candidates covering multiple demographic groups. State-of-the-art BERT models were able to achieve 76% macro-F1 score using monolingual data in the binary case. We also show that large-scale monolingual data is still needed to create more accurate models, despite recent advances in multilingual approaches. An error analysis and experiments with multi-label classification show the difficulty of classifying certain types of toxic comments that appear less frequently in our data and highlights the need to develop models that are aware of different categories of toxicity.