Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network
Author(s): Ziqi Zhang, David Robinson, Jonathan Tepper
Full text: submitted version
Abstract: In recent years, the increasing propagation of hate speech on social media and the urgent need for effective counter-measures have drawn significant investment from governments, companies, and empirical research. Despite a large number of emerging scientific studies to address the problem, a major limitation of existing work is the lack of comparative evaluations, which makes it difficult to assess the contribution of individual works. This paper introduces a new method based on a deep neural network combining convolutional and gated recurrent networks. We conduct an extensive evaluation of the method against several baselines and state of the art on the largest collection of publicly available Twitter datasets to date, and show that compared to previously reported results on these datasets, our proposed method is able to capture both word sequence and order information in short texts, and it sets new benchmark by outperforming on 6 out of 7 datasets by between 1 and 13 percents in F1. We also extend the existing dataset collection on this task by creating a new dataset covering different topics.
Keywords: hate speech; cyber hate; twitter; social media; neural networks; deep learning; cnn; gru
Review 1 (by Michael Granitzer)
(OVERALL EVALUATION (*RESOURCES AND IN-USE TRACKS ONLY*, RESEARCH REVIEWERS PLEASE ONLY PUT "N/A")) Summary: ----------- The study investigates forms of appearance of hate speech on social media and reveals in a comparative evaluation across several datasets and modeling techniques the challenges for data-driven hate speech detection on Twitter. The authors cast the problem as a supervised text classification problem. They identify two categories of commonly employed methods: (1) classic methods which rely on manual feature engineering and (2) deep learning based methods which learn to extract features of higher-order correlations. They contribute to the latter category by proposing a neural network architecture composed of a convolutional network (CNN) to extract features and a gated recurrent network (GRU) to account for the word order in a sentence. They compare the classification accuracy of their CNN+GRU architecture against a support vector machine and evaluate the impact induced from an enhanced feature set, the addition of the GRU network and minor modifications in the CNN+GRU architecture. The authors additionally created an new hate speech dataset and made it publicly available. An analysis of classification errors shared by SVM and CNN+GRU methods concludes the study. Review: ----------- The paper is well-written, well-organized and thoroughly embedded in related work via a comprehensive synthesis of previous papers about hate speech detection. The paper claims 3 contributions: The CNN+GRU architecture, the creation of a new hate speech dataset and the comparative evaluation. I consider the creation of an annotated dataset and the comparative evaluation as the strongest points in this work. The comparisons to state-of-the-art methods on 7 datasets summarize what has been achieved and which methods currently work best for hate speech detection. These are, apparently, CNNs+X. The 'X' is in my opinion some problem specific constraint guiding feature extraction. Here 'X' corresponds to dropout, global max-pooling, elastic net regularization and the correlation between consecutive CNN features induced from the capacity of the GRU layer. According to Table 2, the effect of modeling the sequential dependency of CNN features along the sentence (CNN+GRU) is negligible. Since the datasets are very small and the experiments are only executed once, I try to be conservative with an interpretation of a change in F1 from 0.91 to 0.92. Nonetheless, what matters most is some form of model regularization: Dropout, global max pooling, elastic net regularization. This is confirmed with the results. The error analysis in the final section is by far the most interesting part of the work. It tells us that the presence of abstract concepts such as 'sexism', 'racism' or 'hate' is very difficult to detect if solely based on some textual expression. Apart from the class imbalance problem, the authors identify non-distinctive features, subtle metaphors, questioning or negation and stereotypical views as major obstacles in the automatic detection of hate speech. In my opinion these findings motivate the need to go beyond pure text classification and think about possibilities to model and integrate knowledge about social groups and mutual communication modes. Recommendation: ------------------ Even though this paper does not match perfectly the topics of a Semantic Web conference I would recommend to accept it. The creation of a new dataset is of value for any scientific community, the comparative evaluation provides a cornerstone for further progression in the small domain of hate speech detection and the error analysis motivates the integration of external knowledge bases. Minor: - In 3.2: "... into a a real vector domain." -> "..into a real.." - "Each of the 25 dimensions can be considered an 'extracted feature'" -> Isn't it more each of the 100 dimensions (the feature maps)? - Table 2: what is 'best' features?; "SVM, Davidson " should be swapped -> "Davidson , SVM"
Review 2 (by Stefano Faralli)
(RELEVANCE TO ESWC) The topic addressed in this work is only partially of interest to this conference. (NOVELTY OF THE PROPOSED SOLUTION) I consider that the novel approach and the novel dataset represent a good contribution to the field. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) there are no flaws on the presented solution. (EVALUATION OF THE STATE-OF-THE-ART) To the best of my knowledge the related work is inclusive of recent publications in the field (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) As already remarked I believe the authors conducted a proper experimental evaluation of the proposed approach (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The proposed methodology and the experimental setup is well described to the best of my knowledge there should not be problems about the replication of the experiments (OVERALL SCORE) Aa already remarked the presented work is a step forward in Hate speech detection providing a comprehensive evaluation of the proposed approach against a varaity of sota systems. In this work a novel dataset is also released and evaluated.
Review 3 (by Vito Bellini)
(RELEVANCE TO ESWC) The paper is suitable for the Research track. (NOVELTY OF THE PROPOSED SOLUTION) Proposed solution is quite novel and uses deep learning techniques which are widely adopted nowadays. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) Proposed approach is complete and well described. (EVALUATION OF THE STATE-OF-THE-ART) State of the art treated properly, both classic methods and novel ones such as deep learning are treated. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) Authors show the effectiveness of their proposed approach. (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Experiments are reproducible. They use publicly datasets and they provide a new publicly dataset they made. (OVERALL SCORE) Strong points: 1. Novel approach that uses deep learning techniques and it combines CNN and RNN to detect hate speech on twitter. 2. Publicly datasets that make the experiments reproducible. 3. They fully treated the state of the art, comparing methods from very different approaches like SVM and deep learning ones. Weak points: 1. Authors should describe better why they took the highest value after the pooling layer for each timestep. I think they do this because 100 is the length sentences and they want to reconstruct a sentence made of latent features of that length, so they pick up the next latent feature with highest probability at each timestep.
Review 4 (by anonymous reviewer)
(RELEVANCE TO ESWC) The paper is making a good contribution to the automatic detection of hate speech (NOVELTY OF THE PROPOSED SOLUTION) This paper proposes a novel technique to classify hate speech in tweets by combining convolutional and gated recurrent neural networks. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The paper the balance the existing datasets in terms the amount of true positives to true negatives as well as how balanced is the newly proposed dataset after merging the 3 batches into a single dataset. (EVALUATION OF THE STATE-OF-THE-ART) The literature review on the topic is concise. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) Overall the presentation of this paper is good in terms of the overall organization, language and writing style. (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Since using existing datasets and baselines, reproducibility is ensured (OVERALL SCORE) This paper proposes a novel technique to classify hate speech in tweets by combining convolutional and gated recurrent neural networks. The paper also extends currently available datasets in terms of quantity and subject coverage. The authors compared their proposed approach against several baselines. The literature review on the topic is concise. As for the approach, it might be a good idea to examine the effect of replacing the word vector representation in the word embedding layer with another model for example GloVe trained on Twitter. As for the datasets, it is good to show how balanced are the existing datasets in terms the amount of true positives to true negatives as well as how balanced is the newly proposed dataset after merging the 3 batches into a single dataset. Regarding the experiments, the authors compared their system against two baselines in addition to the reported results of the state of the art systems on the same datasets. It might be effective to show the precision and recall as well in order to show the effect of the false positives and false negatives (adding two more columns to table 2 will not affect the paper space that much). Overall the presentation of this paper is good in terms of the overall organization, language and writing style.
Review 5 (by Michael Granitzer)
(RELEVANCE TO ESWC) See overall evaluation (NOVELTY OF THE PROPOSED SOLUTION) See overall evaluation (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) See overall evaluation (EVALUATION OF THE STATE-OF-THE-ART) See overall evaluation (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) See overall evaluation (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) See overall evaluation (OVERALL SCORE) See overall evaluation
Metareview by Stefan Dietze
This paper presents an approach towards detecting hate speech on Twitter using deep neural networks. Results indicate superior performance compared to a number of baselines. Reviewers agree that this is a sound and well-presented submission which contributes to the state-of-the-art in the field. Whereas the topic is of relevance to ESWC in general, we would still like to encourage the authors to deepen the discussion of their contribution to the semantic web community in the final camera-ready version of their submission.