Paper 65 (Research track)

Ontology-Driven Sentiment Analysis of Product and Service Aspects

Author(s): Kim Schouten, Flavius Frasincar

Full text: submitted version

camera ready version

Decision: accept

Abstract: With so much opinionated, but unstructured, data available on the Web, sentiment analysis has become popular with both companies and researchers. Aspect-based sentiment analysis goes one step further by relating the expressed sentiment in a text to the topic, or aspect, the sentiment is expressed on. This enables a detailed analysis of the sentiment expressed in, for example, reviews of products or services. In this paper we propose a knowledge-driven approach to aspect sentiment analysis that complements traditional machine learning methods. By utilizing common domain knowledge, as encoded in an ontology, we improve the sentiment analysis of a given aspect. The domain knowledge is used to determine which words are expressing sentiment on the given aspect as well as to disambiguate sentiment carrying words or phrases. The proposed method has a highly competitive performance of over 80% accuracy on both SemEval-2015 and SemEval-2016 data, significantly outperforming the considered baselines.

Keywords: aspect-based sentiment analysis; product and service reviews; domain ontology; SVM

 

Review 1 (by anonymous reviewer)

 

(RELEVANCE TO ESWC) The paper integrates a semantic strategy.
This makes it sound for ESWC.
(NOVELTY OF THE PROPOSED SOLUTION) The novelty of the proposed approach is limited but interesting with respect to the state of the art.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The presented approach is correct and complete with respect to the problem they want to address.
(EVALUATION OF THE STATE-OF-THE-ART) The related work should be slightly improved as mentioned in the general review.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) Discussion is proper.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Results are reproducible.
(OVERALL SCORE) The paper discusses an approach mixing semantic and statistical strategies for improving the effectiveness of aspect-based sentiment analysis systems.
The idea behind this paper is timely since the way reviews are written by users needs the capability of extracting from text the set of pairs <aspect, opinion-words> in order to analyze them separately.
Actually, I don't have any major concern about this paper expect for some minor stuff.
1) Ontology analysis.
In order to better provide a final judgement about the contribution of this paper, it would be helpful to give access to the ontology in order to perform a more detailed analysis.
From the paper the ontology seems quite simple, but even this is not a problem for the Reviewer, it would be interesting to analyze it in order to provide (eventually) some feedbacks that might be of help for improving it.
2) Related work.
The related work shoud be better linked with aspect-based contribution proposed with, for example, the recent editions of the workshop on sentiment analysis run at ESWC in the last years.
A brief recap should be provided.
3) Evaluation.
By considering the particular structure of the SemEval datasets, it would be interesting to have an error analysis of the results.
I know that the space is quite limited, but a graph with a brief explanation of the main insights would be helpful for the reader.
---------------------
I thank the authors for their effort in preparing the rebuttal.
After reading their reply, I confirm the score given earlier.

 

Review 2 (by anonymous reviewer)

 

(RELEVANCE TO ESWC) Ontology-driven hence appropriate for ESWC.
(NOVELTY OF THE PROPOSED SOLUTION) The novelty of the solution is fair compared to authors previous work. The ontology have been improved with more specific knowledge and a two stage approach is introduced instead of a combination of the knowledge and machine learning based approaches.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The authors proved that the proposed solution achieves a good performance in the SemEval evaluation.
(EVALUATION OF THE STATE-OF-THE-ART) The related work section is quite comprehensive and the authors compare previous approaches with their proposed solution.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The different aspects are presented in a comprehensive manner, the algorithm is well explained.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The experimental setup being an open evaluation, the experimental study is reproducible.
(OVERALL SCORE) Summary of the Paper
The paper describes an ontology-based sentiment analysis method. It is based on domain knowledge described in an ontology and when not applicable, it relies on a bag of words model. The method is proven to have a high performance on the 2015 and 2016 editions of the SemEval Aspect-Based Sentiment Analysis task. 
Strong Points (SPs)
The paper is easy to read and well written and structured. 
The algorithm is clearly explained with simple and useful examples.
The evaluation section is exhaustive well commented with clear conclusions.  
Weak Points (WPs)
The usefulness of two stage method (which is one of the main contribution of the paper) compared to the previously proposed hybrid solution is not clear. As mentioned by the authors, the performance does not depend on the amount of available training data but it is still outperformed by hybrid approaches even with small sets of training data.
The main contribution is the new classification of the sentiment words which even if it is shown to be effective stays in my opinion relatively low in terms of novelty of the approach.

 

Review 3 (by Víctor Rodríguez Doncel)

 

(RELEVANCE TO ESWC) I have carefully gone through the bullet points at:
Natural Language Processing
and I have found no clear match to any of the points (at least in a strict sense). The role of the Ontology in the proposed method is marginal, namely, other data structures might have been used with equal sufficiency. The reasoning task only deals with subsumption and the class hierarchy is very small. Of course, it is of interest for the ESWC in a broad sense, but I believe there are other more appropriate fora for the discussion of works like this. (NOVELTY OF THE PROPOSED SOLUTION) I do not see a strong novelty, specially with respect to their previous work [14]. The contribution is best highlighted in page 3 (second half of the page), and can be summarized in (a) the classification of sentiment words in three classes (b) using ontology-based + machine learning in cascade, and not being the former only a source of features for the latter. The first claimed novelty is small, the second has been proposed more times in the literature. Figure 1 is 100% as in [14] Figure 2 is 75% the same as in [14]. A simple reference would have sufficed. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The solution is correct and complete. Something that could be improved: how the ontology lexicalization has been made, with a few examples. This point is central to the paper and it is only superficially mentioned. (EVALUATION OF THE STATE-OF-THE-ART) Only a few works are considered in Section 2, but the literature is so extense and the paper length in this conference so small that I consider that the space in the SOTA section is very well used. A mention to other hybrid techniques would not have harmed, though (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) There is no reason to believe that the comparison made with other approaches in the SemEval is fair: whereas others presented their algorithms blindly (not knowing the test entries), the authors have enjoyed time enough as to craft an ad-hoc dictionary to grant that enough matches exist. The ontology is not disclosed either, so this can be even less judged. Thus, I can only take as valid the comparison presented in Table 1 and 2 (but not in Table 3) (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The ontology has not been disclosed. The implementation of the algorithm has not been disclosed. The algorithm is not online for testing. Not much to reproduce, then. NEW REVIEW: ONTOLOGY AND SOURCE CODE IS ONLINE. GREAT! (OVERALL SCORE) This paper describes a method for aspect based sentiment anlysis of English texts. The method is a cascade of a dictionary matching and a machine learning method. The entries for the dictionary matching are organised as lexicalizations of an ontology. The paper is well presented: very readable English and very good structure. Actually, the beauty and elegance of Figure 3 has surprised me. This paper is a logical research stage in the career of the first of the authors, whose work is well presented in http://www.kimschouten.com/publications/. Kim has published a very good survey on the matter, and he must be very well knowledged on the topic. However, I see at least three strong reasons to reject this paper: 1) The results presented in Table 3 can be disregarded, as they are not fair: word matches with hand-made lexicalizations of a hand-made ontology may have been specifically crafted to get good results. 2) The relation to ESWC is not so direct, as the role of the ontology is minor and many other conferences exist that fit better. 3) The novelty and contribution of this paper are not enough for a good conference as ESWC (see comments above) As other minor comments: - I would have appreciated a more straightforward indication of which task within SemEval (e.g. SemEval ABSA, Subtask 1, Slot 3) I have only found one typo: Section 4.2, paragraph 2, line 3: change than->then I have found difficulties with understanding the English in this Section. The rest of the paper was perfect. The abstract can be improved: the first sentences can be dropped/shortened. I encourage authors to present this work in a more reproducible fashion. Only one tiny question: is the method in Section 4.4 the same as in [14]? UPDATED REVIEW: - I APPRECIATE YOUR EFFORTS OF PUBLISHING THE ONTOLOGY AND SOURCE CODE (ALTHOUGH IT DOES NOT COMPILE!) - I APPRECIATE YOU UNDERSTAND MY CONCERN ABOUT THE FAIRNESS OF THE EVALUATION AGAINST PAST SEMEVAL EDITIONS. - THANKS FOR ANSWERING MY QUESTION I HAVE IMPROVED MY SCORE ACCORDINGLY.

 

Metareview by Valentina Presutti

 

The paper proposes an ontology-driven approach to aspect based sentiment analysis. It is well written and introduce novelty with respect to previous work and related literature. It is appreciated that the ontology and the source code are available, helping to support reproducibility. The authors are recommended to follow reviewers' suggestions to bring the paper at its best form and in particular to include a fair account on the comparison with SemEval.

 

Share on

Leave a Reply

Your email address will not be published. Required fields are marked *