Paper 31 (Research track)

Multiple Models for Recommending Temporal Aspects of Entities

Author(s): Tu Nguyen, Wolfgang Nejdl

Full text: submitted version

camera ready version

Decision: accept

Abstract: Entity aspect recommendation is an emerging task in semantic search
that help users discover serendipitous and prominent information with respect
to an entity, of which salience (e.g., popularity) is the only important factor in
previous work. However, entity aspects are temporally dynamic and often driven
by happening events. For such cases, aspect suggestion based solely on salience
features can give unsatisfactory results, for two reasons. First, salience is often
accumulated over a long time period and does not account for recency. Second,
an aspect that is related to an event entity is often strongly time-dependent. In this paper, we study the task of temporal aspect recommendation for a given entity,
which aims at recommending the most relevant aspects and takes into account
aforementioned challenges in order to improve search experience. We propose
a novel event-centric ensemble ranking method that learns from multiple time
and type-dependent models and dynamically trades-off between the salience and
recency characteristics of entity aspects. Through extensive experiments on real-world query logs, we demonstrate that our method is robust and achieves better
effectiveness than competitive baselines.

Keywords: Semantic search; Entity Aspects; Temporal Dynamics

 

Review 1 (by Christian Dirschl)

 

(RELEVANCE TO ESWC) The Topic covered fits very well into the scope of ESWC. The NLP track is explicitly asking for entity extraction submissions. Also Semantic Web Technologies and datasets are in the core of the contribution.
(NOVELTY OF THE PROPOSED SOLUTION) According to the related work section, there is no work yet on Event time and type modelling for entity extraction. So this is new and indeed relevant and novel.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The Approach taken seems to be valid and the extensive Evaluation section also supports that. Since this is at least in parts a novel Approach, the completeness is not always given, but this is acceptable.
(EVALUATION OF THE STATE-OF-THE-ART) I am working in industry, so I do not have a detailed overview on the state-of-the-art in this area. But according to work in the paper itself, it seems to be solid in this area.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) This is the only part where I am not completely satisfied with what has been presented. I would have liked to see more examples on what a temporal Event type actually could be. There are a few examples in the Evaluation section, but this is not very illustrative and does not give a complete Picture. So in my Point of view, more examples could have helped to better understand the Approach taken.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) This is fine. Open datasets, clear definitions and a proper Explanation of the maths behind the approaches taken should ensure reproducibility.
(OVERALL SCORE) I would like to hear more on that at the conference. The Topic is relevant and has for sure a Business Impact potential.
Strong Points are:
- Structure: I have seen all relevant parts for such a paper in the necessary depth and also the writing itself is very clear and can easily be consumed
- Relevance: In case the time aspect of Event entities can be better covered, I see a clear benefit to the Quality of semantic search results. So the work here is relevant
- Evaluation: The Evaluation section is extensive and still focused. So future work will benefit from that
Weak Points are:
- As I said, more examples would help the Reader to better understand the potential and Impact of the work presented
- Since there is no real previous work on this specific Topic, a result which Shows that it is superior to the baseline is not really a strong Argument. So in future work, strong Focus should be on additional KPIs in order to make sure that things are moving in the right direction

 

Review 2 (by Vito Bellini)

 

(RELEVANCE TO ESWC) This paper is relevant to the Research Track.
(NOVELTY OF THE PROPOSED SOLUTION) The proposed approach is well-structured and novel.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The proposed solution is correct and well described. Mathematical formalism are adequate.
(EVALUATION OF THE STATE-OF-THE-ART) State-of-the-Art is fully evaluated.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) Proposed approach results to be effective in the task the authors are pursuing, outperforming baselines and bringing advancements in the field using multiple time and type dependent models to provide temporal aspect recommendation for entities.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Experiments are reproducibles. All the details have been explained.
(OVERALL SCORE) This paper is well written and it fully explains how the experiments are conducted.
Proposed solution is novel and it consider temporal aspects to improve entities recommendation.
Used datasets are publicly available therefore experiments are reproducible.
I encourage authors to continue their experiments trying to exploits even the search context.

 

Review 3 (by Maria Koutraki)

 

(RELEVANCE TO ESWC) The paper is relevant to the topics of the conference.
(NOVELTY OF THE PROPOSED SOLUTION) The paper proposes a novel solution to the problem of recommending temporal entity aspects.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The problem tackled in this paper is well defined, and the proposed solution is justified and supported by their experimental evaluation and the prior state of the art.
(EVALUATION OF THE STATE-OF-THE-ART) The authors could elaborate more on the related work section and draw an actual comparison to the related work that it is mentioned. Instead of just listing them.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The paper proposes a reasonable and well explained algorithm  to solve the problem of recommending temporal entity aspects. This is followed by a solid experimental evaluation of the different components of the algorithm.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The proposed approach is reproducible since all the steps are well explained as well as the tuning of the proposed models is mentioned.
(OVERALL SCORE) ** Short description **
The paper presents an approach for ranking temporal entity aspects with the aid of query logs. It ranks the temporal aspects of each entity using a novel time and type-specific ranking method that learns multiple ranking models. 
** Strong Points **
1) The paper is nicely written and well structured.
2) Detailed analysis of the proposed features in sections 4.2 and 4.4 as well as the ranking models in 4.3.
3) The paper has a solid experimental evaluation of all the components proposed in it.
** Weak Points and Questions to the Authors **
1) The authors could elaborate more on the related work section and draw an actual comparison to the related work that it is mentioned. Instead of just listing them.
2) Why didn't you provide a comparison to paper [22] (as provided by your Reference list) ? 
3) What is the complexity of the algorithm in section 4.1?
=======
I would like to thank the authors for their rebuttal. They might consider to integrate the answers to their final version of the paper. My score remains "accept".

 

Review 4 (by Giuseppe Rizzo)

 

(RELEVANCE TO ESWC) The paper well answers the call for papers (see below)
(NOVELTY OF THE PROPOSED SOLUTION) The approach aims to address an critical pain in NLP with a novel idea, however the approach is rooted on traditional AI techniques
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) (see below)
(EVALUATION OF THE STATE-OF-THE-ART) Rigorous analysis.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) Demonstration is clear and the discussions are relevant.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Authors left enough traces to reproduce the results. Dataset is available.
(OVERALL SCORE) This paper addresses the task of filtering temporal aspects of entities that are relevant to a certain time window by first proposing a study on recommending temporal entity aspects, then a learning method to recognize entity type and time, and an ensemble ranking method for determining the relevancy of entity aspects. 
The approach presented in this paper leverages the information available on the Web for both model creations and determining the necessary signals to segment the search space according to time and thus for ranking the aspects. The learning models implement traditional learning models based on SVM. An in-depth experimentation has been proposed, which highlights the large margin of gain achieved wrt baselines. The acceptance of this paper is advisable because of the in-depth study and the achieved performance of the approach presented in this paper. 
Strenghts: 
- rigorous problem statement and approach formulation
- rigourous description of the experimental setup and datasets (which can be downloaded), definition of the configuration parameters, and tools used to run the validation
- outstanding figures achieved in the conducted experiment certify that this approach can be considered as a future baseline for further analyses and studies for filtering temporal aspects
Weaknesses:
- authors propose a set of features meant to instruct supervised models for the recognition of entity time, entity type and aspect ranking. Despite the enormous effort in engineering these features, there is little explaination about why those and their impact to the models
- use of TagMe for annotating search terms and no further study has been performed to understand the impact of it in the overall results 
- unclear if the lenght of the time series has an impact to the recognition of entity type and time?
- unclear the application of the t-test indicated in Table 4 and how the test has been performed 
- there is a subtle ambiguity in the paper: entity vs event entity. Even though the former is conceptually linked with the latter from which more signals can be derived in order to collect & filter temporal aspects, it's unclear if the approach focuses only on event entity. Please clarify this in the camera ready paper  
- such an approach has as tight dependency from data & labeled data used to instruct the learning models and to accumulate data for a long period. This data is supposed to be archived or accessible somewhere as in the datasets used for experimentation. However, there isn't any mention that past (very far) search logs are scarce to be found since they constitute a very valuable asset for the owner. Thus, I'd have appreciated a mention on the feasibility of this approach and its application in other real scenarios than of popular search engines
- It is advisable to lower the concept of recommendation (which is meant to describe the interaction of user/items or user group/items) and talk about filtering since the overall paper targets a global (of all) optimization based on public attention (of all)
A Few typos spotted: 
- an distinct -> a distinct
- We description of -> The description of
- Identfying event entities -> Identifying event entities
- Maximum Likelihood Estimatior -> Maximum Likelihood Estimator
- in the the range -> in the range
- 1740 -> 1,740 (this applies to all figures: 3050, ...)
- with all features combined, gives -> ... combined gives

 

Metareview by Valentina Presutti

 

The paper proposes an approach for recommending temporal aspects of entities. The paper is well written, the approach novel and the evaluation is sound and looks reproducible. It is appreciated its potential impact also in business settings. The authors are required to address all comments and suggestions raised by the reviewers for supporting a better understanding of the proposed method and the reproducibility of the evaluation experiments. Adding examples, better explaining "why" and "how" of the impact of the chosen features, discussing existing relevant studies with a comparing perspective, are all recommended actions.

 

Share on

Leave a Reply

Your email address will not be published. Required fields are marked *