Investigating Semantics-driven Query Rewriting Using Plausible Patterns for Medical Knowledge Discovery and Decision Support
Author(s): Hossein Mohammadhassanzadeh, Samina Abidi, William Van Woensel, Syed Sibte Raza Abidi
Full text: submitted version
Abstract: Plausible reasoning is the manifestation of the “plasticity” element of human reasoning, which allows dealing with incomplete data and still discover unknown associations by leveraging semantics of concepts. We proposed SEmantics-based Data ANalytics framework (SeDan) that integrates plausible reasoning with fine-grained biomedical ontologies. Using this framework, an initial query with no answer can be transformed to an expanded version that effectively infers new knowledge. In this paper, we investigate the efficiency of SeDan in a real-world medical setting by using the framework to pose intelligent medical queries form BioASQ challenges over the Semantic MEDLINE database. We have developed a Semantic Web-based framework that stores data from databases into an RDF storage and the semantics from two biomedical OWL ontologies conduct the query rewriting. Experimental results show SeDan can expand the query answering coverage of SemMedDB by resolving up to 45% (depending on the type of the questions) of the initially unanswered questions. The correctness of the plausibly inferred answers was verified by a domain expert.
Keywords: Plausible Reasoning; Query Expansion; Semantic Web Reasoning; Semantic Analytics
Review 1 (by anonymous reviewer)
(RELEVANCE TO ESWC) The presentation is good. (NOVELTY OF THE PROPOSED SOLUTION) What is the difference to GCLRR? What is novel wrt to the previous papers of the authors? (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) ok. (EVALUATION OF THE STATE-OF-THE-ART) I don't know what state-of-the-art here is, but they did not compare to GCLRR or other approaches. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) Very good examples. (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) ok (OVERALL SCORE) Mohammadhassanzadeh et al. present a method called SeDan for query adaptation with application to a medical knowledge base. The paper is well written. I think this is an interesting manuscript, which profits from an intuitive presentation of examples. However, I have some major concerns regarding the study. Major points: I wonder what exactly the novelty of the algorithm is. The authors write that their algorithm is inspired by GCLRR, so a natural question is, what is the difference in the presented study? If there is no difference, then what is the novelty of this paper? If there is a difference, wouldn't it be obvious to have a comparison to GCLRR in the results section? Another major concern is based on Table 4. Here the authors present some examples for "Semantics conducting the Query Expansion". I wonder if the authors have really looked into their data, because it is easy to see that there are serious errors and reversed relationships in their knowledge base. The authors write that "psychiatric problem causes psychophysiological insomnia". It is obvious that this is a wrong statement. Same for "carcinoma precedes malignanant neoplasms". Also, they write that "malignant neoplasm IS_A prostate cancer". The authors should re-analyse their data, because if these axioms are contained in their knowledge base, then inferences based upon this, are hard to trust. In the results I wonder if the number of tests can be increased, as I find 44 to be very few. This problem becomes especially apparent in Table 3, where one can see that only 7 queries have actually been rewritten and answered. This is a really low number and I am not sure that one can use it to reliably assess the quality of the algorithm. Is there any chance to increase this number? The used patterns are never described, i.e. the authors do not define the abbreviations GEN, SPEC, SIM, DIS, FORT, INTP. However, these are central elements of the paper and must be defined properly. Minor points: Section 2: First sentence is directly taken from abstract. This should be revised. Introduction: This is not a sentence: "Given that Semantic Web reasoning..." Please re-read and revise. Page 6: You are using "(step 6)" before the reader has any idea what this might refer to. FORT or AFORT? What is correct? Both is used.
Review 2 (by Ioanna Lytra)
(RELEVANCE TO ESWC) The article entitled "Investigating Semantics-driven Query Rewriting Using Plausible Patterns for Medical Knowledge Discovery and Decision Support" proposes the use of plausible patterns for inferencing knowledge from Knowledge Graphs, in order to answer queries in the medical domain. It is very relevant and interesting to the ESWC community. (NOVELTY OF THE PROPOSED SOLUTION) One of the authors' previous works, also cited in this paper, entitled "SeDAn: a Plausible Reasoning Approach for Semantics-based Data Analytics in Healthcare" and published in 2017 at [email protected]*IA has several overlappings with the current work. By comparing the two works it is difficult to see the contributions of the ESWC paper: in fact, apart from the texts, even the tables, the figures and the algorithm are repeated and for the evaluation exactly the same dataset is used. The only difference I see is in the presentation of the experimental results. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) - The authors have not formalized the rules of Table 1. - There is no proof under which conditions the plausible patterns are able to support the retrieval of correct answers. - It is not clear if the exensions to OWL of Table 2 are generalizable. (EVALUATION OF THE STATE-OF-THE-ART) The publications co-authored by the authors of this work (, ) have several overlappings with this paper. Apart from this, I am missing a Related Work section comparing this work to existing approaches. Although, there are some references in the Introduction to DL-Lite and to inference techniques with incomplete data, there is no discussion about works on Knowledge Discovery (e.g. for big biomedical data), which is a very related topic. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) As indicated in the previous subsection there is no discussion of the properties of the proposed approach. The plausible patterns introduced are not well justified, their properties are not demonstrated and the conditions under which the framework can rewrite SPARQL queries are not clear. What are the assumptions made by the authors? (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The experimental study is a bit limited in my point of view and does not provide enough evidence for the effectiveness of the SeDAn framework. I would suggest to the authors to use the QALD questions in the biomedical domain as well the training set of BioASQ which is pretty big (>1500 for BioASQ 2017) in order to increase the number of questions (SPARQL queries) under study. Which version of BioASQ has been used? Why only a small portion of the training and test sets (44) are relevant to this context? The experimental study needs definitely to be extended. It would be interesting to see if the proposed approach has applicability to other domains other than the biomedical domain as well. (OVERALL SCORE) Summary of the Paper The article entitled "Investigating Semantics-driven Query Rewriting Using Plausible Patterns for Medical Knowledge Discovery and Decision Support" proposes the use of plausible patterns for inferencing knowledge from Knowledge Graphs, in order to answer queries in the medical domain. The main contribution of this work is the consideration of plausible reasoning methods in order to rewrite SPARQL queries, so that more answers from the Knowledge Base can be retrieved. The evaluation is performed with queries from the BioASQ challenge - the proposed framework can answer more queries than the generated SPARQL queries, not all of them correctly. Strong Points (SPs) - Clear presentation of the motivation - good motivating example - Proved improvement of answering SPARQL queries from the BioASQ challenge - Clear presentation of the approach, using examples Weak Points (WPs) - Big overlappings with previous work of the authors - No related work - Contributions are not clear - Limited evaluation to a small number of queries Questions to the Authors (QAs) - How do you position this work with regard to  and ? - Why did you choose only a few questions from BioASQ? - Under which assumptions is SeDAn able to return correct and complete answers? - How do you compare your approach to other Knowledge Discovery techniques? After rebuttal -------------- I would like to thank the authors for their explanation. However, by comparing again the previous works with the current submission, I still think that there are many overlappings. Apart from this, I think that the authors need a more extensive evaluation.
Review 3 (by anonymous reviewer)
(RELEVANCE TO ESWC) The paper's novel contribution concerns the evaluation of a Semantic Web-based framework (SeDan), already presented in another work, that by storing data from databases into an RDF storage and adding the semantics from biomedical OWL ontologies, conducts a query rewriting in order to expand the query answering coverage. I think that this topic could bring good discussions to the conference. (NOVELTY OF THE PROPOSED SOLUTION) The paper consists of 14 pages and substantially three parts: a first general introduction that should also give an overview of the state of the art on the subject, a second that discusses the Sedan framework introducing the concept of plausible reasoning and the owl extension previously proposed by the authors and, subsequently, discusses the architecture of SeDan; the third part consists of experimental results, that is the system evaluation on medical queries from BioASQ challenges. However, this last part, which is the only original contribution of the paper, is reduced to about 28% of the article and some points are unclear. In both quantitative and qualitative terms, therefore, the paper does not produce a highly original contribution. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) Although there are some shortcomings on the background, on the state-of-the-art and the goal is not immediately clear to the reader, the paper is presented as self-contained, readable and well written. (EVALUATION OF THE STATE-OF-THE-ART) No state-of-the-art is presented, there are no references to the existence or non-existence of similar systems in the literature and, consequently, the innovations proposed by the SeDan framework with respect to existing systems are not possibly disclosed. References to the introductory part are lacking, mainly referring to previous authors' works. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The experimental results show an effective extension of the query answering coverage w.r.t. the initial answered queries, but: i) the query rewriting algorithm is actually used only on 30 queries, responding correctly to 7 queries, with a total efficiency of 23%; ii) the algorithm strongly depends "on the richness and correctness of the captured domain knowledge"; iii) The percentages related to the experimental results are scattered throughout the text, don't provide the reader a summary scheme that can summarize the results. In some points, the information is conceptually conflicting. First of all, the objective of the paper is not clearly explained both in the abstract and in the introduction. In fact, if in the introduction it is stated "In this paper, we present a plausible reasoning approach, implemented as a set of plausible patterns etc." it seems that the paper's new contribution is both to present and to evaluate the efficacy of SeDan framework, but then authors refer to the article in which SeDan framework has been actually presented. Another discordant point that, in my opinion, is the major weakness of the paper, is that what is shown in example 3.2, is not then reflected in section 5 (see the overall evaluation). (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The experimental study is quite general however, reproducibility is limited because there is no information on how to access the SeDan framework. (OVERALL SCORE) The vast amount of available healthcare data poses, among the challenges, not only the caption of medical knowledge but also the extraction of trends, relationships and patterns from data, potentially useful for clinical decision making. SeDan is a framework developed by the authors whose purpose is to integrate deductive and plausible reasoning and to exploit Semantic Web technology to solve complex clinical decision support queries. Plausible reasoning mechanisms include inductive reasoning, which generalizes the points in common between the data to induce new rules and analogical reasoning, which is guided by the similarities of the data to infer new facts. Aim of this paper is to demonstrate the functionality of SeDan by answering to real world medical queries from BioASQ challenges and, at the same time, to evaluate the efficiency of SeDan framework in this setting. As stated before the experimental results show an effective extension of the query answering coverage w.r.t. the initial answered queries. **Strong points (SPs)**: i) the paper is well written and the topic is very interesting; ii) the paper is presented as self-contained, providing all the fundamental required for understanding the evaluation experiment; iii) the SeDan architecture is well explained, the example help the reader understand the functionality of the framework and, in particular, the potential of using query rewriting as a technique to implement plausible patterns to solve failed queries. **Weak Points (WPs)** i)in my opinion, the paper completely lacks a state-of-the-art, useful for contextualizing the SeDan framework with respect to an analysis of the literature and to enhance, aventually, the original contribution given by the framework; ii) it is not well evidenced what the paper contribution is w.r.t. what has already been proposed by the authors in their previous work. Neither the abstract nor the introduction highlight the novel contribution from the paper. Why would someone who read your previous works read this too? Moreover, in my opinion, when evaluating the performance of a tool, it is important to give non-confusing information, trying to summarize the results in a well-readable way. For example, you can not expect the reader to do the calculations to figure out what 28% is and then what 45% is w.r.t. the same data, if you do not report everything in a table or list, it could cause confusion. iii) Finally, the goal of the Experiment study is not clear. In section 3.1 authors write "SeDan leverages query rewriting as a technique to implement plausible patterns and solve failed queries that were initially unresolvable using the available knowledge", then Example 3.2 (page 6) shows how the use of SeDan " the plausible reasoner leverages assertional knowledge and ontological constructs to transform a (deductively) failed query to a new plausibly-inferred query", leading the reader to think that, even if the conventional deductive reasoning gives a (wrong) answer, the new-plausibly-inferred queries can infer a plausible answer that might be more correct. In the experimental results Section this idea disappears again and only answered (and not of acceptable answers) or not answered queries are taken into account, and on the answered initial queries, no type of verification is carried out. Then, the real evaluation of the framework is carried out only on the 30 queries not initially answered and, in that case, only the 23% of the queries are resolved. In my opinion, this point needs to be clarified.
Metareview by Jeff Z. Pan
The paper proposes plausible reasoning methods in order to rewrite SPARQL queries, so that more answers from the target knowledge base can be retrieved. The main concern from the reviewers is the overlapping with existing publications from the authors. Also, they think more extensive evaluation is needed. We hope these comments are useful for future revisions of the paper, which should have a different structure, so as to better present the novelties suggested in the authors' response.