Task-Oriented Complex Ontology Alignment – Two Alignment Evaluation Sets
Author(s): Elodie Thieblin Thieblin, Ollivier Haemmerle, Nathalie Hernandez, Cassia Trojahn
Full text: submitted version
Abstract: Simple ontology alignments, largely studied, link one entity from a source ontology to one entity of a target ontology. One of the limitations of these alignments is, however, their lack of expressiveness which can be overcome by complex alignments.
Although different complex matching approaches have emerged in the literature, there is a lack of complex reference alignments on which these approaches can be systematically evaluated. This paper proposes two sets of complex alignments between 10 pairs of ontologies from the well-known OAEI conference simple alignment dataset. The proposed alignments involve equivalence correspondences linking single entities of a source ontology to constructions of entities of the target ontology. The methodology for creating the alignment sets is described and takes into account the use of the alignments for two tasks: ontology merging and query rewriting. The ontology merging alignment set contains 313 correspondences and the query rewriting one 431. We report an evaluation of state-of-the art complex matchers on these proposed alignment sets.
Keywords: ontology matching; complex alignment; evaluation; task-oriented ontology alignment
Review 1 (by Pavel Shvaiko)
(RELEVANCE TO ESWC) The submission provides two task oriented (ontology merging and query rewriting) complex alignment evaluation sets, accompanied by a methodology to construct them. The problem addressed is relevant and is worth further investigations. (NOVELTY OF THE PROPOSED SOLUTION) The related work and more generally the background are adequate. The exact technical contributions of this submission with respect to the past works by the authors, e.g., in [31,32] have to be detailed. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The submission provides a general methodology, followed by its instantiations for specific tasks: ontology merging and query rewriting. Step 2 (page 7) concerns the creation of the complex correspondences based on simple ones, such that the complex correspondences fit the purpose of the alignment. In sufficient technical details are provided on how this actually occurs, e.g., how the complex correspondences of Table 2 were constructed, e.g., manually, if so, who and how many were the creators and with what expertise? What where the difficulties and the easy parts of the exercise? (EVALUATION OF THE STATE-OF-THE-ART) The alignment sets constructed are available online. The evaluation was performed on three state of the art approaches. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) Details on different types of heterogeneities on page 12 appear to be excessive with respect to the observations drawn (Section 7). (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) A perspective discussion is provided. The exercise can be reproduced. (OVERALL SCORE) Based on the comments above and taking into account the rebuttal in my view this submission qualifies for weak accept.
Review 2 (by Armando Stellato)
(RELEVANCE TO ESWC) Not much to say: ontology alignment is one of the core topics of ESWC. There are detractors and understandable criticism (e.g. it has been squeezed beyond the limits of interest, it’s a lot of in-vitro experiments and not much real stuff etc..) yet it is in the focus of the conference. (NOVELTY OF THE PROPOSED SOLUTION) It is not the first, but one of not-too-many works on complex alignments, plus it brings an interesting resource (the complex alignments) on the plate (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) In general, a mildly rigorous methodology (see comments later in the review) and an alignment dataset produced without convergence/consensus from a group (>1 person) of annotators. Just one clarification needed: In the abstract a complex mapping is referred as a link from a single atomic entity of the source ontology to a complex entity of the target one, while in section 2.1 the complex entity can be any of eo1, eo2. Conceptually this is not a big issue, though since the alignment is defined as directional, it is better to clarify. Minor comment on the style in 2.1: used the expression “the example” , but no example has been introduced. The authors should introduce “an example” and its purpose, and then say “the example” when referring to it FootNote 2 could be more concise: “transformation functions can not be formalised into DL” (if they were, then it would be possible to state correspondences involving them as well, so no need to mention correspondences) (EVALUATION OF THE STATE-OF-THE-ART) State of the art is properly presented, though I would consider earlier works that by first introduced the need for complex alignments and solutions for it, not only in terms of automatism, but as of representation and management. E.g. Alexander Maedche, Boris Motik, Nuno Silva, and Raphael Volz MAFRA — A MApping FRAmework for Distributed Ontologies EKAW '02 Proceedings of the 13th International Conference on Knowledge Engineering and Knowledge Management. Ontologies and the Semantic Web, Pages 235-250, October 01 - 04, 2002 , Springer-Verlag London, UK, ISBN:3-540-44268-5 http://disi.unitn.it/~accord/RelatedWork/Matching/MAFRA02.pdf minor comment: in page 4, slightly better to put the  together with Alignment API, than at the end of the sentence referring to the possibility to transform alignments into OWL axioms, especially considering their very generic name, which is actually the name of the framework described in  (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The distinction between ontology and schema (section 3.1) is rather subjective. An ontology provides a schema as well. The authors could simply say that the paper is focused on ontology alignment and does not deal with matching of other kind of schemata. The methodology used for producing the mappings is rather poor in description, details and probably simple in the approach (though it is worth mentioning the difference expressed by the authors between the two presented tasks and how this affects the production of the alignments) (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The resource has been made available, the methodology for producing it has been explained, thus allowing reproduction of the resource (even though subject to the individual skills of the human producing the alignments, and not work has been done for evaluating convergence of different humans on the alignment task). The experiments report on the application of well-known complex alignment systems to the published alignments (OVERALL SCORE) The paper describes a methodology for building alignment datasets with complex correspondences. The methodology has been applied producing two sets of complex alignments between 10 pairs of ontologies from the OAEI benchmarks. Three known systems producing complex alignments have been then evaluated on the produced complex-alignment benchmarks The paper is a nice reading and is rich in references to recent literature. Having touched the aspects of alignments in my research in the past years, I felt I got a far better critical “refresh and update” on the topic from this work than from others I had the occasion to review or read recently. However, going to the core contribution of the work, I felt a general atmosphere of “work in progress” (see list of weak points ahead) that, if not getting me disappointed, made me reflect if the achieved level of maturity is enough for this conference. Weak points: * the proposed methodology (in the general part) just relies on available alignments and suggests to “complete” them “if necessary”. Establishing the “if necessary” is actually a central aspect in alignment dataset production, and would deserve more attention and originality, unless there is no particular methodology to offer, and this is just a report on how the alignments have been produced, before being adapted/improved for the specific tasks * the differences between the methodologies for query rewriting and ontology merging could be discussed more. For instance, while I understand that query rewriting alignments may use complex correspondences using an expression language that goes beyond SROIQ (and indeed beyond any family of DL languages) I do not find equally trivial to state that those alignments should not be necessarily consistent. * even though this has been declared explicitly by the authors, the absence of a rigorous method involving more persons producing the alignments results in: ** a lack of an interesting, quantifiable, report on the difficulty of producing such alignments and on the convergence of humans on the task (e.g. inter-annotator agreement) ** a resource (the produced alignments) with questionable quality, as produced by a single person or by more people (we do not know from the paper), without a methodology for converging towards a final result. Strong points: * addressing the issue of complex mappings is a value per se, as it seems to be (almost) missing from literature, thus not encouraging further work on the matter * providing a resource (the alignments) that can be used to bootstrap a new phase of evaluation of complex alignment tools * providing a first evaluation of existing complex alignment tools over the alignments introduced with the paper I have a question that is related to a statement in section 7 pag. 12 “With this in mind, we make the assumption that when a human beings consider an ontology for a given task, this impacts their interpretation of this ontology.” [fix: beings --> being or “a human..” --> “human”] “Therefore, giving task-oriented..” I think here the concept of “task” is blurring the definition of task as in the paper (e.g. ontology merging and query rewriting, and with the second sentence the authors make it clear that they are talking about this meaning of “task”) with the idea of a task/domain that an ontology can be used for. In this sense, by first ontologies aspire to a more objective perspective on their interpretation and on their modeling itself (a requirement for reusability) than the one offered by “old school” modeling of database schemata, which imposes them to be built differently for satisfying different needs. Secondly, and most important, the (however possible, and that’s why “semiotic heterogeneity” is still and always will be a problem) variabilities in interpretation are more related to different perspectives on the domain and different needs for a task (as of the final task of the application using the ontology) than to the tasks (related to alignment/merging) as defined by the authors. Some clarification would help. Despite the various criticisms I expressed, I still think this paper could make an interesting contribution to the conference Minor typos or stylistic flaws not addressed in the previous points: Pag. 5, section 3.1: “Approaches that rely…” Not introduced by a listing, nor present in a itemized list, thus needs a verb before the “:”. (e.g. “approaches that… are”) End of section 3.1: remove the comma, which should not break the connection with the verb. The same stylistic error has been reiterated along the paers a few times. Please check. Start of section 3.2. Consists of --> consists in (“consists of” is used to mean “be composed of something”, while “consists in” means “the thing mentioned as the only or most important part.” (e.g. if the expression on the left of the “consists in” is a concise term for the definition after it) Section 4: Methodology to create an alignment for ontology merging. “Incoherence” should be “Inconsistency” --------------------- I read the authors rebuttal (thanks to them for the explanations), my criticisms still stand as well as my overall mild yet positive opinion on the paper.
Review 3 (by anonymous reviewer)
(RELEVANCE TO ESWC) The content of the paper is relevant to the ESWC conference. (NOVELTY OF THE PROPOSED SOLUTION) The paper provides a novel challenge concerning the ontology matching topic. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The paper is correct and complete with respect to its purposes. (EVALUATION OF THE STATE-OF-THE-ART) The state of the art is well discussed and presented. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The discussion is sufficient. (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Currently, the dataset has not been linked and it is impossible to check the content. I will eventually change the score if I will have the opportunity to have a look into it. (OVERALL SCORE) The paper presents a methodology for creating complex alignment between ontologies. The contribution of the paper is interesting, but I think that it is more adapted for the Resource track. Indeed, from the pure research perspective, I have some difficulties in finding a real contribution. While, from the resource perspective, I think that the generated dataset would be helpful for the community (especially for researchers working on the ontology matching problem) for improving their algorithms. I invite the authors to include the link to the generated resource in order to enable the Reviewers to provide further feedback to this work. --------------------- I thank the authors for their effort in preparing the rebuttal. After reading their reply, I confirm the score given earlier.
Review 4 (by Ernesto Jimenez-Ruiz)
(RELEVANCE TO ESWC) The paper presents a task-oriented alignment evaluation that requires the discovery of complex alignments, such task is a very relevant contribution to the ontology matching community. (NOVELTY OF THE PROPOSED SOLUTION) The need of benchmarks containing complex alignment sets is not new, but there are not currently such benchmarks in the OAEI campaign and systems need to be challenged with complex tasks. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The papers contains a few technical flaws that should be corrected (see detailed comments below). Main main concern about benchmarks with complex alignments is the complexity to discover the alignments (as shown in the evaluation). As it happened in the past (OAEI 2009) complex benchmark may not attract enough systems and are discontinued. For a complex alignment task, system may need to overspecialize which is not always desired. To allow more generic system to participate the benchmark should provide more than only the ontologies and the reference alignment. It should include (1) training data (1 or many input sets), and/or (2) guidance about what to extract, for example, given 1 (or more entities) specify which pattern is expected for them. Then the system is required to find suitable entities to fit the pattern. Guidance may also make the task very simple, so a trade-off should be found. As suggested by the authors the inclusion of instances may also help participating system. (EVALUATION OF THE STATE-OF-THE-ART) The overview of the state of the art is good and 3 systems are evaluated with the proposed benchmarks. However it is missing a discussion with approaches mapping RDB to RDF [s3]. Not exactly the same problem but it shares some characteristics. For R2RML benchmarking there are some contributions in the literature (e.g. RODI [s4]). RODI was sort of a follow up of the OA4QA track. Although pointed out by the authors in the conclusions, the evaluation of the systems does not include the use of semantic precision and recall. This is very relevant to also evaluate system producing simple alignments. As shown in [s5] simple alignments may also lead to complex alignments. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The discussion about the generated complex alignments could be improved. Sections 5, 6 and 7 could be extended since they are the core of the paper. Till section 5 the paper is very descriptive but the mentioned sections could have given more insights and take home notes. (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The alignments are publicly available and with a license. However it is not available the detail about the evaluated system and how to reproduce their evaluation. (OVERALL SCORE) First of all, I would like to thank the authors for the rebuttal. Some points are more clear now and I will keep my original score. As a personal note, although the response is well organised, I would recommend to (also) provide direct responses to reviewers comments, for example adding references to R1, R2,... It also helps to understand what was replied to other reviewers. --------------------------------------------------------------- Comments: - Section 2.1 presents some examples of complex alignments but only in Section 5 the set of used complex alignment is introduced. The presentation of the complex alignment could also be improved as it is currently in the caption of Figure 2. - Section 2.2. I agree that many system do not take into account a subsequent use of the ontology alignments. But system like LogMap or AML ,although producing simple mappings, do care about producing a coherent merged ontology. Which is the first step for making the alignment usable. - About the merging task. I do not think it is a task per se, but an intermediary step to make the alignment usable. - For the query rewriting task the expressiveness of the ontologies may play a role. The constrain may not be on the rule but on the ontologies (refer to FOL-rewritability property [s1, s2]) - Section 4. Decidability of the reasoning tasks when merging: * O1 and O2 may be in SROIQ but the complex alignment (even being SROIQ axioms) may make the union to be outside SROIQ if complex object properties are used in a cardinality restrictions (See https://www.w3.org/TR/2012/REC-owl2-direct-semantics-20121211/#Inference_Problems). They are allowed independently but not together. - For the query rewriting the alignment or merged ontology does not need to be coherent? Incoherences will definitely affect the queries. - In page 9, it is mentioned that domain and range are not in SROIQ, but dom(R, C) == "R some Top subClassOf C", while range(R, C) == "Top subClassOf R only C" - The union(rel) example "R1 union R2 union R3 sub S" could be split into "R1 sub S", "R2 sub S" and "R3 sub S" and be in SROIQ. - Section 7 does not read like a discussion section. I was expecting more information/discussion about the benchmark and the evaluation. - In conclusions, HermiT could be used to detect violations of the conservativity principle using the general notion of the semantic difference. Strong Points (SPs) - Benchmark Very welcome in ontology matching community. - The paper introduces some patterns for the complex alignments - A preliminary evaluation with suitable systems is provided Weak Points (WPs) - The discussion about the presented benchmark and evaluation could be extended - The paper contains some important errors with respect to the underlying ontology semantics as described above. - It is unclear how this benchmark will attract generic alignment systems. Questions to the Authors (QAs) - EDOAL has some limitation. And even OWL axioms if transformation functions are involved. Have the authors considered the use of SWRL? - Do you have concrete plans to define a new task within the OAEI? The HOBBIT platform would open the door to new tasks involving a more complex evaluation. - How do you plan to address the problems previous complex benchmarks faced? - How do you plan to cope with semantic difference to evaluate complex alignments? that is, which notion of the semantic precision/recall are you planning to use? Minor comments: - The second sentence in second paragraph (page 2) seems to be broken - Missing conference information in  - o1 and o2. In several places the space after o1 is missing. In other places the 1 and the 2 doe snot appear as subindexes - Reference  could be updated with [s6] - Reference  could be updated with [s7] - Page 5: * as complex as -> as complex since (otherwise it reads like a comparison) * etc.. -> extra point * I would try to avoid starting a sentence with a reference or set of references. Preferably Thiéblin et al. [x] or The approaches in [x, y, z]. - Page 7: * The ontology must be decidable. I understand but it is more correct to say "the reasoning task is decidable for a given ontology language/expressiveness". * induced -> entailed - Table 2 is out of margins Suggested literature [s1] Data complexity of query answering in description logics [s2] https://www.w3.org/TR/owl2-profiles/#Computational_Properties [s3] https://www.w3.org/TR/r2rml/ [s4] RODI: Benchmarking relational-to-ontology mapping generation quality. Semantic Web 2018 [s5] Ontology Integration Using Mappings: Towards Getting the Right Logical Consequences. ESWC 2009 [s6] Minimizing conservativity violations in ontology alignments: algorithms and evaluation. KAIS 2017 [s7] Improving the interoperability of biomedical ontologies with compound alignments. Journal Biomedical Semantics 2018
Metareview by Emanuele Dellavalle
All reviewers agree that this paper, even with its current limitations, should be accepted. Please consider the comments provided by reviewers when preparing the camera-ready version.