Paper 28 (Research track)

The knowledge increase estimation framework for integration of ontology instances’ relations

Author(s): Adrianna Kozierkiewicz, Marcin Pietranik

Full text: submitted version

Abstract: The previous authors’ research showed that it is not only possible, but also profitable to estimate a potential growth of a level of knowledge that appears during an integration of ontologies. Such estimation can be done before the eventual integration procedure (or at least during such) which makes it even more valuable, because it allows to decide if a particular integration should be performed in the first place. Until now, authors prepared a formal framework that can be used to estimate the knowledge increase on the level of concepts, instances and relations between concepts. This paper is devoted to the level of relations between instances.

Keywords: ontology; knowledge increase; instances’ relation; ontology integration

Decision: reject

Review 1 (by Petya Osenova)

(RELEVANCE TO ESWC) The idea of estimating the efforts for integration of instances from various ontologies is worth discussing, although it has to be put in some controlled area to be feasible, I think.
(NOVELTY OF THE PROPOSED SOLUTION) The paper seems novel with respect to its attempt to propose a thorough framework for estimating effectiveness of ontology integration.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The proposed procedure is described in detail. However, I think that evaluation is not suffice to prove the utility of the method.
(EVALUATION OF THE STATE-OF-THE-ART) The state-of-the art section gives a detailed overview of related works.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The demonstration of the idea as well as the discussion relate to ideal situations only, not to real data/tasks.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) I think that it would be difficult to reproduce the ideas, unless controlled conditions are provided.
(OVERALL SCORE) The paper evaluates its proposed approach only indirectly - through human inspection. However, it also needs  some real application of an integrated ontology for a specific NLP task.


Review 2 (by Brian Davis)

(RELEVANCE TO ESWC) Previous work by the authors have described a formal framework that can be used to estimate the knowledge increase at the level of concepts, instances and relations between concepts (classes) in the context of ontology integration.  The contribution in this paper proposes to have extended their previous framework for relations between instances.   Overall the research is interesting and valuable but the paper is written in certain sections in a convoluted and/or vague manner which detracts from the contribution.  There is no automatic or data driven evaluation rather a manual experiment which is somewhat acceptable to support the claim. 
However the experiment is described poorly in parts - in particular with respect materials, questionnaires, rational for choice of subjects, links to materials for reproduction of experiment and data inspection.   Given that the success of the paper hinges on this section I would have expected more precision here.
Hence the reader is left unconvinced of any significant contribution.  That in conjunction with a confusing layout and poor writing in places brings one to the conclusion that this paper needs more work for publication.  I suspect the content and results are there but the presentation needs work.
(NOVELTY OF THE PROPOSED SOLUTION) The contribution is valuable.  It appears to be an increment on previous work.  However, the second last paragraph involving the  discussion on page 4 in relation to Section 3 on related work does not leave the reader convinced.  It reads quite vague ("give a big picture of the performed integration" ).  Why is this important?  This section was very important and it does not clearly summarise explicitly in critical manner why other approaches are lacking  and how their contribution addresses this gap.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The work appears to be formally well describes however from a readability perspective it was challenging to inspect Algorithm 1 nearly 3 pages later after its description.  Parts of section 4 are well written which compliment the formula.  The difficulty arises in Section 5, see below.
(EVALUATION OF THE STATE-OF-THE-ART) See comments on Section 3.    The second last paragraph involving the  discussion on page 4 in relation to Section 3 on related work does not leave the reader convinced and is written in a very casual (uncritical and unsystematic) way.  Here it fails to position the authors work in clearly in relation to the state of the art making it difficult to understand their contribution beyond an increment of their own previous work.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) See comments above with respect to Section 4.   The paper needs a thorough proof reading in places.  "For short" and "Answerers" is written a lot which does not exist in English.   A future resubmission requires that authors to:
1) Work on the spacing between the discussion on Algorithm 1 and the position of the figure.  I realise this is not easy but it improves readability. See Section 4
2) Figure 1 is not readable at all.  Consider another method that is clearer to capture the examples of knowledge increase.  I checked the PDF and I needed to zoom to 300% percent! to read the labels.
3) Section 3 - see my comments on state of the art above to improve.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) This section content wise appears to be potentially the real strength of the paper but the Section structure and writing leaves the reader with too much detective work and this is a shame.     
One major issue is why not take some representative samples from some common ontologies in automatic manner which would be more data-driven, thus generating real world samples?  The justification seems weak and the issue is sidestepped.  Consequently can your experiment generalise to real ontological data?
The methodology is confusing in places and raised numerous questions:
1) What are the 20 questions - examples and where are they available to inspect?  Are they based on previous work?
2) One cannot read the figures describing the knowledge examples.  More detail is required here and they should be available for inspection on line for reproduction
3)The population was self selected and biased? Why.  Who were you targeting? Knowledge engineers? or the lay person? Which pool did you draw sample from? What was their prior knowledge of the ontologies? 
4)What consensus have you determined and why is [14] which is previous work mentioned.  Are you describing already published work as result.  This is very unclear.
5) What are the 20 elements from the first sample?  
6) In the second sample calculated from Algorithm 1 - automatically or by whom?
(OVERALL SCORE) Previous work by the authors have described a formal framework that can be used to estimate the knowledge increase at the level of concepts, instances and relations between concepts (classes) in the context of ontology integration.  The contribution in this paper proposes to have extended their previous framework for relations between instances.   Overall the research is interesting and valuable but the paper is written in certain sections in a convoluted and/or vague manner which detracts from the contribution.  There is no automatic or data driven evaluation rather a manual experiment which is somewhat acceptable to support the claim. 
However the experiment is described poorly in parts - in particular with respect materials, questionnaires, rational for choice of subjects, links to materials for reproduction of experiment and data inspection.   Given that the success of the paper hinges on this section I would have expected more precision here.
Hence the reviewer is left unconvinced of any significant contribution.  That in conjunction with a confusing layout and poor writing in places brings one to the conclusion that this paper needs more work for publication.  I suspect the content and results are there but the presentation needs a lot of work.
Strong Points
Interesting and important research
Section 4 is good in parts
Has potential for publication but needs work on layout and writing, argumentation.
Weak Points
Poor presentation of evaluation and experimental methodology
Writing and structure needs improvement.
Does not clarify the contribution and nor does it position itself well wrt to the state of the art.
Same Questions as above
One major issue is why not take some representative samples from some common ontologies in automatic manner which would be more data-driven, thus generating real world samples?  The justification seems weak and the issue is sidestepped.  Consequently can your experiment generalise to real ontological data?
1) What are the 20 questions - examples and where are they available to inspect?  Are they based on previous work?
2) One cannot read the figure  describing the knowledge examples - Figure 1.  More detail is required here and they should be available for inspection on line for reproduction
3)The population was self selected and biased? Why.  Who were you targeting? Knowledge engineers? or the lay person? Which pool did you draw sample from? What was their prior knowledge of the ontologies? 
4)What consensus have you determined and why is [14] which is previous work mentioned.  Are you describing already published work as result.  This is very unclear.
5) What are the 20 elements from the first sample?  
6) In the second sample calculated from Algorithm 1 - automatically or by whom?


Review 3 (by Christoph Lange)

(RELEVANCE TO ESWC) The paper proposes an algorithm for estimating the knowledge increase during the ontology integration process.
(NOVELTY OF THE PROPOSED SOLUTION) The idea of assisting the ontology integration process with a metric that evaluates the added value of knowledge is an open area of research.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The rules, equations and algorithm steps were explained in detail. However, it was not easy to understand the flow of the paper; the sections need a better structure. Also sometimes the paragraph fails to elaborate the main ideas.
The paper is supposed to present a framework for estimating knowledge increase when integrating ontologies. However, the paper presents only an algorithm and the framework was not discussed.
(EVALUATION OF THE STATE-OF-THE-ART) The related work section lists many research efforts but it lacks categorization and discussion, and some of the explanations of these approaches are non-descriptive and sometimes ambiguous. For example, page 3 “Some modifications …” what are these modifications, did this improve the approaches.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The paper proposed an algorithm and formal function to calculate the knowledge increase during ontology integration. The evaluation example explains four direct relationships but how would the algorithm support more complex relationships and hierarchies (e.g., what happen if we have more than one hierarchical level?).
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The proposed algorithm does not address a specific domain; so it can be applied to ontologies in other domains, but the performance of the algorithm in relation to the size and properties of these ontologies is not described in the paper.
(OVERALL SCORE) The paper proposes an algorithm for estimating the knowledge increase during the ontology integration process. The algorithm should evaluate the benefit of alignment between ontologies.
The title of the paper includes a framework but the paper doesn't explain a framework.
The example and evaluation, using a survey, was not clearly presented.


Metareview by Christoph Lange

The reviewers agree on several negative points:
* It is not clear to what extent the assumptions that you are making about the ontologies hold in reality; more details on the eligible ontologies would help to clarify this.
* The value of the contribution is reduced by the indirect, manual evaluation – but not a systematic, automated, data-driven evaluation.
* The clarity of presentation and language need improvement.
Therefore, the paper cannot be accepted in its current form.


Share on

Leave a Reply

Your email address will not be published. Required fields are marked *