Paper 168 (Research track)

Device-Independent Visual Ontology Modeling

Author(s): Vitalis Wiens, Steffen Lohmann, Sören Auer

Full text: submitted version

Abstract: The development of ontologies typically involves ontology engineers and domain experts with different backgrounds in semantic technologies and ontology modeling.
Domain experts, who provide the conceptualization of the knowledge domain, often lack modeling skills and find it hard to follow logical notations in OWL representation.
Visualizations of ontologies, in particular graph visualizations in the form of node-link diagrams, are commonly used to support ontology modeling and related tasks.
In order to more directly involve domain experts in ontology modeling, approaches that are immediately available, easy to use, and independent of the device and interaction context are needed.
We present a device-independent approach for visual ontology modeling that reduces the entrance barrier to engage in ontology modeling.
The device-independence is achieved by different modes of operation, including mouse and touch interactions, exploitation of device sensors and smart interaction techniques (such as speech recognition).
Guidance in the ontology modeling and editing process is provided with respect to the OWL specifications using built-in constraints.
The results of a comparative user study clearly indicate the benefits of the presented approach.

Keywords: Ontology Engineering; Visual Modeling; Mobile Devices; Touch Interaction; Device-Independence; Visualization; OWL; VOWL; WebVOWL

Decision: reject

Review 1 (by Silvio Peroni)

(RELEVANCE TO ESWC) The work presented in the papers is very relevant to the conference.
(NOVELTY OF THE PROPOSED SOLUTION) I was looking for something like that since ages. Happy that someone is working actively on the topic, which is a quite novel contribution to the community.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The solution presented is very good, but lacks still some features. First of all, I've tried it on an iPhone and an iPad, and some functionalities (e.g. double-tapping a class for changing its name) do not work properly. In addition, it seems not to be possible to create restrictions – something that VOWL actually does. I think is due to the current implementation stage of the tool, though.
(EVALUATION OF THE STATE-OF-THE-ART) It is a fine discussion of the related works.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) There are some aspects of the interface that should be improved - and I hope that could be even done quickly by the authors:
- it was not immediate to understand how to create a subclass relation - something more direct on the interface would be better, I believe;
- related to the previous issue, it would be good to have, for instance, a way for visualising the diagram as a tree according to the subclass of hierarchy when requested, or alternatively to have a new panel in the left that shows that hierarchy using the usual tree-link structure;
- it would be good to provide (as default) the same base IRI to all the entities created starting from the IRI of the ontology, and to assign as local name of the IRI of an entity an adapted version of its label (e.g. the class "document component" would have local name "DocumentComponent").
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) This is the most delicate aspect of the whole work. The evaluation the authors have performed was convincing, but it was a bit limited in terms of users and devices used. First of all, the data gathered by the user testing sessions have not be published online (e.g. on Figshare), and only the experiment description and questionnaire was. Secondly, there is no evidence of the actual composition of the users in the paper (their expertise, age, etc.). Again, the task to perform was rather simple for an expert - maybe it would be worth to provide different tasks to address according to the expertise of the users involved. 
Finally, and crucially, the test was done only on one device, while the whole paper is about to offer a multi-device tool for enabling ontology development. This, to me, was the major concern of the whole work, that hopefully can be addressed before coming to a final decision on this paper - by running a similar test on the same number of users, but by using a different device, e.g. a computer. In this way it would be possible to understand if a certain tool behaves better on a particular device or it excels on any possible device.
Another comment: I was surprised the SUS questionnaire was not provided to users, since it would allow one to get some numeric information about the usability of the tools even in presence of a small set of participants (see
(OVERALL SCORE) In this paper, the authors present a series of interaction guidelines for an ontology editor so as to guarantee its full use independently on the particular device adopted for creating an ontology. Then, they introduce a preliminar prototype implementing such guidelines called WebVOWL Editor (based on WebVOWL), and evaluated it in comparison with other two device-independent and Web-based tools, i.e. WebProtege and Turtle Editor. The experiments showed that WebVOWL Editor is superior of the others when addressing the creation of a small ontology by means of a tablet.
Strong Points (SPs)
1. Very interesting conceptualisation of multi-device interaction for ontology editors
2. It's a rather novel solution for a visual ontology editor (as an open source tool)
3. The evaluation is very interesting
Weak Points (WPs)
1. Only a limited number of users involved in the evaluation
2. No data about the outcomes available
3. Just one device tested
Questions to the Authors (QAs)
1. Can the tool be customised by using other kinds of widgets (e.g. Graffoo's one)?
2. How can I create class as restrictions?
A final remark: I would love to use it, since it is something which is really missing in the community - and that I (as well as other ontology developers) really need. Having such tool available in a way that all the features of VOWL are fully covered is, to me, the real killer app for pushing VOWL as one of the most adopted standard languages for visualising and creating ontologies. I hope the authors can address proactively my comments, since I would love to see the paper presented at the conference.
--- after rebuttal phase
No rebuttal has been provided by the authors. Thus I confirm my score.

Review 2 (by Tomi Kauppinen)

(RELEVANCE TO ESWC) This paper contributes to the intersection of ontology engineering and visual modelling, both which are very relevant for advancing SW.
(NOVELTY OF THE PROPOSED SOLUTION) This paper proposes a novel approach to create ontologies in a strongly visual way. The requirements for the system were gathered in discussion domain experts from relevant domains. While individual requirements are not very unique, the combination is, at least for visual modelling within SW. The tool is built on extending earlier work by authors: "The proposed approach is implemented by extending the existing web-based framework WebVOWL".
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The paper makes use of both existing, well-studied technologies and methods, and develops a combination which is sound. However, the evaluation is a rather superficial and leaves a door open to wonder about the value of the approach. This is pity since the tool & WOVL are very promising.
(EVALUATION OF THE STATE-OF-THE-ART) The state-of-the-art is well covered in related work section.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The approach was both implemented and evaluated. The evaluation was done with a rather small group (six participants). Authors state that "Participants with experience in both visual modeling and ontologies had been classified as expert users, whereas having experience in only one of the areas leads to a classification as intermediate user." but it hardly makes any sense to divide a group of six persons to two groups based on their experience.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The approach somewhat supports reproducibility:
- the editor is online
- the user evaluation form is available online
so in a way other people can run a similar kind of evaluation and compare the results. However, for generality the evaluation in this paper was done with a very limited number of people, and from the same institute as the authors. There is also no data about this testing available for a comparison with a reproduced experiment.
(OVERALL SCORE) Summary of the Paper 
Short description of the problem tackled in the paper, main contributions, and results
Strong Points (SPs) 
- This paper contributes to the intersection of ontology engineering and visual modeling
- The approach was both implemented and evaluated
- Both the editor and the user evaluation form are available online
Weak Points (WPs)
- The evaluation was done with a rather small group (six participants), and from the same institute as the authors
- Dividing a group of (only) six persons to two groups based on their experience does not really make sense.
- There is also no data about this testing available for a comparison with a reproduced experiment.
Questions to the Authors (QAs) 
- Do you foresee that the results of evaluation would change if you would run the evaluations with, say, 30-40 participants?
- Are you planning to release the editor as open source?
Minor details:
- correct spelling, e.g. "operationton" -> "operation"

Review 3 (by María Poveda-Villalón)

(RELEVANCE TO ESWC) The topic of the paper is relevant for ESWC.
(NOVELTY OF THE PROPOSED SOLUTION) The proposed system uses a new graphical oriented interface for sketching ontologies in a graph-based way.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The system only handle a very limited set of OWL primitives and an important bug was found.
(EVALUATION OF THE STATE-OF-THE-ART) There are only few of this systems that are being used and useful actually but authors missed to mention some similar approaches like and
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The demonstration, in terms of evaluation of the system is quite weak (see overall comments).
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The experiment could be repeated in other settings but in this case it is not general enough (see overall comments).
(OVERALL SCORE) This paper presents an ontology modeling tool with a web based interface build on top of VOWL specification. The system is independent of the device used for modelling ontologies.
My reasons to reject the paper are:
1) The evaluation is too weak. There are only 6 users, that by the way some belong to the authors' organization and which is not representative. Users are classified by types but no data about the resulting classifications is provided. Also the resulting ontologies generated by user are not provided so that I cannot check with error users did with the tool and with the other tools. In addition, the testing exercise is too simple. Finally, in the evaluation is is measured how simple, easy, etc. is the tool with respect to others but it is not measure how complete it is regarding OWL and how it could meet ontologist expectations. That would be a nice complementary analysis.
2) The system seems to have bugs and lead to important modelling mistakes (see below).
3) The paper presents a system rather a research work it might fit better the resources track.
Other comments:
I have some disagreement with authors, for example in page 7, paragraph 2, author claim "However, most of the approaches described in Section 3 require an understanding of a relatively complex visual notation (e.g., UML). We address the different backgrounds of domain experts and other user groups by using the VOWL notation that is easy to understand." How is this proven? It is difficult for me to assume that VOWL is just easier to understand than UML. It seems to me that full UML expressivity is more difficult to understand that a limited set of OWL constructs in VOWL, which is not a fair comparison. To claim such the same expressivity in both paradigms should be compared, for example, if we compare the 5 elements taken into account in Table 1 in VOWL and UML at least I would still prefer UML based profiles like the one in [1], for example the datatype indicated within boxed, that represent classes, makes easier to understand at a glance that it a datatype property instead of a object property. Also hierarchies would more intuitive in UML than VOWL.
The last paragraph in page 7 seems also misleading and difficult to understand. The issue is that authors seems to define classes attributes as those relations (named self-reflexive by authors) that has the same domain and range, see "...the T-Box of an ontology can be seen as a set of conceptualizations (classes) and their attributes, which are represented by self-reflexive relations...". It took some time for the to guess that "attributes" might refer to object properties with the same class as domain and range, actually I'm still not sure about this. In that is the case, why calling it "attributes" if the corresponding implementation is an object property? The ontology construct used for that is the same regardless the (optional) domain and range values. In general, for my experience in ontological engineering, one thinks more about datatype properties when mentioning "attribute". If that is not the case, what is the meaning of class attribute here?
An important question is: how do you model an object property with more than one class in the domain? Creating the same property and attaching them to 2 classes. I have tried so and see that:
.- The domain are then set by default in two axioms, then being understood as the conjunction of the classes, which is a very important, for the consequences, and common modelling mistakes done by beginners and that this tools seems to promote. In addition, how do I set that I want them to be the union? See code:
#--------------------------- Property 4------------------------- 
<> rdf:type owl:ObjectProperty ;
rdfs:label "relacion"@en  ;
rdfs:domain <> ; 
rdfs:range :Class8 . 
#--------------------------- Property 5------------------------- 
<> rdf:type owl:ObjectProperty ;
rdfs:label "newObjectProperty"@en  ;
rdfs:domain :Class1 ; 
rdfs:range :Class8 . 
.- In this test I also realized that, for properties appearing more than once, if I change the label of one arrow the labels in the same property remains the same in other visualizations of the property, that is, I should be changing the labels everywhere manually in order to have a consistent view of the diagram. 
.- In this test I also realized that for some reasons there appear properties stated between classes (see lines 3 and 4 of the following code). It seems that the tool generates this also apart from the domain and range axioms, that is stating the existence of a property between classes, which actually runs into OWL full. Is this a bug? 
<> rdf:type owl:Class ;
rdfs:label "Aaaa"@en  ;
<> :Class1 ;
<> :Class8 .
Also authors mention in page 10 that there is a rule for came case naming and that it is force as is it a good practice. Two things here, first, I've been able to change the naming convention, actually using more than one and the system did not complain. Second, the useful thing from the system would be checking that only one naming convention is used rather than forcing to an specific one, consider that different collective/organization might want to follow their own conventions. Finally, the end of the paragraph is just so confusing for me about the exception for owl:Thing and rdfs:subClassOf. What do you mean with that?
Regarding the results and discussion, I also have a question about the claim "The double tap interactions for creating a node or editing a label were transferred to the other approaches, that did not support this functionality. This emphasizes that the chosen interactions are intuitive and easy to learn." How is this conclusion derived? Have you tried if users start modelling with other approach whether they transfer them to the next tool? Did different users try the system in different orders? In addition, the next sentence is "Nevertheless, a short introduction, to the interactions, in particular how to create a node, was necessary." and it seems a bit contradictory, either it is intuitive or it has to be explained in advance.
I haven't been able to make hierarchies trying out a bit the tool. I don't say it is not possible, it is not intuitive at least for me.  
Table 4: What do you mean by "Class type"? I see in the editor that you can choose owl:CLass or owl:Thing, option that converts the class in the OWL top concept. What it is this for?
Regarding usability, I think the fact that changing the elements tag in the diagram only changes the label is a bit annoying as I have to go to the details and change also the URI all the time. There are already systems that from the label generates a camel case equivalent for the ontology element and users need to edit only once when creating the ontology element.
.- Footnote numbering is re-started in page 11.
.- page 4, "Visual Notation for OWL Ontologies (VOWL)" section has no numbering.
.- I found out the color scheme for the bar diagram in figure 3 out of causality when open the pdf. In the printed version one just can't tell what bar is which system. You could use patterns to increase readability or just put the name of the tool at the bottom of each bar.
.- Typo Page 10 "an default" -> "a default"

Metareview by Hsofia Pinto

The paper presents an ontology modeling tool with a web based interface build on top of OWL specifications. While very relevant and interesting to the Semantic Web community, the reviewers do not perceived the current submission as having met the standards of ESWC in order to be recommended. Authors are encouraged to use the reviews provided.

Share on

Leave a Reply

Your email address will not be published. Required fields are marked *