Paper 175 (Resources track)

An Ontology of Software Quality Relational Factors from Banking Systems

Author(s): Paolo Ciancarini, Andrea Giovanni Nuzzolese, Valentina Presutti, Daniel Russo

Full text: submitted version

Abstract: Quality, architecture, and process are considered the keystones of software engineering.
ISO defines them in three separate standards. However, their interaction has been poorly studied, so far. The SQuAP model (Software Quality, Architecture, Process) describes twenty-eight main factors that impact on software quality in banking systems, and each factor is described as a relation among some characteristics from the three ISO standards.
Hence, SQuAP makes such relations emerge rigorously, although informally.
In this paper, we present SQaAP-Ont, an OWL ontology that formalises those relations in order to represent and reason via linked data about software engineering in a three-dimensional model consisting of quality, architecture, and process characteristics.

Keywords: Ontologies; Software Quality; Software Process; Software Architecture; Software Engineering

Decision: reject

Review 1 (by anonymous reviewer)

I'd like to thank the authors for the comments in the rebuttal.  There's nothing that changes the content of my original review.  
Based on discussions with other reviewers I have changed my score to +1 in order to achieve more of a consensus with these other reviewers.
-------------------------------------
This paper presents the SQaAP-Ont, which is an ontology that models concepts from software quality.  The focus is on the banking domain but the ontology seems general enough so that it can be applied to other domains.  The ontology specifically addresses the relations between Quality, Architecture and Process.  The authors claim that SQaAP-Ont may serve as a reference resource and a practical tool for scholars and practitioners of software engineering who want to assess the quality of a software system.  It's not clear to me how easy it would be for such people to do this in practice or how useful this ontology would be for such people, however, the paper is a good description of a resource that is published and documented using best practice.
Overall, the paper is fairly readable.  I do have trouble remembering ISO numbers though.  If the authors can squeeze the names of the ISO standards inline where they are mentioned this would be very helpful!    
There is a review of related work at the start of the paper.  To the best of my knowledge this review is comprehensive.  The authors also make a case for novelty.  In that sense the ontology breaks new ground and plugs some sort of gap.  I honesty don't have a sense of how important it is to plug this gap though.  Beyond the fact that this is an ontology and there are other ontologies in the software engineering domain, one of the main points of novelty seems to be that this ontology uses standard ontological design patterns that were developed by the community.  I therefore think that this ontology resource will be of interest to those working in the ontology pattern community - i.e. the interest in the ontology might extend beyond those specifically interested in the software domain aspect of the ontology.
The authors have provided enough detail for me to feel that I have an understanding of how the ontology was developed.  The development stems off-the-back of previous work that involves managers and domain experts from the banking domain.  There's certainly a robust methodology behind the development of the ontology.  I also think it's a good example of how the Delphi method can be used for the knowledge elicitation step in ontology development.  The authors also used other established techniques like competency questions.
With regards to the description of the ontology in the paper, it's hard to get a true feeling of the ontology from reading the description, but this is the case for many ontology papers (or resource papers). 
Figure 2 desperately needs some sort of key or comprehensive description in the caption.  It's not clear what the blue arrows mean e.g. :Metric is connected to :Parameter via :hasParameter but this doesn't give me any idea of how this is encoded in OWL.  In fact, this particular example highlights a discrepancy between the description in the paper and in the actual ontology.  After loading the ontology into Protege I can't see this "relation" modeled anywhere in the ontology.  I look at :Metric and there's no logical description for it.  The same is true for Parameter.  Some clarification of this is needed.  In implementation details the authors refer to "https://w3id.org/squap/" as the namespace - this is actually the ontology IRI and I think it's best to refer to it as such.  
In terms of the actual resource, it meets many of the requirements of this track.  Specifically, the ontology,
- reuses "standard" design patterns.
- is annotated with OPLa to help in its understandability and reusability.
- has an open license (CC-BY-4.0).
- is published online at is URL as recommended by the OWL specification.
- has html documentation published online, with diagrams (some of which are shown in the paper).
- source is RDF/XML and is published in GitHub.
- is externally aligned with other well known ontologies e.g. Dolce.
- contains annotations that use standard annotation properties to label and describe the ontology and who created it.
- loads into Protege without errors.
- is constructed using standard proven patterns for annotations.
- is published using standard content negotiation techniques so that users can request different serialization formats.
The authors do provide an example of how to use the resource.  While it's good that some example has been provided, this section is incredibly brief and it's not that easy to follow.  I don't find it overly convincing. The example files are available online in some form though.  However, https://w3id.org/squap/examples/gqm points to invalid RDF (it's HTML + RDF it seems).
In terms of potential impact, the authors do make a case for this.  I think that the authors do a good job of highlighting the particular (major) case that has driven the development of the ontology, but it remains to be seen how useful the ontology actually is in practice.
Overall, a well written paper that is precisely targeted at the resources track.  Both the paper and the ontology fulfill the requirements for acceptance in my opinion.


Review 2 (by Ernesto Jimenez-Ruiz)

First of all we would like to thank the authors for their response. We agree with some of the limitations suggested by other reviewers; but we believe, with the suggested changes, the paper has the potential of becoming a good description of an overall valuable resource.
--------------------------------------------------------------------
# Overview of paper
The paper presents an ontology for describing quality of software.
The ontology is a formalization of the SQuAP model, which consits of
a set of factors spanning three different ISO standards. The factors
are the result of an empirical study of assesment of software in the banking domain,
and relates characteristics from both architecture, software quality and process.
# Strong points
The ontology
* formalizes factors that are obtained via an empirical study of a concrete domain;
* relates characteristics from three different ISO standards;
* seems specific enough to capture the key aspects of modelling quality of software,
yet sufficiently generic to capture different use-cases; 
* is formalized in OWL with well-known ontology design patterns;
* and has alignment axioms to the DOLCE Ultralight ontology.
The paper
* provides a helpful overview of the core of the ontology;
* gives an example of use of the ontology;
* and is convincing it its potential impact.
# Weak points
The ontology
* seems to need construction of IRIs in SPARQL-CONSTRUCT queries for parts of the reasoning;
* lacks a more thorough example or a tutorial.
The paper
* is missing some motivation for some of the choices made, e.g.~why provide alignment
to DOLCE and not other upper ontologies such as BFO or ISO 15926;
* is lacking a example of a formalization of a factor;
* lacks a concrete description on how the ontology would be used in an actual
assesment of the quality of a software system, that is, which typical queries
over the ontology would be of interest to an end-user;
* and the modelling is somewhat difficult to understand from the description in 4.1,
examples of an individual for each class would help.
# Conclusion
The ontology described in this paper seems valuable for its intended application and
formalizes standards which does not seem to have been previously combined in the same
model before. The intended domain of the resource contains many potential use-cases,
and concrete applications, supporting the adoption of use of semantic technologies in
the assesment of the quality of software systems.
The ontology is developed according to a concrete design methodology and
abstracts over certain key aspects (e.g.~metrics) which seems to make it general enough for
reuse accross multiple use-cases. The ontology is published online under an open license.
However, there are still some questions that is left unanswered after reading the paper:
* Why is nothing of the previously made ontologies on the same domain reused, such as the
formalization of the ISO 42010 or any of the other ontologies for the ISO standards
referenced in the Related Works section?
* Why is punning necessary? DL queries can be posed over classes, and as far as I can see,
the SPARQL-query does not need punning.
* How would the applicability of the ontology be evaluated and compared to existing approaches?
* How different are the use-cases of the ontology, and is the ontology sufficiently general to
handle these?
# Other Comments
* There is a ')' without a '(' in the second paragraph in 4.1.
* 'relaiont', in third paragraph of 4.1.
* Missing ')' at end of third paragraph of 4.1.
* 'Section 4.2' -> 'Section 4.1' in first sentence of 4.2.
* Some of the axioms in 4.2 are redundant (e.g. ArchitecturalAlignment SubclassOf SoftwareQualityCharacteristic is entailed by the equivalence axiom for SoftwareQualityCharacteristic).
* There are some missing axioms in 4.2. E.g. isAffectedBy should be inverse of afftectsMeasurementOf described in 4.1.
* In example RDF-graph in Sec. 5 the :sonarqube-value-b is squap:graph-related instead of squap:value-related to "B", and 'parametrizes' is missing colons (:).
* Missing 'é's in Protégé in Sec. 5.


Review 3 (by anonymous reviewer)

----------------------
I acknowledge the rebuttal and would like to thank the authors for this effort. However, I was expecting more concrete answers instead of a description of what the authors count to implement in case the paper is accepted. For that reasons, I will keep my original scores. 
---------------------
This paper presents the SQuAP-ont, an ontology that formalizes a set of relations between ISO standards describing software quality, architecture and process characteristics. 
These relations come from the SQuAP model, what originally describes quality factors of the banking system domain. This expertise has been gathered from experts in this specific domain. 
The ontology results from the application of ontology design patterns, in particular Description and Situation and Parameter Region patterns. The resulting ontology has been (manually) aligned to the DOLCE-Ultra-Lite top-level ontology. The resource is published under CC-By-4.0 license and is accessible online. 
Although the paper presents an interesting case study applying semantic technologies to the banking domain, many points should be better clarified/introduced in the paper in order to convince the reader of its novelty in terms of resources :
- what are the specificities of the domain that have been taken into account when constructing the SQuAP model ? i.e., is this model domain-oriented or is designed to be reused in other domains ? is it a generalization of the relations in this specific domain ? In fact, the methodology for the construction of SQuAP seems to rely on experts from this particular domain (i.e., empirical study in the banking factor) ;
- the theoretical coding approach [36] that have been followed to map the factors to the ISO standards, the core of the paper, could be better developed (very shortly introduced);
- why the factor 26 have been chosen to be detail ? could we have more information on the other factors ? There is no clear transition when presenting this specific factor. Furthermore, Figure 1 does not bring so much;
- the competence questions described in Table 1 have been established by how many experts ? Do they correspond to a subset of the competence questions used to guide the construction of the ontology ? Or this list is exhaustive ?
- the authors describe how the Description and Situation pattern have been used but very few words is said about how the Parameter Region pattern have been used;
- the example in Section 4.1 could introduce a D&S n-relation in the banking system domain;
- how the alignments between SQuAP-ont and DOLCE+DnS UltraLight have been established ? It seems that they have been manually generated. Why the authors use this specific version of DOLCE ?
- no evaluation on the proposed resource is reported. It could be interesting to have an evaluation on the benefits of using such resource in the system development process, as a feedback from experts.


Review 4 (by anonymous reviewer)

The paper proposes SQuAP-Ont, an ontology which should be devoted to verify functionality, reliability and performance of information systems in the banking and finance fields. The modeling derives from a previous effort determining SQuAP, i.e., a model evidencing factors that could have some impact on software quality in the above contexts.
The paper is basically well written (a part from some unclear sentences or incorrect English formulations and not properly readable tables and figures), but it is not well organized. There is a lot of space employed for motivating the approach: from the introduction to the Section 6, the authors continuously put in evidence the relevance and impact of their proposal with reference to the general scenario it moves from. This is done strongly sacrificing most relevant technical and experimental descriptive parts of the manuscript. 
The overall scientific/technical contribution is too small to make the work accepted, the proposed approach is not completely novel and its added value is questionable. No details are provided w.r.t. the comparison with possible competitor solutions (just as an example, Model Checking based software verification is returning on the scientists and practitioners attention and could be taken into account for a complete assessment of the proposal). In general, the related work section is not properly organized: it is too small if compared with the overall introductory section and it omits many possible similar approaches. It must be also considered that a not negligible part of the manuscript is reserved to the description of the SQuAP model (Section 3) with a quite boring discussion leaving room to the further more interesting technical sections. The model origination the ontology engineering is presented in a too qualitative way. The authors basically refer to the paper in [32] and largely recap specific elements not giving an exhaustive vision of the approach. Further parts are treated in a rushed way (see for example the formalization one) or present too obvious elements to be of some interest. This is mostly the case of the implementation section, which does not provide a good experimental view and a relevance assessment of the proposal.


Share on

2 thoughts to “Paper 175 (Resources track)”

  1. Definitively an interesting and well-founded ontology. I would really like to see this extended to other fields beyond Banking systems.

    I would have a minor question. If I understood correctly, you can specify whether a measurement affects a factor (via “affectsMeasurementOf”). Is it also possible to quantify this affectation (e.g. in a scale)? Would this be interesting to consider in your experience with stakeholders?

  2. I have two issues with this ontology, one regarding conceptualization and the other regarding impact.

    Conceptualization:

    In my opinion OWL Punning should be used for dealing with cases where a concept can be represented both as a class and an individual. In this ontology having statements like “Accountability subclassOf Security” (http://stlab.istc.cnr.it/squap/#d4e1562) is conceptually wrong; both concepts cannot have any instances, hence they should not be classes. Moreover, accountability is not a kind of security (as the subclass relation suggests), not even a narrower concept. If accountability here has a security-specific meaning, then it should get a better name and a clearer description.

    The problem with concept names and problematic relations is present throughout the ontology. For example. there is the statement “Improvement subclassOf Organizational” (http://stlab.istc.cnr.it/squap/#d4e1396). If you try to express this in plain English, it makes no sense.

    Potential impact:

    I am missing a mode detailed discussion on the barriers of adopting this ontology and on how to overcome them:

    – What is the current penetration/usage of OWL in the Italian IT banking sector? How equipped (in terms of tooling and know-how) is the sector to start using OWL models?
    – What, if any, are the currently used software quality data models in the sector? Are there specific complaints about them that this ontology explicitly addresses?

Leave a Reply

Your email address will not be published. Required fields are marked *