Paper 207 (Research track)

Audio Commons ontology- a data model for an audio content ecosystem

Author(s): Miguel Ceriani, György Fazekas

Full text: submitted version

Abstract: Multiple online services host repositories of audio clips of different kinds, ranging from music tracks, albums, playlists, to instrument samples and loops, to a variety of recorded or synthesized sounds. Programmatic access to these resources maybe used by client applications for tasks ranging from customized musical listening and exploration, to music/sounds creation from existing sounds and samples, to audio-based user interaction in apps and games. To facilitate interoperability among repositories and clients in this domain, we designed an ontology that covers it. There was no previous comprehensive data model for this domain, however the new ontology relates to existing ontologies such as the Functional Requirements for Bibliographic Records for the authoring and publication process of creative works, the Music Ontology for the authoring and publication of music, the EBU Core ontology to describe media files and formats, the Creative Commons Licensing ontology to describe licenses. This paper documents the ontology and outlines a preliminary evaluation.

Keywords: ontology; audio; creative commons

Decision: reject

Review 1 (by Christian Mader)

(RELEVANCE TO ESWC) The paper reports on the design and development of the Audio Commons ontology. While the ontology is certainly of use within the project I miss a bit the coverage of (i) the specific challenges of the domain and (ii) lessons learned that can be of use for other ontology development endeavours in various domains, covering both the ontology design and the development process.
(NOVELTY OF THE PROPOSED SOLUTION) The ontology satisfies requirements that arise from integrating data that is contained in multiple portals, but the novel aspect of this work is not clear for me. AC ontology builds on existing ontologies and reuses and adopts them in a smart way, with good rationales provided. Still I wonder about the particular research challenges addressed in the paper (in contrast to the engineering goal of data integration).
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) Although checks for inconsistencies and best practice violations have been performed by the authors, I'm not convinced by the evaluation(s). Especially the functional evaluation should give some hints if the ontology can really provide some added value like improving search and retrieval capabilities for cross-portal searches or answering a particular set of required competency questions.
(EVALUATION OF THE STATE-OF-THE-ART) State of the art is provided in the form of related ontologies and data models. Their relation to the AC ontology is well covered and the need to complement these existing resources with the AC ontology is clearly argued in multiple places.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The AC ontology's design is well described. However as already mentioned I miss a clear upfront coverage of the set of requirements which should be the basis of each ontology development process. A good discussion would relate each ontology design decision to the requirement that caused the decision. I see that this is done in subsections 4.n "implicitly" in some locations but a grounding to the requirements would make things even more clear.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The ontology is available for review and download on the Web. Therefore, the formal evaluation can be reproduced. The functional evaluation as provided in the paper is insufficient. It may cater for data integration tasks but for the research track I expect an evaluation that refers to particular research challenges tackled by the ontology.
(OVERALL SCORE) The paper reports on the development of the Audio Commons ontology, which has been created in the course of the Audio Commons project. The main contribution is the ontology itself which aims to improve interoperability between existing audio repositories.
** Strong Points (SPs):
1) takes into account and alignes to existing ontologies
2) method of using a layered approach for the ontology's design
3) provides rationale for design decisions (e.g., audio collections)
** Weak Points (WPs):
1) missing description of the actual (research) challenge (e.g., domain complexity, ontology development process, query performance/efficiency,...)
2) actual set of requirements not described, key competency queries that the ontology should support to answer would have been helpful
3) not very clear why the described ontologies were exactly suitable for adoption/reuse and what the alternatives could have been (e.g., was DCAT also considered?)
4) Listings hardly readable
5) Development methodology and process not explicitly covered (collaborative challenges, continous testing,...)
6) No summary with lessons learned for application with other contexts
I very much thank the authors for addressing the comments, my decision however is not affected.


Review 2 (by Silvio Peroni)

(RELEVANCE TO ESWC) The topic is enough relevant, since we are talking about an ontology after all.
(NOVELTY OF THE PROPOSED SOLUTION) I cannot see a huge amount of novelty in the work presented, in particular since some crucial ontologies – that could be used for addressing specific aspects, such as the collections – were not reused at all, and part of the solution is addressed exactly in the same way.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) Understanding the correctness and completeness of an ontology is always a difficult task to do, since it depends on the particular scenario one has to address. However, there are specific development processes (e.g. http://dx.doi.org/10.1109/MIS.2002.999223, http://aaaipress.org/Papers/Symposia/Spring/1997/SS-97-06/SS97-06-005.pdf, http://www.aiai.ed.ac.uk/publications/documents/1995/95-ont-ijcai95-ont-method.pdf, http://ceur-ws.org/Vol-516/pap21.pdf, http://dx.doi.org/10.1007/978-3-319-34129-3_39, 10.1007/978-3-319-54627-8_5) that, if followed, at least guarantee a minimal form of correctness. However, I was really disappointed that none of the aforementioned ontology development tools and principles have been followed for the development of the ontology.
Another aspect which is delicate is the way the authors have handled collections. The natural constructs from the RDF vocabulary (rdf:List and rdf:nil) are unavailable in  OWL DL because they are used in its RDF serialization - thus using them as the authors have done can result in issues from the OWL point of view. Instead, other approaches can be adopted, e.g. the one introduced by the Collections Ontology (http://purl.org/co, or its paper at https://doi.org/10.3233/SW-130121), which may be imported in the ACO for addressing this specific aspect.
(EVALUATION OF THE STATE-OF-THE-ART) Some important reference is indeed missed, such as the Collections Ontology (for handling collections), the RDF representation of mimetypes for handling digital formats (see http://www.sparontologies.net/mediatype/), and the reason why dcterms:format is not used at all for linking a resource with its format.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) There is a quite extensive description of the ontology in the paper. However, the authors fail in providing some examples of possible uses of the ontology. In this sense, it could be valuable to have an exemplar scenario in natural language that is then modelled by means of the ontology. In addition, listing SPARQL queries translating key competency questions (each one mapping a particular requirement the ontology should address) is another valid and crucial contribution in that direction. Finally, there is not appropriate documentation available, which makes the ontology difficult to understand.
The fact of avoiding to follow a precise process for the development of the ontology basically boycotted the creation of this important material – since it would have been created and used since the beginning. I would suggest to the authors to look at some ontology papers that have been published recently in various Semantic Web conferences, so as to understand how to structure a paper of this kind, in particular in accordance with the guidelines provided by the Resources track (e.g. "Ontologies developed for an application, with a focus on describing the modelling process underlying their creation and their usage"), which is the best track where to publish these kinds of contributions.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The ontology has been tested by means of automatic tools (reasoners and validators). However some aspects (e.g. the handling of collections mentioned above) has not been part of the evaluation at all. It is not clear the role of the "functional evaluation" provided in section 5.2.
(OVERALL SCORE) In this paper the authors introduce the Audio Commons ontology, i.e. an ontology developed for describing audio resources (music, sounds, etc.) in RDF. It reuses several ontologies already developed and published that cover part of the aspects related to the audio domain (music, mainly), and extends them so as to be reused in broader contexts.
Strong Points (SPs)
1. It covers a quite generic scenario where an audio item must be described
2. Reuse FRBR, which is a great specification for handling these kinds of information
3. Reuse of other existing and relevant ontologies
Weak Points (WPs)
1. It should be a resource paper, not a research one
2. No description about the process used for creating the ontology
3. No appropriate documentation available
Questions to the Authors (QAs)
1. Where is the documentation of the ontology? How can a person understand what the ontology is about and how it can be used without reading it?
2. What was the ontology development process used for creating the ontology?
3. Where is the material collected during the process?
4. What are the guidelines used for guaranteeing the correctness (from which perspective?) of the ontology?
--- after rebuttal phase
First of all, I would like to thank the authors for their rebuttal, where it appears clear since the first line they agreed and accepted all the reviewers' comments – that hopefully can be used for improving the work for a future submission.
A bunch of additional comments about specific sections of the rebuttal:
> Regarding novelty, we consider our work to be novel in the specific domain considered and relevant (although certainly not completely novel) in the general context of ontology design.
Answering to a criticism by saying that the authors don't agree with it is not enough, for sure, to convince a reviewer. The authors should provide clear evidences why they disagree with the reviewer comment. This is an important aspect, otherwise the rebuttal won't be effective as the authors may think - at least from my perspective.
> We considered submitting to the Resources track, but then chosen the Research sub-track as a plausible venue for this work: in which a specific ontology is discussed, but with substantial research perspectives, rather than just as a publicly available vocabulary.
I have to confess I still miss the "research perspectives" behind this work. In addition, there were clear guidelines in the CFP for having an ontology eligible to be submitted at the research track:
"the ontology described must be available for reviewing and the paper must comply with at least one of the following types
1. Systematic and clear description of the ontology, including its construction process, describing a particular complex and challenging domain (e.g. formalising 150 pages of a particular standard). Focus: how the ontology has been developed, which new techniques have been adopted to deal with the complexity of the domain, how inconsistencies (if any) have been solved, etc.
2. Introduction of an (even simple) ontology in the context of its applications worldwide. Focus: in which applications it has been used, what are the advantages of adopting such ontology, outcomes of a comparative evaluation of adopting such an ontology for addressing particular tasks (which should at least be compared with a run of the same tasks without the use of such an ontology), etc."
Honestly, the authors failed in showing how their work is compliant with any of the aforementioned points. Thus, a Resources track submission could be a better choice, honestly.
> On the design process, we acknowledge that we omitted its description for brevity and in order to favour a detailed description of related works and the ontology itself. We agree that we will need to find a better balance and we will amend the paper in this regard.
Again, answering to a question by saying that the authors will provide an answer to it in the revised version, without telling the reviewers, in the rebuttal, how they intend to address that point is not enough. In this case, a two paragraph brief description of the process followed for the development or (even better) a link to an external document explaining such development clearly could have been more persuasive.
> A complete functional evaluation will be possible through an extensive use of the ontology in the project and uptake outside. The evaluation presented in the paper it is fulfilling the basic requirements we had to assess the ontology before deployment, but we argue that this allows us to confirm that the solution is so far appropriate for the given domain and expected use cases.
Honestly, I do think that the evaluation presented in the paper is not enough for justifying it to be accepted as research paper at ESWC/ISWC. Still, I strongly suggest for the next time to consider the Resources track instead.
Thus, I confirm my score.


Review 3 (by Phillip Lord)

(RELEVANCE TO ESWC) A paper about an audio commons ontology is clearly relevant.
(NOVELTY OF THE PROPOSED SOLUTION) The paper is not that novel, but then this is an in-use track, so I struggle to see the importance of novelty. In fact, I'd say reuse is a good thing and novelty not so. Hence I give this a good score for not being so novel.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) It is not that "complete" as the authors acknowledge.
(EVALUATION OF THE STATE-OF-THE-ART) This is well covered.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) They have some clear preliminary data showing the system working.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) This is a N/A really, but I don't have that option.
(OVERALL SCORE) This is a paper describing the development of the audio commons
ontology -- which is designed to enable description and access via an
API to music that is available for general purpose reuse.
The paper is well-written and clear; the development of the ontology
is careful, integrated well with previous efforts such as the Music
Ontology and FRBR, making extensions where necessary and reusing where
appropriate. Their briefly describe their development process as
implementation driven which seems sensible given the domain.
The main limitation of the paper is, I think, that the work is
relatively early stage; the authors are clear about this in the paper,
however, so there is no overclaiming. I would, however, have liked to
have seen more information of their plans for the future, and how
they will manage their development process.
Other than this, I think it's a nice paper and well worth of
presentation.
I have a few questions about their modelling; their idea of music
seems fairly focused on a sound-drive approach -- i.e. someone makes a
sound which is recorded. I wonder how this will cope with music
generated from a computational representation, such as Lilypond or
SonicPI. Similarly, audio recordings which contain other audio
recordings, or where a recording is produced by a transformation of
another raw sound.


Review 4 (by Christoph Lange)

(RELEVANCE TO ESWC) This paper presents a new ontology, called "Audio Commons" ontology and therefore it fits into "Vocabularies, Schemas, Ontologies" subtrack.
(NOVELTY OF THE PROPOSED SOLUTION) While construction of new ontologies and vocabularies is appreciated in research communities, the paper lacks novelty and the authors do not have any scientific contribution to offer.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) Although measuring correctness and completeness of an ontology is a challenging task, there are some best-practices that ensure a minimum degree of completeness. For example, defining some competency questions and translating them into SPARQL queries in one way to show the correctness of an ontology, which is lacking in the paper.
(EVALUATION OF THE STATE-OF-THE-ART) The authors have investigated existing ontologies and vocabularies in the audio domain and have reused some concepts from state-of-the-art.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The paper discusses the usual steps in defining classes and properties, tailored for the European Audio Commons project. But justification of reusing other vocabularies should be stronger. For example, regarding the licensing information, there exists a more comprehensive ontology, called ODRL (Open Digital Rights Language). It is a W3C recommendation and is also linked to the CC vocabulary [1]. Therefore, a short explanation is expected which addresses the reasons for not using this (highly-appreciated) ontology.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The authors have evaluated their ontology by performing a formal evaluation (correctness, logical consistency, ...) and also by applying it to real world use-cases.
However I expect the functional evaluation part to be more convincing, e.g., defining some competency questions and addressing them for the ontology evaluation.
(OVERALL SCORE) Summary: 
This paper presents a new ontology, called "Audio Commons" ontology, in order to facilitate the interoperability among different audio repositories. Numerous concepts from existing ontologies (FRBR, Music ontology, Creative Commons, EBU Core, etc.) have been reused in the Audio Commons ontology. The design of the ontology is based on a layered approach with the layers corresponding to: Event (recording, publication, ...), Content (sound, signal, ...) and categorization (by source, by genre, ...). As a first step towards using the ontology, a web API has been implemented which provides integrated access to existing audio repositories. The authors have evaluated their ontology by performing a formal evaluation (correctness, logical consistency, ...) and also by applying it to real world use-cases. 
Strong Points: 
- The motivation behind this work is interesting and useful.
- The implementation of a web API which employs the new ontology and integrates different audio repositories, is encouraging.
- The paper is well-structured and well-written.
Weak Points: 
- Lack of novelty.
- In an ontology construction paper, the basic specifications of the vocabulary should be reported, e.g., the number of classes and properties. Although a formal description is available online, the paper should be self-contained.
- It is not clear how an independent software agent can access the Audio Common Web API. Is it free for public use?
I thank the authors for their rebuttal, but their comments have not changed my decision.


Metareview by Christoph Lange

The reviewers agree that this work lacks novelty; 3 out of 4 reviewers consider this clearly negative.  
Given the strong focus on the ontology itself, the paper might be more suitable for the resource track.
While it is good that the ontology reuses state-of-the-art ontologies, there are weaknesses in other design patterns.
Other than a formal validation of the ontology, there is no suitable evaluation of the approach.
Also, there is no clear reference to the state of the art in tools and methodologies.


Share on

Leave a Reply

Your email address will not be published. Required fields are marked *