Paper 162 (Research track)

Monitoring and Executing Workflows in Linked Data Environments

Author(s): Tobias Käfer, Andreas Harth

Full text: preprint

Abstract: The W3C’s Web of Things working group is aimed at addressing the interoperability problem on the Internet of Things using Linked Data as uniform interface. While Linked Data paves the way towards combining such devices into integrated applications, traditional solutions for specifying the control flow of applications do not work seamlessly with Linked Data. We therefore tackle the problem of the specification, execution, and monitoring of applications in the context of Linked Data. We present a novel approach that combines workflows, semantic reasoning, and RESTful interaction into one integrated solution. We contribute to the state of the art by (1) defining an ontology for describing workflow models and instances, (2) providing operational semantics for the ontology that allows for the execution and monitoring of workflow instances, (3) presenting a benchmark to evaluate our solution. Moreover, we showcase how we used the ontology and the operational semantics to monitor pilots executing workflows in virtual aircraft cockpits.

Keywords: Linked Data; Workflows; Automation; Web of Things; Internet of Things; Processes; REST

Decision: reject

Review 1 (by Hideaki Takeda)

(RELEVANCE TO ESWC) The authors assume that Read/Write Linked Data interface as a part of information system and proposed the set of the rules for workflow by Linked Data. But the former and the latter are not well combined, rather separated.
(NOVELTY OF THE PROPOSED SOLUTION) The authors claimed that the proposed the set of the rules is defined in RDFS and allows reasoning. But as long as the paper, the rules are presented while no reasoning capability is shown.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The description of the rules is actually a logical form, rather than Linked Data (RDF/RDFS/OWL). It should be formalized under some logic system, i.e., a subset of predicate logic with some axioms and inference rules. Otherwise the set of the rules cannot be evaluated formally.
(EVALUATION OF THE STATE-OF-THE-ART) The authors analyzed  the stat-of-the-art of methodologies for building large distributed systems and pointed out the combination of Restful interactions, Linked Data and workflow as a challenge to overcome the weakness of the current methodologies. The point shows the new direction for Linked Data research.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) he use of the real data is interesting and persuasive to show the validity of the proposal. But there need more descriptions how it is used and how it worked.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The onotlogy, rules and examples are available on line.
(OVERALL SCORE) The paper proposed the RDF/S based rule language for describing workflows on RESTful interactions. The authors show the ontology to express workflow models and instances and Condition-Action rules for workflows. They validated the rules as the workflow language by using Petri Nets, and tested it with a real situation. The main contribution is to propose the set of the rules for workflow in a logical way. Since it is defined in a logical form, it can be tested formally and evaluated automatically.
The strong points are (1) The proposal of the set of the rules for workflow in RDF. Since it is indeed defined in a logical way, it can be tested formally and evaluated automatically. (2) The authors analyzed  the stat-of-the-art of methodologies for building large distributed systems and pointed out the combination of Restful interactions, Linked Data and workflow as a challenge to overcome the weakness of the current methodologies. The point shows the new direction for Linked Data research. (3) The use of the real data is interesting and persuasive to show the validity of the proposal.
The weak points are (1) Miss-match of technology: The description of the rules is actually a logical form, rather than Linked Data (RDF/RDFS/OWL). It should be formalized under some logic system, i.e., a subset of predicate logic with some axioms and inference rules. Otherwise the set of the rules cannot be evaluated formally. (2) Miss-match to the goal: Linked Data for describing the rules and Linked Data for expressing data are totally irrelevant in the paper. In section 1, the authors claim the importance of information systems built with workflow of Read/Write Linked Data, but the current solution is just another workflow language for Restful interfaces. It does not seem that the features of Linked Data are embedded in the solution. (3) Miss-Match to the goal: The reasoning issue is explicitly described in the paper while the authors emphasized it in section 1. It looks that the set of the rules has capability of reasoning but it is not described. 
Minor comment: in rules, there are confusion between “hasChildActivity” and “hasChildActivities”. As long as I understand, “hasChildActivity” in the second rule of V may be “hasChildActivities”, otherwise “l” is not well bound. Similarly, 6th and 8th rules in V look the same.


Review 2 (by Valeria Fionda)

(RELEVANCE TO ESWC) The paper tackles the problem of the execution of workflows of activities in the Linked Data. The paper is interesting for the Semantic Web community.
(NOVELTY OF THE PROPOSED SOLUTION) The proposed solution is nice and combines RESTful services with reasoning in linked data environment.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The authors describe in Section 6 the operational semantics of the workflow language proposed in the paper. Even if all the rules are reported in the section, no examples are provided to explain the rules and make the proposal easy-to-understand. I do not understand why Fig. 2 is provided as an UML class diagram and not directly as an RDF/RDFS graph.
(EVALUATION OF THE STATE-OF-THE-ART) Related work are discussed in Section 2. Even if the discussion is succinct I'm pretty happy with the content of the section.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The evaluation section is quite short and it actually only reports some running times. However, I appreciated that the workflow models used are available online.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) I appreciated that the workflow models used are available online.
(OVERALL SCORE) The paper introduces a workflow language for linked data environments. After discussing the preliminaries, the operational semantics of the proposed language is introduced by enumerating the rules for the control of the workflow state, activities and transitions. Then, the paper briefly reports on some experimental evaluation.
Strong Points
1- The topic is interesting
2- The data used in the evaluation are available online
3- The proposed language is discussed by detailing all the rules
Weak Points
1- The paper could be organized better
2- Section 6 lacks of any example to help the reader
3- The experimental evaluation is not very informative
The organization of the paper can be improved in the following directions:
- A running example should be provided in the introduction and used later in Section 6 to help the reader in understanding the rules. 
- Section 3 has no reason to be kept as it is. It would be better to incorporate it in the experimental evaluation section.
- Section 5 is too short to be a section by itself. It only counts 8 lines and a figure. Moreover, it would be much better to give Fig 2 as an RDF/RDFS graph.
- Section 7.1 has no sense as a subsection of the evaluation section. There is no evaluation in it.
Typos:
- page 2 line 2: “an uniform” —> “a uniform”
- page 2: the footnote 11 is 2 page later, please move it to page 2
- page 12 line 1: “We formally showing” —> “We formally show”


Review 3 (by Daniel Garijo)

(RELEVANCE TO ESWC) The paper is highly relevant to the conference, as it presents a way to execute workflows using restful interactions and reasoning.
(NOVELTY OF THE PROPOSED SOLUTION) There has been a lot of work in workflow execution, and plenty of workflow systems exist. However I haven't seen an application such as the one proposed in this paper. That said, there is a previous workshop paper where the approach is described. The contribution of this one is on the ontology and operational semantics, hence the weak accept for novelty.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) It's difficult to assess the completeness of the approach, for which one has to download, reuse and understand a set of tools and assess the use cases (I haven't had the time to do all this, frankly). If an online instance had been set up, this task would have been easier. Regarding correctness, I think the proposed approach is interesting, but I wonder if it could be possible for anyone to interfere with the workflow instances. By just sending PUT requests to the different instances, one could trigger activities and mess around with the workflow execution. It would seem like the authors would need to devise some sort of security mechanism to make sure that the workflow ran safely end to end.
(EVALUATION OF THE STATE-OF-THE-ART) The authors have reviewed a fair amount of methods for workflow execution services. However it remains unclear to me why these existing approaches in the state of the art are not sufficient to model the problem described. For example, workflow managements systems like Taverna and WINGS use ontologies to represent workflows. I don't think the concepts illustrated in the ontology are significantly different from what workflow management systems use, despite the change of paradigm.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) [Actually I think the score here is borderline, but there is no such option]
The ontology doesn't seem to have been evaluated. It is unclear where are the requirements that were used for its design. The operational semantics seem to have been evaluated, although Table 1 is not discussed in the text at all. 
I also don't understand what section 7.3 is aiming to achieve.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Resources are online, and I have been able to access all links proposed in the paper. My only concerned is that the ontology is not documented (which makes it not very reusable) or properly published (content negotiation), which should be addressed if the paper gets accepted.
(OVERALL SCORE) --------after rebuttal---------
I would like to thank the authors for their responses. I am happy to maintain my original score.
--------original review--------
This paper introduces an ontology for describing workflows and workflow instances in the context of Linked Data applications. The authors introduce a set of operational rules for ensuring that different workflow execution patterns can be run successfully, and run a small evaluation using intelligent building data.
The paper reads well and is relevant for ESWC, although the contribution is incremental with respect to previous work. Summarizing the points explained above, I think the approach has some advantages: novel and original approach, a very easy to extend approach for adding constraint reasoning and having all the resources accessible online (even if not properly documented).
On the negative side, it seems a little difficult to design workflows in this fashion. Maybe a user study would help in this regard, or a user interface to facilitate workflow specification. In addition, any additional explanations regarding the potential vulnerabilities of the system would be appreciated.
I think the paper would be interesting for the ESWC crowd, and thus I recommend to weakly accept it.


Metareview by Ruben Verborgh

We had a long discussion about this paper, and especially about its added value as a research contribution to ESWC over the existing workshop contribution.
The main problem, from a research perspective, is that a clear research methodology with research questions is missing. While the text announces “an empirical evaluation of our approach”, the paper only provides timing information, which does not cover the approach but only a very specific (and likely not the most relevant) parameter of it. The authors measure the average runtime of workflows, but it does not seem that this was a problem in the first place. E.g., if the problem were that existing workflow engines are too slow or have the wrong scaling behavior, and you have an algorithm to improve on this, you could set up the corresponding research questions and validate them with such an experiment. However, in the current paper, it is unclear what exactly the evaluation is supposed to prove. In that sense, it appears too much as an engineering contribution to which an evaluation has been appended for form, but it does not answer any questions (which were not raised either).
We would suggest that the authors try to find the right research setup for this contribution, by finding the right research questions and answering them (and I think these questions will not in the first place be about timing). Alternatively, this could evolve into an engineering/in use contribution, or a fully theoretical contribution.


Share on

2 thoughts to “Paper 162 (Research track)”

  1. For completeness, here is our rebuttal text (which did not alter the reviews, except for reviewer 3’s acknowledgement and updates to the numeric verdicts):

    We thank the reviewers for their valuable time and the thorough review of our paper.

    In turn, we discuss the reviewers’ points.

    Reviewer 1
    ==========

    We would like to comment on reviewer 1’s list of three weak points:

    1) “The description of the rules is a logical form, rather than Linked Data (RDF/RDFS/OWL)”.

    For execution, we give the rules in Notation3 (RDF/Turtle-based). A Notation3 file containing the operational semantics is available at [A] and linked in the ontology [C], which is linked in the paper.
    For the paper, we chose binary predicates instead of Notation3 to make the presentation more concise. Moreover, most rules in the paper do not define inferencing, where RDF(S)/OWL would fit, but cause actions, hence the section “Condition-Action Rule Language”.

    We will add a pointer to said Notation3 file [A], and a reference to a forthcoming paper, where we describe how the rule language’s semantics relates to Linked Data and RDF’s model-theoretic semantics [B].

    2) “The current solution is just another workflow language for Restful interfaces”, “it does not seem that the features of Linked Data are embedded in the solution”

    We disagree on both points:

    To the best of our knowledge, existing workflow languages do not embrace the state-based nature of REST. Our language/solution is based on REST, and embraces the state-based nature of REST, ie. operates on Richardson Maturity Level >= 2.

    We use the features of Linked Data as follows:
    * For the control flow, the operational semantics:
    ** performs lookups on the HTTP URIs of the resources that represent the workflow models and instances, described in RDF and residing in a Linked Data Platform (LDP) container,
    ** follows links, eg. from the workflow instances to the models, and the activities’ instances,
    ** performs some OWL-LD reasoning, eg. transitivity, see [C], 3), and section 7.2.
    * For the activities’ actions, RDF is exchanged in HTTP requests.
    * For the activities’ (post)conditions, SPARQL ASK queries are executed on RDF downloaded using HTTP from different linked sources, integrated using reasoning.

    3) Reasoning is part of the goal, but not described in the paper.

    Indeed, we focused on the condition-action rules in the paper, as we consider those rules as our core contribution. Yet, our approach involves off-the-shelf monotonic reasoning (OWL-LD), eg. when deriving triples with :hasChildActivity property from the RDF list in the object position of :hasChildActivities triples (this should also clarify the confusion that led to reviewer 1’s minor comment). Moreover, as our approach is entirely built on RDF, HTTP, and rules, we can easily add rules for data integration (RDFS or other entailment regimes) to process the data for (post)conditions and activities.

    We will add remarks to the paper:
    * how we employ reasoning
    * how our approach enables to use reasoning in a workflow context

    Reviewer 2
    ==========

    We highly appreciate the suggestions for improving the structure of the paper and will consider them for the final version.

    Reviewer 2 wondered about the purpose of section 7.1. The mapping to Petri-Nets should formally evaluate by showing correctness. We will make this more explicit.

    Reviewer 3
    ==========

    Reviewer 3 remarked that our paper is merely incremental over a previous workshop paper. While we have published on the rule language/interpreter [21, B], and on a project use-case [14], the submitted paper is, to the best of our knowledge, our first paper that provides an in-depth formalisation of the handling of workflows.

    Reviewer 3 named a potential attack. We agree, but can point to off-the-shelf remedies, eg. restricting access to the instance LDP server using an IP white-list. We will add a corresponding brief note.

    Reviewer 3 wondered about the purpose of section 7.3. Here, we show the applicability in an industrial setting. We will make more clear how our approach and semantic reasoning helped achieve the project goals. FWIW, a user interface to design, run, and monitor workflows has been developed, used, and successfully evaluated in the project, but never published.

    Of course/as requested, we will briefly discuss Table 1, and add more documentation and content-negotiation (RDF/XML and Turtle is already provided) to the ontology.

    [A] http://purl.org/wild/semantics
    [B] Käfer, Harth: “Rule-based Programming of User-Agents for Linked Data”, Proc. 11th [email protected] (2018). Forthcoming.
    [C] http://purl.org/wild/vocab

  2. Reviewers 1 and 3 criticised our paper as merely incremental over a previous paper. The meta-reviewer adopted the criticism in his/her verdict by writing:

    “We had a long discussion about this paper, and especially about its added value as a research contribution to ESWC over the existing workshop contribution.”

    However, we are unaware of such a paper. We are curious to know what paper the reviewers have in mind to defend the novelty in a future submission (see rebuttal text, after which no substantiation to the criticism of incrementalness was provided).

    FWIW, you can find this submission on arxiv, which we will update with future versions:
    https://arxiv.org/abs/1804.05044v2

Leave a Reply

Your email address will not be published. Required fields are marked *