Paper 96 (Research track)

Ensuring Workplace Safety in Goal-based Industrial Manufacturing Systems

Author(s): Ahmed Shafei, Jack Hodges, Simon Mayer

Full text: submitted version

Abstract: One of the most critical challenges in human-robot collaborative work settings is ensuring the health and safety of the involved human workers. We propose to integrate semantically represented workplace safety rules that are published by regulatory bodies with task-level planning, meaning that our system can adapt to produce different variants of a product while respecting workplace safety regulation. Our prototype system interacts with human workers and machine agents via Activity Streams and a speech synthesis interface, and we have shown that it can scale to scenarios that incorporate complex products and many agents. The current system state and action logs of the agents and products are easily observable using a dashboard interface. The semantic models were evaluated by five experts in workplace safety and process engineering who expressed confidence about using, maintaining, and extending the models themselves after only negligible training, a crucial factor for the real-world adoption of such systems.

Keywords: Workplace Safety; Automated Planning; Semantic Web; Web of Things; Internet of Things

Decision: reject

Review 1 (by Agnieszka Lawrynowicz)

(RELEVANCE TO ESWC) The paper is relevant to ESWC2018 as it discusses the use of ontologies and semantic technologies. 
More detail on semantic models (ontological models of manufacturing processes and safety regulations) would be beneficial to the ESWC2018 audience.
(NOVELTY OF THE PROPOSED SOLUTION) The paper presents an innovative application of semantic technologies to automatically enforce workplace safety regulations in goal-driven industrial manufacturing processes where human workers and robots collaborate.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The paper describes a complete system for enforcing workplace safety regulations while generating plans for manufacturing. 
However, the paper lacks details on the proposed components, such as ontologies, the presentation is very high-level. 
It is unclear from the paper, what is an exact coverage by the system of the modeled domain. 
The described ontologies are not publicly released.
There is lack of any quantitative information on the developed models.
(EVALUATION OF THE STATE-OF-THE-ART) There is no evaluation with respect to the state-of-art. 
Moreover, it is unclear from the paper what could be the state-of-art, since the main contributions of the paper and research questions behind this work are unclear.
Are the main contributions:
-ontological models?
-their implementation?
-a collaborative operational model?
-all of these altogeher?
-a system?
The paper should make more clear what is the main research question and contribution and then refer to it in the related work section, describing how this work advances the state-of-art. 
Regarding modeling the domain of safety engineering, hazards, etc. I could point the authors to my co-authored works - the authors are free to decide whether they are going to include these or not:
[1] Agnieszka Lawrynowicz, Ilona Lawniczak: Towards a Core Ontology of Occupational Safety and Health. OWLED 2015, Lecture Notes in Computer Science 9557, Springer 2016, ISBN 978-3-319-33244-4,  134-142
[2] Agnieszka Lawrynowicz, Ilona Lawniczak: The Hazardous Situation Ontology Design Pattern. WOP 2015
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The system is described on the level of the architecture and its components.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The experimental study is not reproducible.
The described models/ontologies are not publicly available.
The paper describes two evaluations.
First evaluation deals with demonstrating the system to 5 domain experts and asking for their opinion via a generic questionnaire consisting of 4 questions. 
First of all, the conducted experiment is limited in scope and in the number of the participants, secondly, the conclusions which are drawn are not justified, too strong, in particular:
1) „four out of five participants expect that it would be easy or very easy for them to extend the ontology (question b).” 
leads to the conclusion: 
"We were able to demonstrate that [...] the system’s underlying semantic models can be understood and extended by subject-matter experts with only negligible prior training"
The performed experiment did not show that, it only showed that the experts believe it would be easy for them. 
For that (measuring extensibility) I would recommend to conduct an experiment where the experts actually extend the models, because in the current setting, the conclusion that may be drawn is that the participants think the models would be easy to extend, but there is no practical evidence for that.
2) „The participants said that they find our concrete prototype system – in particular the dashboard interface – straightforward to use for configuring and controlling manufacturing processes (question c).”    
For that, measuring ease-of-use, the subjects should have been given a task to solve with use of the interface (e.g., similar to assembling the 4-leg stool), and after that it could be measured for instance: percent of completion of the task within a given time, time, correctness of completion of the task etc. with respect to a baseline approach (e.g., the current interface they use). I would not draw conclusions on the ease of use based on only impressions of the participants after seeing a demo.
The second evaluation deals with measuring the scalability of the system. 
From what I have understood, the two performed experiments evaluated the system when 150 agents where available to be assigned to produce 1 piece of furniture (a 4-leg stool) and when some parts of this 4-leg stool where somehow replicated to form more complex product but still it was 1 piece? 
For that, I would expect more variety in the complexity of the product and I am also not convinced that the experiments dealing with production of 1 piece of furniture show well the scalability of the approach. 
I would recommend the authors to reduce the claims and conclusions drawn from these experiments.
(OVERALL SCORE) The paper presents a prototype system for automatic enforcement of workplace safety regulation in goal-driven industrial manufacturing, which is capable of creating collaborative manufacturing plans for human and machine agents while taking into account individual as well as generic constraints of the agents with use of semantic technologies. 
The main result is an innovative, prototype system. 
It is hard to make a decision on the paper. From one side, the paper presents an innovative and important application of semantic technologies, which I believe is potentially impactful and would like and recommend to be presented at ESWC, from the other side, the paper does not provide the details regarding semantic technologies and the evaluation part is weak, and the experiments not reproducible.
I am also not sure whether the paper fits into the "Reasoning" topic (since it does not discuss any new reasoning algorithms, theoretical results etc.), or whether it maybe could fit better to "In-use" track or a demo session.
Strong Points (SPs)  ** Enumerate and explain at least three Strong Points of this work**
* Interesting, and potentially important application of semantic technologies 
* Innovative combination of several technologies
* Potential impact of this work in an important domain
Weak Points (WPs) ** Enumerate and explain at least three  Weak Points of this work**
* The main contribution/contributions of the paper are unclear  
* The described resources are not publicly available, any detailed, nor quantitative data about the resources (ontologies, constraints) are provided, and thus the experiements are not reproducible
* The evaluation part is weak and the drawn conclusions not fully justifiable 
Questions to the Authors (QAs) ** Enumerate the questions to be answered by the authors during the rebuttal process**
* fischertechnik - please, explain the meaning of this word
* wrt the experiment dealing with scalability: 
How those 150 available agents where assigned to the task - using any priority or particular strategy or first-available? 
Where the times presented in Figure 7 average over several trials or was it only 1 trial?
Overall, the work is really nice.  However, it lacks research depth. 
I am very in favour for this work to be presented at "In-use" track.
For this work to be appropriate for the research track, the paper should concentrate on a selected aspect/aspects in more depth, e.g.:
-3.2 Ontological Modeling of Manufacturing Processes and Safety Regulations
-3.3 Safety-aware Flexible Workflow Planning
to the extent that it would allow other researchers to build upon this work and its contribution further to progress research by either: making some of the models re-usable or making it possible to reproduce described experiments or providing a baseline with sufficient details to which subsequent approaches can be compared.
*****After rebuttal*****
I thank the authors for their comprehensive answers provided to the issues raised during the rebuttal. 
I am in favour of accepting the paper to the in-use track.

Review 2 (by anonymous reviewer)

(RELEVANCE TO ESWC) The paper is about many things (human-robot collaboration, ontologies for health and safety, manufacturing, Microsoft kinect, AI planning, graphic interfaces, etc). The authors are presenting an approach where automated reasoning is used to create and monitor workflows that are to be executed by humans in collaboration with robots. In particular, ontological reasoning is used to ensure that the workflows obey certain safety requirements (e.g., a worker might be ordered to wear headphones if some loud drilling is going to happen next).
(NOVELTY OF THE PROPOSED SOLUTION) It was very interesting to read the paper.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The paper is very well written in the sense that it is easy to read. However, the paper is so broad in scope that it lacks depth. Many interesting and appealing features are mentioned, but no particular aspect is discussed in detail. So the question after reading the paper is: what is the concrete solid new knowledge being communicated via this paper? For instance, the proposed system hinges on a number of ontologies that are only superficially described and which seem to be unavailable to the reviewers or the general public to see.
(EVALUATION OF THE STATE-OF-THE-ART) The approach involves AI planning for generating workflows. AI planning is a very well-established area in AI, with a number of works on different languages to express planning problems, as well as a number of systems for automated planning. AI planning is usually computationally difficult (intractable). The authors should discuss this fact. The experiments described in Section 4.2 oversimplify the situation in the sense that they don't seem to represent AI planning problems that one would encounter in real life (a 4-legs stool made by 150 agents?).
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Since the paper is not very deep, reproducibility is questionable.
(OVERALL SCORE) Please see above.

Review 3 (by anonymous reviewer)

(RELEVANCE TO ESWC) The paper uses ontologies, rules and reasoning, so the topics are relevant for the ESWC.
(NOVELTY OF THE PROPOSED SOLUTION) The idea and the use case are not really novel. There has already been some work in this area.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The proposed solution is nicely described. The main concern here, is that it is very high-level and broad, lacking approach and implementation details.
(EVALUATION OF THE STATE-OF-THE-ART) The section needs to be improved.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) There are no links to the ontologies, code, rules etc. The results are not reproducible
(OVERALL SCORE) The paper presents an approach and a system for automating manufacturing systems, by taking into consideration activity streams and rules in order to be able to guarantee workspace safety. The authors use ontologies, rules and reasoning as the core of the proposed solution, therefore, the work is relevant for the ESWC community. The idea of automating industrial manufacturing systems and especially taking into consideration safety regulation is not necessarily a new one. Initial work in this context has be published under the topics of smart factories, starting about 3-4 years ago. However, the topics is still relevant and current. 
The paper is nicely written, easy to follow and the sections are clearly structured. The line of argumentation is also clear and the language use is good (except for some minor errors, s.b.). 
One concern that I have is that the paper would have been more suitable for the in-use track. The main idea of the paper is to present a system and a specific use case. The in-use track would really have been a better match. This leads to the second main concern, which is that the paper, as it is currently written, lacks fundamental research and research depth. Everything, the architecture, the ontologies, the rules are described on a very high level, without sufficient detail. The results of the evaluation would be in no way reproducible and the system, as well as its parts, cannot be reused (there are not links to repositories, ontologies, etc.). The paper lack sufficient research depth for the research track. Some detailed comments are given below:
1) The state of the art section needs to be improved. There is quite some work in this area. Ex. Keppmann & Harth
2) Introduction section “model of agency”, you mean agents?
3) Do not use abbreviations such as isn’t
4) Clearly state when changes to the setup can be made— at runtime or only at design time. Does the system need to be redeployed/ restated if changes are made?
5) The system architecture looks more like a workflow. I would suggest that you clearly define the system borders, the actors, the input/output devices, data storage/data management components, etc. Make it a prober architecture that can be reused by other researchers with the same problem. Make sure that you differentiate between a general architecture and a use case implementation (which you currently have). 
6) It is quite nice that you integrated voice control. Using Kinect is not really new but voice control is nice.
7) Do you shutdown the system is the person is in danger? Sometimes warning are not enough…
8) It would be nice to have a clear definition of what you consider to be an agent. Unfortunately, there a quite a few definitions, some of them even inconsistent, so in term of approach and implementation, it is good-practice to define the terms that you are using (especially, if they are inconsistent) 
9) How is a furniture piece tracked? Kinect cannot really do that directly. People yes, furniture not really. Did you implement this yourself?
10) Information model architecture — it is really nice that you are using standards, as much as possible. Here you would definitely need to link to the ontologies. The description is very high level, you need more details for a research paper. Section 3.2 wehre are all the models available?
11) 3.3 here you suddenly talk about SVM-classifiers, this is a bit surprising. You did not talk about this until now. 
12) 3.3 Who creates the rules. It would be very important to give specific examples (or include links to some)

Review 4 (by Laura M. Daniele)

(RELEVANCE TO ESWC) This is a relevant paper to ESWC that proposes the application of semantic technologies to Smart Industry and Manufacturing, a very important domain (see Industrie 4.0 in Germany and related initiatives in Europe) that unfortunately did not find yet enough attention in the Semantic Web community. I feel we should encourage more submissions in this area in the near future, given the major role that semantic technology can play in solving interoperability issues in the manufacturing domain and, more in general, in the Internet of Things.
(NOVELTY OF THE PROPOSED SOLUTION) There is little novelty in the proposed architecture and the use of ontologies as a means for interoperability, but its application to collaborative manufacturing, combined with the use of international standards (for workspace safety in the paper), makes the proposed solution particularly valuable.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The proposed solution consist of a prototype that helps to ensure workspace safety in collaborative manufacturing tasks between humans and machines. The prototype is well presented throughout the paper. The authors also provide an evaluation of the prototype in terms of 1) usability, by means of interviews with five domain experts, and 2) scalability, via simulations that increasingly add complexity to the task at hand and measure the system’s performance accordingly. Although I understand that it is difficult to acquire real data from industry for a research prototype, this is a rather weak evaluation for a paper submitted to the research track at ESWC. I expected the paper to provide a contribution on reasoning related to ontologies, rules and the Web (reasoning sub-track), but it was actually quite hard to find an actual research contribution. The paper should have rather been submitted to the In-Use & Industrial Track.
(EVALUATION OF THE STATE-OF-THE-ART) The evaluation of the State-of-the-Art is missing. The issue of semantic interoperability in the Internet of Things is only mentioned in the introduction, but no further elaboration nor related work is provided*. Given the importance and relevance of smart industry and manufacturing in our society, I would have expected a reference to the various national initiatives in Europe that support digitalization in manufacturing**, including a positioning of this work in the context of reference architectures for Industry, such as the RAMI model [1] used in the Industry 4.0 initiatives for the alignment of various standards at different levels of abstraction.
* See for example how the AIOTI- Alliance for Internet of Things Innovation - ( addresses this problem in the dedicated Semantic Interoperability group, part of the Working Group 03 on Standardization. Note that AIOTI has also a dedicated Working Group (WG11) that specifically focuses on Smart Manufacturing. 
** For example:
• The platform Industry 4.0 in Germany
• The Smart Industry initiative in the Netherlands
• Industria 4.0 in Italy
• Industrie du future initiative in France
• Etc.
[1] Adolphs P, Epple U, et al. Status Report Reference Architecture Model Industrie 4.0 (RAMI4.0). Düsseldorf, Frankfurt 2015. VDI – The Association of German Engineers, ZVEI – German Electrical and Electronic Manufacturers’ Association
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The prototype system is clearly presented. However, the paper does not explicitly present nor discuss an approach.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The paper introduces a number of ontologies, but I could not find any link to them in the paper. It seems that no URL is provided, while it is best practice to publish ontologies at a persistent URL. In this way the work of the authors cannot be reused by the community. Moreover, the reviewers cannot evaluate the quality of the ontologies used as basis for the prototype presented in the paper (if there were issues to publish these ontologies, you could have made them available at least for the reviewers).
(OVERALL SCORE) The paper presents an interesting application of semantic technologies to the smart industry and manufacturing domain. In particular, the authors propose a prototype based on a number of ontologies and associated reasoning that can help to ensure the health and safety of human workers in the factory by automatically enforcing safety regulations. An initial evaluation of the usability and scalability of the prototype is also presented.  
Strong Points:
• Interesting and well written paper on a relevant topic, nice to read.
• Valuable application of ontologies and reasoning to a real industry scenario (i.e., workplace safety regulation in goal-driven industrial manufacturing processes). 
• Integration of international standards (e.g., ANSI/ISA-88 for batch process control) in the ontology framework 
Weak Points:
• It does not provide a scientific contribution, it would better fit in the In-Use track
• The work is not positioned with respect to the various initiatives in Europe that support digitalization in manufacturing
• The ontologies used as basis for the prototype are briefly introduced in the paper, but no URLs are provided. 
• Reasoning is only mentioned in the paper. It would have been nice to see an example of the SPIN rules used to automatically enforce safety regulations in the workplace.

Metareview by Diego Calvanese

The paper presents an interesting application that provides an innovative combination of semantic technologies to automatically enforce workplace safety regulations in goal-driven industrial manufacturing processes where human workers and robots collaborate.
There is agreement that the paper is well written, and that it presents an interesting application that would be worth being discussed at ESWC.  However, the paper is too broad and high-level, and lacks research depth.  Therefore it is not suitable for the ESWC Research Track (Also, a switch to the In-use Track, where the paper might be better suited, appears infeasible, due to the separate reviewing processes of the two tracks.)
The authors should also consider to make the used ontologies publicly available, and ensure reproducibility of the experimentation.

Share on

Leave a Reply

Your email address will not be published. Required fields are marked *