Synthesising Machine Learning and Semantic Technologies for Enhanced Teaching of Virtual Agents
Author(s): Nicole Merkle, Stefan Zander
Full text: submitted version
Abstract: Major problems of autonomous virtual agents in complex IoT environments are their lack of sufficient startup knowledge as well as their incapability to adequately adjust their internal knowledge base during runtime to specific user requirements and preferences. In Ambient Assisted Living as well as Health-Care use cases this is problematic, since an agent has to expediently operate from the beginning of its lifecycle and adequately address the target users’ needs. However, without prior knowledge about the user and the environment this is hardly possible. Our approach addresses these problems by providing a semantic knowledge representation framework in order to enable a simulator to simulate device events and user activities based on semantic task- and ambient descriptions. Through that simulated environment, we allow agents to learn the best matching actions and to adjust their policies based on generated datasets. In this way, the target user benefits from adequate and approved agent services.
Keywords: Agent-based systems; Semantic Web; Machine Learning; Simulation; IoT; WoT
Review 1 (by Evgeny Kharlamov)
The authors propose a knowledge representation framework that allows to model reinforcement learning environments, including agents, states, policies, and simulation. Assisted Living and Health Care are the corresponding use case domains, where agents have to be simulated before going into a real-world environment. Although the quality of writing and presentation is good, this work seems very preliminary. There are some nice ideas within the proposed framework, e.g. the agent profiles and environment description could allow to flexibly set up simulation environments for RL. However, this is not descibed in detail. Instead the authors describe the semantic representation of DQNs and policies with high-dimensional numerical states, which is not convincing. Also, I'm missing a concrete use case description in one of the above mentioned domains to qualify for an In-Use track. The authors claim that their approach solves the cold start problem, but they don't discuss it in detail. From my point of view this would mean that instead of the final policy, the pre-trained DQN parameters would need to be shared, so that an agent has meaningful starting parameters, but can learn further on. Also it would be interesting to see how to actually constrain a DQN to consider only possible actions in a given state instead of all actions that are pre-defined in its profile. Positive points: - Focus on sharing of trained ML models between agents in form of rules - Flexible set up of simulation environments based on domain knowledge Negative points: - Policy representation of DQNs using SWRL or SPARQL seems infeasible - Lack of description how to translate numerical representation of states and actions into rules - Benefits not clearly shown nor evaluated
Review 2 (by Andriy Nikolov)
The authors describe an approach to facilitate learning of agent’s policies in an IoT environment. The approach combines the semantic web technologies for information representation (models for states, agents, tasks, and devices) with the neural network-based reinforcement learning algorithm for deriving optimal policies. Overall, the topic of the paper is clearly relevant for the conference and is potentially interesting for the audience (particularly, combination of semantic web and reinforcement learning). However, I have two main concerns regarding the paper: - Relevance for the In-Use track. There is no description of a real-world use case. The content rather fits a Research Track paper. The proof of concept discussed in section 4 represents a rather artificial scenario, unrelated to the domains mentioned in the abstract (healthcare and smart home). - There is no evaluation of the approach. A proof of concept implementation is described, but there is no report of the performance, merely a statement that “With proceeding time, the agent gets on average more rewards than punishments”. It would be interesting to know, at least, the efficiency of the machine learning procedure. There would be interesting to have a discussion on the requirements of machine learning. The PoC is a rather limited one with a restricted set of actions and sets. What would be the characteristics of a real smart home environment? What would be the implications for the training time and the complexity of semantic descriptions of different states/actions?
Review 3 (by Alessio Gugliotta)
The paper presents a framework for autonomous virtual agents in complex IoT environments and, specifically, the authors propose an approach to address the cold-start problem of virtual agents, by means of a simulator that is able to train ML-driven virtual agents. Overall the proposed idea is interesting and the use of semantic technologies, as shared knowledge base for all agents, is appropriate. However, at this stage, the paper is not mature enough for an in-use track. All the reported examples and descriptions lack of a concrete ground to real-world domains and the experimentations are limited to a very specific and quite artificial case. In addition, in some parts, the paper is not easy to follow and there are some inconsistencies. More in details: - Section 1 describes well the problem to solve, but then it starts to describe the proposed approach, basically anticipating (not in a very clear way) the same concepts that will be then discussed in Section 3. Moreover, in this section, the authors refer to the healthcare domain; but this aspect it will not further discussed in the rest of the paper. If this is the target application domain, it would have been nice to detail here to a real concrete case/scenario that could be used in the following sections to exemplify the technical descriptions. - Section 2, as above, the healthcare domain is quoted without a clear motivation. - Section 3, the description of the approach is quite complex and not easy to follow without a concrete example as a reference. This is true for a research-track paper and indeed essential for a in-use track paper. For example, the paper refers to IoT devices, but what device? any device? also healthcare devices? and how all of this is related to clinical pathways? The descriptions reported in Section 3.1 (i.e. the semantic descriptions of the reference world) are absolutely generic. - Section 3.2, the simulation should be one of key contributions of this paper (parts of the presented solution has been already presented in past publications) since it drives the training of the agents. However, this section is limited to a brief, generic discussion of the module, without any detail about how the IoT devices are simulated and how they generate data to be consumed by agents. And again, for any type of IoT device? in any domain? - Section 4, the chosen proof of concept does not seem very relevant (how does this relate to healthcare? are we now speaking of moving robots?) and the reported discussion is quite qualitative rather quantitative, because it does not report any concrete evidence / tests / experiments (e.g. about the benefits of using semantic technologies in this context).
Review 4 (by Anna Tordai)
This is a metareview for the paper that summarizes the opinions of the individual reviewers. The reviewers acknowledge that the approach presented in the paper is interesting. It is not appropriate for the In-Use track as it lacks application and evaluation. It could be a candidate for a research paper at ESWC but the work would need to be more mature. Laura Hollink & Anna Tordai