Benchmarking LinkChains- Performance of Blockchain-backed Trustworthy Linked Data Querying
Author(s): Allan Third, Kevin Quick, Michelle Bachler, John Domingue
Full text: submitted version
Abstract: Certain uses of knowledge graphs require strong guarantees of data integrity — for example, medical or financial applications — which can be verified by the end user automatically and reliably. Distributed ledger technologies based on blockchains are able to provide a trustworthy guarantee that records have maintained their integrity since publication. There are multiple different scenarios for providing such guarantees, varying in terms of where data is stored — from traditional quad stores through distributed file systems to directly on a blockchain itself — and the strength and scope of the integrity guarantee which can be offered — with or without reliable timestamping, and at the level of whole datasets or query results. In this paper we present the results of a number of experiments across these dimensions. In particular we use the lightweight Linked Data Fragments (LDF) standard as a frontend to each experimental setup, and benchmark their performance relative to a standard LDF server using a SPARQL backend, to evaluate each for performance as candidates to implement a trusted Semantic Web.
Keywords: Linked Data Fragments; Blockchain and Distributed Ledgers; Data integrity; Benchmarking
Review 1 (by Steffen Staab)
(RELEVANCE TO ESWC) The paper to be reviewed contains primarily an evaluation of an interesting idea that has already been presented in another paper. (NOVELTY OF THE PROPOSED SOLUTION) As I have already said, the approach underlying the evaluation has already been published in another paper, so it can not be regarded as completely new. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) It is unclear to me whether the times measured in Table 1 refer only to data retrieval or also to integrity checks. As far as I understand it, this is just a measurement of the duration of the query, excluding the integrity check. However, since the evaluated concept has as an essential component the use of blockchains to enable verification, I think that the time needed for verification should be investigated as well. (EVALUATION OF THE STATE-OF-THE-ART) Since block chains are definitely a current topic, I think it is absolutely legitimate to think about integrity checks of data using blockchain technologies and to examine approaches in this respect. I cannot judge to what extent the approach of the LinkChains, which are linked here with blockchain techniques, is up to date. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The essentially discussed characteristic of the approaches compared refers to the time which, in my opinion, is not meaningful due to the above-mentioned lack of consideration of the verification times. (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Since in my opinion important details about the exact design of the used blockchain instance are missing in my opinion and the mentioned points are unclear, I would not be able to reproduce the evaluation. (OVERALL SCORE) Summary of the Paper: The paper is based on a previously introduced concept that considers storing data in LDF standard on blockchains to provide a secure and efficient way to verify data integrity. The present paper examines this concept and various possible versions of it with regard to the execution time of requests in comparison of the concepts with each other. Strong Points: - I think the approach of evaluating the use of block chains to verify data integrity is very good. - It is essential to evaluate such concepts in terms of their performance, which is the main objective of the paper. Weak Points: - I miss clear statements as to how the presented measured values are composed and whether the duration of the respective data integrity checks was also examined there. - I miss more detailed information about the structure of the blockchain - Depending on the specification of the block chain to be used, it can be helpful to include other parameters (e. g. required memory, degree of replication) in the evaluation as well as the execution time. This was not discussed in this paper. Questions to the Authors (QAs): s.a.
Review 2 (by Jeremy Debattista)
(RELEVANCE TO ESWC) The paper deals with integrity of SPARQL queries at the triple-level using blockchain to verify the integrity of data. (NOVELTY OF THE PROPOSED SOLUTION) The authors consider the combination of using a blockchain with a triple-store (through Linked Data Fragments) to provide some metric of integrity for SPARQL queries. Since this is currently not possible or addressed directly by triple-stores or comparable solutions, the proposed approach can be considered to be novel. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) While the authors provide evaluations and metrics of their approaches, it is not sufficiently explained or dealt with in detail. The authors' provided link to online resources mentioned in the paper (ref. ) is not accessible (HTTP 404). Without a more in-depth explanation of the experiment setup, indication of why the datasets were chosen for that particular setup, why chosen statistical tests were applied, and their relation to the evaluation methodology, it is difficult to interpret the results apart from the author's claims in the paper. The authors also do not mention any information about the setup environment of their experiment such as which triple-store was used, or the parameters of their blockchain (which can widely vary in all aspects). Therefore, it is difficult to interpret what metric was evaluated and which was left out or not considered in the experiments. *Update*: The link to online resources seems to be working, and contains the resource mentioned in the paper. The author's comments also mention the setup environment of their experiment. However, given the limit of the response, it is (still) difficult to understand the functioning of the experiment itself without a good explanation. It is suggested that the authors work on this towards improving their paper. (EVALUATION OF THE STATE-OF-THE-ART) While the authors provide a clear and lucid explanation of the underlying research surrounding knowledge graphs and distributed ledgers, they fail to provide any awareness of recent advances in blockchains/distributed ledgers - both related to semantic web, as well as in use in altcoins. These are important advances in the field as they demonstrate the customisation of the original blockchain architecture to achieve certain technological goals. One example of this would be the use of distributed ledgers for storing financial information or regulatory information or medical data. Some of these approaches use the blockchain to store triples (RDF) and therefore their (or lack of) approach/metric/evaluations should have been referenced and taken into consideration. The authors provide their reference architecture using Ethereum and its concept of smart contracts but fail to explain it (or its modifications) in sufficient details for the purposes of their experiment. *Update*: The authors claim that the field is novel, which is only partially true. While the particular application of technologies in this case may be a novel idea, related research exists which may be of interest. This included other blockchain-based approaches towards storing and querying linked data, as well as the implementations of contracts over blockchain (alternatives to Ethereum and Parity). There exists a vast body of research regarding verifiable integrity of data, which may not use Linked Data or RDF, but is still relevant because of its approach, models, and ideas towards solving relevant research problems. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The motivation for the approach is clearly laid out at the start of the paper. However, its demonstration is lacking a clear and relevant description. The authors discuss some underlying technologies in brief before discussing the experiment without a discussion of the approach taken and what it is based on, or which aspects they have explicitly (or implicitly) chosen to ignore/not consider. (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The authors provide a link to their work (ref ) which returned a HTTP 404 (not accessible), which makes it difficult to reproduce or understand the inner workings of the experimental workflow. The paper, also, does not provide any description of the experiment (apart from the dataset being created from DBpedia). Things such as execution environment and tools used (which triple store? which blockchain? what was the block size? verification time? what OS environment?) which have a bearing on the metrics presented in the paper are absent. The authors present a high-level and abstract explanation of the experiment and present the evaluation of metrics without a sufficient background of the experiment itself. *Update*: The links the authors have provided are accessible, and house the resources mentioned in the paper. However, this looks more like a data dump rather than an actual reproducible workflow. (OVERALL SCORE) Summary: In the paper, the authors present their experiment and results for a mechanism for verified queries using blockchains. They use LDF as the benchmarking mechanism which is then queried using SPARQL. They evaluate an on-file method (IPFS), a triple store (which is termed Base), and their combinations with Blockchain. The paper presents some metrics on different datasets from DBpedia and concludes that blockchains can be used to verify the integrity of query results by combining them with a triple-store to serve results. Strong Points: Evaluation metrics look to be detailed and multiple tests were run for suitability. Relevant approaches were considered in experiment. Work relates to integrity of triples served which is novel. Weak Points: Experiment needs more description, with particular focus on relating it to the motivation, and explaining approach taken towards evaluation and metrics. Work needs to be made accessible in the interest of reproducibility. State of the art needs to be more extensive to consider advances in related approaches. Questions to Authors: * Can the work be made accessible? * Can the authors provide references to state of the art related to the work and compare their approach through it? * Is it possible to provide more description(s) of the experiment and its evaluation and relate to why the particular approach was chosen? * The conclusion of the experiment is not clear in relation to the theme presented in the paper? Can this be elucidated? *Update*: The paper can be considered for acceptance given the authors incorporate the feedback/comments in to the final version of their paper.
Review 3 (by Tomi Kauppinen)
(RELEVANCE TO ESWC) The paper contributes to the semantic web research via "storing knowledge graph data in the form of RDF in a trustworthy and verifiable manner" (NOVELTY OF THE PROPOSED SOLUTION) There is not too many solutions and experiments of blockchain tech for semantic web. This paper provides one and also provides an evaluation. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) I wonder what the sentence "Datasets were created by selection of random subsets from DBpedia" really means. For instance is the "20000 triples" random triples or do those triples form groups by, say, having all triples about Einstein? (EVALUATION OF THE STATE-OF-THE-ART) There is not much written about the joint use of blockchains and semantic web tech. To my understanding the existing papers are well covered. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) Authors do demonstrate that the approach is both implementable and that it also performs surprisingly well. (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) All code, data and tools are online (OVERALL SCORE) Strong Points (SPs) - all code is online, thus supporting reproducibility - there is not too many solutions and experiments of blockchain tech for semantic web. This paper provides one and also provides an evaluation. - authors do demonstrate that the approach is both implementable and that it also performs surprisingly well. Weak Points (WPs) - the paper could provide more clear examples of applying the approach (blockchain with RDF) Questions to the Authors (QAs) - in your evaluation, were the subsets really random triples or do those triples form groups? - can you provide more examples different kinds of joint uses of blockchain and SW tech?
Review 4 (by Steffen Staab)
(RELEVANCE TO ESWC) Having non-refutable, possibly decentralized, triple storage is definitely an important topic for ESWC. (NOVELTY OF THE PROPOSED SOLUTION) The novel contribution of this paper sits somewhat uneasily between different places offering a little bit of benchmarking, a tiny bit of novel technology and a bit of evaluation setup. Overall, it does not provide a significant contribution in any of these directions while clearly having found topics that need to be investigated further. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The description of methods in section 5 is so vague and ambiguous that correctness and completeness cannot be ascertained, which is equally bad as not being correct and complete in the first place. A rewriting of the paper might perhaps lead to a very different judgement, but this would be beyond the scope of the conference-style submission cycle. (EVALUATION OF THE STATE-OF-THE-ART) Given the nature of the paper the target should be made much clearer and the range of its competitors should be argued for much more clearly. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The reason why certain technologies were chosen remain unclear. The description of the use cases would also justify other choices that were not considered. (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) See notes on correctness (OVERALL SCORE) General comment: - The method contribution of this paper sits somewhat uneasily between different places offering a little bit of benchmarking, a tiny bit of novel technology and a bit of evaluation setup – while offering nothing fully - Assuming we need decentralization and integrity, no case is made, why it should be blockchain. I suggest https://cacm.acm.org/magazines/2017/12/223058-bitcoins-academic-pedigree/fulltext which dissects requirements and needed building blocks, for a restart of your analysis. E.g. a case is made for how a couple of banks could come up with a tamper-proof distributed ledger, and it is not clear why Bitcoin would be preferable in your setting – especially taking into account the wasteful nature of bitcoin mining - Given the scenario, I would have expected (at least a conceptual) comparison with an append-only store. Detail comments: Abstract: - Blockchains may lead to floundering transactions. I am not sure that this vows for integrity of data. What I mostly see in blockchains is non-refutability of recorded transactions Introduction: - A case is made for integrity and non-refutability, but no case is made why decentralization and integrity need to go together. This seems in particular true for the medical knowledge use case portrayed in section 4, as I cannot recognize a major need for decentralization there (ok., surely need for a backup). Sections 1 and 2 seem verbose Section 4 does not seem to offer content beyond what was said before Section 5: This is the core section of the paper, but it leaves more questions open than it answers. In particular, I do not understand from the writing what the different cases really do: Which content-addressable network was used? How was the evaluation done? Which kind of simulator was used? What was the implementation of the blockchain? Was it done on Bitcoin or Ethereum? What were the append conditions?
Review 5 (by Khalid Belhajjame)
(RELEVANCE TO ESWC) The paper tackles a timely topic, that is BlockChain, in the context of semantic web (NOVELTY OF THE PROPOSED SOLUTION) The topic is defintely novel, and there are few papers in the semantic web area that tackles blockchains. (CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) This is an evaluation paper, the paper draws the different cases to test in a principled manner and explain the method used in the evaluation and analyze the results. (EVALUATION OF THE STATE-OF-THE-ART) The state of the art in the field of blockchain and linked data is in its infancies. The authors did a good job however in covering the relevant proposals. (DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The evaluation senarios are well explained. (REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The link provided by the authors is not working at the time of reading the paper http://blockchain7.kmi.open.ac.uk/eswc2018-rdf-speedtest (OVERALL SCORE) SP1. The paper address a novel topic that so far has been examined by very few papers in the semantic web community SP2. The paper explains the evalutaion senarios in a systematic maner SP3. The results showed that there are cases where one would have expected more overhead, and that the use of distributed ledger technology for linked data can be reasonably supported
Metareview by Emanuele Dellavalle
The paper generated a long discussion among the chairs. In the end the decision was to reject it, because, as the reviewers said, it lacks too much in reproducibility. Even if all code, data and tools are online, the exact design of the used blockchain instance are missing and this would make it impossible to reproduce the evaluation.The chairs considered reproducibility extremely important for this track and rejected the paper.