Paper 205 (Research track)

Smart Papers- Dynamic Publications on the Blockchain

Author(s): Michal R. Hoffman, Luis-Daniel Ibáñez, Huw Fryer, Elena Simperl

Full text: submitted version

camera ready version

Decision: accept

Abstract: Distributed Ledgers (DLs), also known as blockchains, provide decentralised, tamper-free registries of transactions among partners that distrust each other. For the scientific community, DLs have been proposed to decentralise and make more transparent each step of the scientific workflow. For the particular case of dissemination and peer-reviewing, DLs can provide the cornerstone to realise open decentralised publishing systems where social interactions between peers are tamper-free, enabling trustworthy computation of bibliometrics.

In this paper, we propose the use of DL-backed Smart Contracts to track a subset of social interactions for scholarly publications in a decentralised and reliable way, yielding Smart Papers. We show how our Smart Papers approach complements current models for decentralised publishing and analyse cost implications.

Keywords: Dynamic Publications; Smart Contracts; Blockchain; Ethereum; Open Decentralised Publishing; Collaborative Processes; Trust

 

Review 1 (by Paul Groth)

 

(RELEVANCE TO ESWC) Distributed trust scenarios are of interest to the semantic web community as whole. This article explores such a model for scholarly communication.
(NOVELTY OF THE PROPOSED SOLUTION) This is a novel contribution in designing a system to perform the scholarly communication process using blockchains.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The proposed solution uses two existing and well known technologies in the blockchain space namely Ethereum and IPFS. The smart contracts that were designed are made available. Unfortunately, there was no description about how to actually execute the smart contracts. This would have made the contracts more understandable.
(EVALUATION OF THE STATE-OF-THE-ART) The state of the art is reasonably good but I think it's missing some contextual literature. 
Research objects are often mentioned in the paper. Research objects have been extensively discussed in the semantic web literature and the notion should be at least cited, see:
Sean Bechhofer, Iain Buchan, David De Roure, Paolo Missier, John Ainsworth, Jiten Bhagat, Phillip Couch, Don Cruickshank, Mark Delderfield, Ian Dunlop, Matthew Gamble, Danius Michaelides, Stuart Owen, David Newman, Shoaib Sufi, Carole Goble (2013) Why Linked Data is Not Enough for Scientists, Future Generation Computer Systems 29(2), February 2013, Pages 599-611, ISSN 0167-739X, https://doi.org/10.1016/j.future.2011.08.004 
It would be worth mentioning platforms that use Git as a backend (e.g. overleaf.com) and in general the use of GitHub/Git to facilitate scholarly publication.
Additionally, it would be good to explore how approaches such as https://hypothes.is play into the system described. Essentially, the question I have is how much of the desiderata described actually *need* a distributed ledger? Indeed, E
Finally, and I discuss this more in the general review, the paper hinges on a very bibliometrics view of science assessment. It would be good to mention that this is really only one view of science assessment. That field is large but it would be good to briefly discuss the alternative point of view. A place to start is MacRoberts, M. H. and MacRoberts, B. R. (2017), The mismeasure of science: Citation analysis.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The properties of blockchain and its applicability to scholarly communication are well described. I would suggest also mapping the role of publisher to that of notary in your comparison table. I think that would help in understanding the relation to the existing system.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The experimental study focuses around costs to run the proposed solution on the blockchain. The evaluation is reasonable but it would be good to at least include a discussion of comparative system costs (e.g. the use of a submission system). Additionally, I think it's important to note that there are other kinds of transaction costs.
(OVERALL SCORE) Comments after rebuttal: I thank the authors for their response. However, while they discussed my concerns it's unclear what you intend on changing in the paper to address them, which is a bit unfortunate. I think they deserved at least some concrete changes. 
* Summary of the Paper
This paper proposes a model/infrastructure for scholarly communication based on smart contracts running on the blockchain. The paper measures the cost of executing such smart contracts on an existing blockchain. 
* Strong Points (SPs)  
- Novel concept 
- Evocative discussion
- Well written
* Weak Points (WPs) 
- The evaluation doesn't include a comparator system
- The contextualization within the larger literature around scholarly communication is missing
I'm actually a bit conflicted about this paper. On the positive front, I think it's an interesting approach and the exploration of semantic technologies and distributed ledgers is extremely interesting for our community. 
On the other hand, I find the premise of the paper not justified with respect to the state of practice within scholarly communication. To be honest, the premise is depressing. Essentially, it starts from the notion that scholarly interaction is an inherently untrustworthy endeavor. In fact we know from the literature that science is fundamentally a social & mostly high-trust system. A good place to begin in the literature looking at the overlap of social and technical is:
Christine L. Borgman. 2007. Scholarship in the Digital Age: Information, Infrastructure, and the Internet. The MIT Press.
It's also worth noting the various social practices across the research ecosystem:
Cronin, Blaise. "Scholarly communication and epistemic cultures." New review of academic librarianship 9.1 (2003): 1-24. APA	
I would contend that scholarship makes progress high-trust system. Indeed successful research organizations are based on high-trust (e.g. https://www.mpg.de/39586/MPG_Introduction). Both the massive (Sloan Digital Sky Survey) and small collaborations (phd student and advisor) are constituted on trust established by social interactions. The idea that we can reify those actions in a smart contract is questionable. I would wish for at least some argument as to why one would do this. 
Additionally, the notion that evaluation of scholarship is essentially tied to measurable quantities needs to be confronted or at least mentioned in the paper. 
Finally, I thought it was interesting that the approach describes relies upon ORCID identifiers which is an infrastructure based upon the trust in the existing scholarly infrastructure system.

 

Review 2 (by Mario Cannataro)

 

(RELEVANCE TO ESWC) In this paper the authors presents a methodology to realise open decentralised publishing systems based on Distributed Ledgers where social interactions between peers are tamper free, enabling trustworthy computation of bibliometrics. the peer-review process.
(NOVELTY OF THE PROPOSED SOLUTION) - Smart Papers provide a collaborative platform that preserves a single version of the truth throughout the collaborative process.
- Trustworthiness throughout the collaborative process among peer.
- A distributed authors agreement that guarantee that none of the actors can tamper with the document.
– A mechanism that ensures that annotations made on releases by agents other than authors cannot be repudiated by annotators or their recipients.
– An index of links and data concerning a particular dynamic publication.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The authors present only a description of the proposed methodology without providing any detail about the implementation and how the used modules are integrated among them.
(EVALUATION OF THE STATE-OF-THE-ART) The relevant literatures has been cite
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) The authors provide a exhaustive description of possible use case of the proposed methodology
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) Authors do not provide enough information about the system architecture to be possible to reproduce the methodology
(OVERALL SCORE) In this paper the authors presents a methodology to realise open decentralised publishing systems based on Distributed Ledgers where social interactions between peers are tamper free, enabling trustworthy computation of bibliometrics. the peer-review process.
Strong Points (SPs)
- Smart Papers provide a collaborative platform that preserves a single version of the truth throughout the collaborative process.
- Trustworthiness throughout the collaborative process among peer.
- A distributed authors agreement that guarantee that none of the actors can tamper with the document.
– A mechanism that ensures that annotations made on releases by agents other than authors cannot be repudiated by annotators or their recipients.
– An index of links and data concerning a particular dynamic publication. 
- The paper is reasonably well written, and all the relevant literatures has been cited. 
However, I have several concerns and there are some places need to be improved. The comments are as follows.
1. Comparison between the proposed methodology and the state should be presented in a summary table. Summary table makes easier for the readers to understand the major differences between the state of the art and the proposed methodology.
2. The authors should provide the main architecture of the proposed methodology. Implementation needs to be detailed in more formal manner, making it possible to better understand how the authors implemented in SmartPaper employing Ethereum platform  and IPFS storage. 
 
3. The separation of concerns design principle should be explained in the paper avoiding that readers have to read the full methodology reported in [11]
4. The authors in the paper used the following acronyms without defining them: RDFA, SOL 
5. There are some grammar errors and typos in the paper let see page 2 “reusability is achieved by by an”. Page 5 miss a dot at the end of the sentence: "as stored in the conference’s data store”. page 6 "Smart Papers provide -> Smart Papers provides”

 

Review 3 (by Andrea Giovanni Nuzzolese)

 

(RELEVANCE TO ESWC) Distributed trust is certainly one of the topic of utmost importance for ESWC. 
As a matter of fact we can take into account the last panel session held at ESWC2017 when this topic was highly debated.
Nevertheless, the paper never clarifies the role of semantic technologies for achieving distributed trust.
I understand the revolutionary impact that distributed trust can have if applied to scholarly communication, however I do not fully get how the solution proposed by the authors benefits from semantic technologies. 
*** AFTER THE REBUTTAL ***
I fully get the authors' response. Accordingly, the authors should clarify the relation with semantic technologies in the paper.
(NOVELTY OF THE PROPOSED SOLUTION) The contribution is novel as limited work has been investigate so far in the context of distributed trust for scholarly communication.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) The authors propose a solution based on well known and "hot" technologies like blockchain and Ethereum. 
However, the paper can be improved by adding more details about how smart contract are actually designed.
Nevertheless, the cost analysis is really useful as a preliminary assessment of the solution in terms of potential exploitation.
(EVALUATION OF THE STATE-OF-THE-ART) To the best of my knowledge most of the relevant work in the area has been cited.
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) Considering the nature of contribution I am positively impressed by the description of the context and the potential impact the solution can have on the scholarly community.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) The cost analysis is, in my opinion, a seminal hint for understanding the exploitability of the solution economically.
Nevertheless, it provides only a single dimension perspective. This can be improved with other analysis.
(OVERALL SCORE) I am in favour of the analysis of trust for dealing with distributed scholarly processes and communication.
Additionally, the paper shows a number of positive points, i.e.:
- it is well written and structured;
- generally speaking, it tackles an hot and challenging problem that is relevant not only to the semantic web community, but to the scholarly community in a broader sense;
- the state of the art is fair;
- it introduce a certain degree of novelty.
Nevertheless, I am torn because I like the contribution, but I have to evaluate the paper as research work submitted to ESWC. Accordingly, semantic technologies should play a relevant role in the design, implementation, evaluation, etc. of the work. Honestly, I do not get (at least from the paper) the full connection between semantic technologies and the solution proposed by the author. Certainly, there is room for improvement in order to make this more evident.
*** AFTER THE REBUTTAL ***
The authors addressed all my concerns about the paper.
I think the paper is worth to be presented in the conference as it introduces fair novelty and can contribute to a qualitative discussion of trust methodologies for decentralised scholarly communication.
I fully understand the answer provided in the authors' response about the connection between their work and semantic technologies. Nevertheless, I expressly ask the authors to explicitly clarify this in the paper.

 

Review 4 (by Phillip Lord)

 

(RELEVANCE TO ESWC) It's clearly relevant.
(NOVELTY OF THE PROPOSED SOLUTION) This is a straight-forward application of block chain/ethereum to storing digitial artefacts.
(CORRECTNESS AND COMPLETENESS OF THE PROPOSED SOLUTION) Their arguments are well supported.
(EVALUATION OF THE STATE-OF-THE-ART) The paper is well sited within the current state-of-the-art
(DEMONSTRATION AND DISCUSSION OF THE PROPERTIES OF THE PROPOSED APPROACH) A practical implementation was not provided, but it should work.
(REPRODUCIBILITY AND GENERALITY OF THE EXPERIMENTAL STUDY) N/A
(OVERALL SCORE) This is a nice paper describing how ethereum could be used to store
and validate papers, versions of papers in an distributed, secure and
non-repudiable way. As the authors suggest, this is a useful thing to
do and fits with existing efforts to build a de-centralized framework.
A significant part of the paper describes blockchain and ethereum, but
it does also include a short implementation in the ethereum virtual
machine. An approximate price has been measured also which is
otherwise a trick issue with ethereum.
On the negative side, the paper does not appear to have shown a
practical implementation, nor demonstrated how this would actually
integrate with the decentralized approaches they describe. The overall
price (~2GBP) is competitive with centralized approaches (7USD for
arXiv, for example). However, it is a long way of the mark for more
fine grained systems of publication such as a git based system. More
over, it is not clear with this system how correct structuring of the
metadata (for example, the author list) would be enforced; something
that arXiv achieves straighforwardly.

 

Metareview by Adrian Paschke

 

The paper is a methodology paper about Smart Papers exploiting distributed ledgers and smart contracts to manage trust. Several of the reviewers concerns have been addressed in the rebuttal. In case of acceptance the camera-ready version needs to provide additional information on the architecture, the design of the smart contracts and on the experimental tooling.

 

Share on

Leave a Reply

Your email address will not be published. Required fields are marked *