Learnings from blockchain based development

Blockchain or DLT(Distributed Ledger Technology) is getting good traction in the IT world these days. Earlier this technology was being mostly explored by the banks and other finance-related institutions in the form of cryptocurrency such as Bitcoin, Etherium. Now, it is getting explored for other use cases for building distributed applications. Blockchain technology comes with a decentralized and immutable data-structure which maintains connected block of information. Each block is connected using hash of previous block, and every new block to the chain is validated (mined) before adding and replicating it. This post is about my learnings and challenges while building an enterprise blockchain platform based on  Quorum blockchain technology.

Blockchain based enterprise platform

We have built a commodity trading platform which matches the trades from different parties and store related trade data on the smart contracts. Trade operations are private to the counterparties, however packaging and delivery operations are performed private to all the blockchain network partners. Delivery operation involve multiple delivery partners and approval processes, which are safely recorded in the blockchain environment. The platform act as a trusted single source of truth, and at the same time keeping data distributed in the client's autonomy.  

The platform built on Quorum, which is an Ethereum based blockchain technology, supporting private transactions on top of etherium. A React based GUI is backed by the API layer of Spring Boot + Web3J. Smart contracts are written in Solidity. Read more here to start on the similar technology stack and know the blockchain. 

Following diagram represents the reference architecture : 

Blockchain Development Learnings

Blockchain comes with a lot of promises to safely perform distributed operation on the peer to peer network. It keeps data secure and make them immutable, so that data written on blockchain can be trusted as the source of truth. However it comes with its on challenges, we have learned a lot while solving the challenges and making the platform production ready. These learnings are based on the technology we used, however it can be co-related any of the similar DLT solutions.   

Product, Business and Technology Alignment

Business owners, analysts, products drives needs to understand the technology and its limitation, it will help in proposing well aligned solution supported by the technology. 

• Not for real-time system -  Blockchain technology support eventual consistency, which means data (block) will be added/available eventually on the network nodes, but may not be available in a real-time. It is an asynchronous system. Products/Application built using this technology may not be a real time system, where end user expect immediate impact of the operation. We may end up building lot of overhead to make real-time end-user interfaces, by implementing polling, web-sockets, time-outs, event-bus on smart-contract events.  The ideal end user interface would be have a request pipeline  where user can see status of all its requests. Once a request is successful then only the user will expect the impact. 
• There is no rollback on the blockchain due to its immutability, so atomicity across transaction is not available implicitly. It is a good practice to keep the operation as small as possible and design the end interface accordingly.        
• Blockchain’s transaction order is guaranteed, thus more of a sequential system, if a user is submitting multiple requests then they will go in sequentially to the chain. 
• (Private Blockchain) Each user/network node has its own smart contract data for which the node has participated into the transaction on the start contract, so adding new node/user to the contact would not see the past data available on the contract. It may have implication on business exceptions. 
• Backward compatibility is anti pattern on blockchain. It would be better if we may bind business features to the contract/release versions, new features will be only available in the new contract, otherwise implementing backward compatibility takes huge effort.         

Architecture and Technology Learnings 

• Think multi-tenancy from the start - Blockchain system is mostly about operations/transactions between multiple parties and multiple organisations. Most enterprise platform have multiple layers of applications such as end-user client layer (web, mobile, etc), API layers, authentications and authorisations, indexes, integrations with other system. It is wise think of multi-tenancy across the layers from the start of the project, and design the product accordingly, instead of introducing it at the later stage.
• Security does not come on its own - Blockchain based system generally introduce another layer of integration between API and data layer. We still need to protect all the critical point at multiple network nodes owned by different participants. So defining security process and practices are even more important and complex for blockchain based solution than that of the classic web application. Make sure these practices are followed by all the participant's node.  
• A hybrid solution -  We might need to keep data index/cache (SQL or NoSQL) to meet the need multiple user reads, as reading every time from smart-contract may not meet the performance criteria. Index/cache building module may subscribe to the smart-contract transaction event, and build data index/cache to serve as read copy of block chain data. Each participant's node will have its own index and  end-users interface of that participant organisation. It will definitely add complexity to maintain the index, but until blockchain technology becomes as fast as read cache, we have this option to go Hybrid way (blockchain + read indexes). we can replay smart-contract transaction event as and when required, it helps in rebuilding indexes/cache in case we lost the index.
• Carefully add business validation in the smart-contract - we may add business data validation in the smart-contract itself but it has transaction-cost (performance cost), more complex validation may lead to out of gas issues, also getting detailed exception from the contract is not fully supported. If we are maintaining indexes then, we may do validation before making the contract transitions by reading from indexes but it is also not a proof solution, as there might be race conditions. Idea way is perform business validation before and in case of race condition occurs,  just override/ignore the operation. Choose the approach wisely based on the business impact.
• Fear of losing transactions - What if we could not process a transaction due to any error? Depending on the case, we either need to wait for the problem to be fixed and halt all the further events or just log the error and proceed. As mentioned in the previous section choose to override/ignore wisely, it may have business impact.
• Smart contract limitation - Solidity has limitation on number of parameters in a method, stack depth, string handling, contract size, gas limits. So design contract methods accordingly. 
• Tracing - Think of application tracing from web/mobile <-> api server <-> blockchain <-> indexes from the start. Each request should be traceable. Quorum doesn’t support any tracing/message header kind of mechanism to send co-relation identifier, however we may link the application a trace id to the transaction hash. 
• Configurability/Master Data Management  -  Every enterprise system needs some master data, which can be used across the organisations, since participant's nodes are decentralised, so this data need to be synchronized across the nodes. Device out a mechanism to replicate this data across the nodes. Any inconsistency may result in failure of transactions. 
• Smart Contract Backward compatibility - This is the most complex part of contract. We need to write a wrapper contract on the older contract and delegate call to old one. Note that, we cannot access private data of existing contract, so we need to implement all the private data in the wrapper contract and conditional delegate call to old contract. Managing event listeners for different versions are also complex, we may need to keep multiple versions in event listeners in the code base to support multiple versions of the contracts. We also need to make sure all the participants are on the same version of contracts. It would be better if we bound business features with version, in that case any new transaction operation may not be available on the old contract.
• Scaling the API/Client Layer - Since blockchain process the transactions is a sequential manners, so scaling api/client layer is complex, we need to implement some locking mechanism to avoid same transaction getting performed from multiple instance of API layer. Deployment is another challenge here. 
• Feature toggle on contract side is quite complex

Testing Learnings

• Solidity unit tests takes lot of gas, as the functionality grows, so it is better to use JS API based tests for contract unit tests.
• Define contract migration testing approach, and it's automation up front. 
• Automate environment preparation as it need multiple tenants nodes to interact to test the functionality.
• Api integration tests needs poll the other participants’s api to see the expected impact, it increases the api test development times as well as execution time. 
• Define approach for test automation around index/cache rebuilding
• Version based feature toggles in tests artefacts are complex to maintain.  
• Automation of meta-data management, across the nodes 
• Testing of CICD scripts, as the scripts grow so complex with time, and impact of any issue is huge. 

Continuous Integration and Deployment

• Define blockchain node deployment strategy upfront (private/public cloud), you may not  change it once it is deployment without loosing data. 
• Securely storing secrets on cloud.
• Have access management reviews. 
• Analyze impact of secret rotation on blockchain node setup, automate the process.
• Backup and restore strategy for blockchain
• Making sure all the node are on same version of blockchain and software artifact
• High availability is a challenge when new deployment has to happen, as we need to stop a service completely before deploying new version to avoid corrupt indexes/blockchain due to intermediate state of operations (multi-transaction operation).

Project Management 

• Blockchain is still a growing technology, it is better to have a dedicated sub-team which takes care of blockchain research related work, define best practices for the project and industry.
• Dedicated infra team is needed manage and improve CICD environment.   
• Security testing resources, and audits. 
• Legal point of view on GDPR and storing data in Blockchain. 
• Coordination with participant’s infra team
• Support SLA with each participants. 

Conclusion 

Blockchain/DLT is a growing technology, and a lot is yet to be done in the private blockchain space. I have presented the learnings and challenges that we have faced during the development of a blockchain based enterprise platform. Hope it will help you in avoiding some of the challenges that we have already faced.